From patchwork Thu Dec 23 12:30:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Peng X-Patchwork-Id: 12698228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E3F5C433FE for ; Thu, 23 Dec 2021 12:32:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEB4D6B0085; Thu, 23 Dec 2021 07:32:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B99BE6B0087; Thu, 23 Dec 2021 07:32:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A610A6B0088; Thu, 23 Dec 2021 07:32:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id 874A56B0087 for ; Thu, 23 Dec 2021 07:32:19 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4CF2988487 for ; Thu, 23 Dec 2021 12:32:19 +0000 (UTC) X-FDA: 78948996798.03.9CEF5B2 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf31.hostedemail.com (Postfix) with ESMTP id 0DDD820021 for ; Thu, 23 Dec 2021 12:32:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640262738; x=1671798738; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=4zBy2wqGUUHRa+bg8KFrUxUekBSmOOoI4eennjfoLXQ=; b=Fw3nkNZfh3EOr16a+JImoqvoFJ59BMoAPDcLqF+mMpfoOA5ndoYbkynZ iGxchhpbC0cFbGudkGApMgLnjTjUM84jnPu7+LAliRLBcqszButOGxi0H qrdToqFIzVa2Rv+2Ra+TmxfwTkgly83R8+Fp22hJoa9h9nrz4PbNCzp0y xInXBMY8cCWBDkJf1suPxwtvyDjZ5/QkH3TryDoIPYE8oMiCLeulqd6EC P3/0YKVA0vJKldbITDwtZF05qfqGiprqqUNRB940O2GCsGIVjcCrCCrD4 aaIKDa2gQ6QK+D3asQCT18YoIHW81WePwN2Fk9PRD4/cp66eVgimIIFfP w==; X-IronPort-AV: E=McAfee;i="6200,9189,10206"; a="227661096" X-IronPort-AV: E=Sophos;i="5.88,229,1635231600"; d="scan'208";a="227661096" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Dec 2021 04:32:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,229,1635231600"; d="scan'208";a="522078930" Received: from chaop.bj.intel.com ([10.240.192.101]) by orsmga008.jf.intel.com with ESMTP; 23 Dec 2021 04:32:09 -0800 From: Chao Peng To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, qemu-devel@nongnu.org Cc: Paolo Bonzini , Jonathan Corbet , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H . Peter Anvin" , Hugh Dickins , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Yu Zhang , Chao Peng , "Kirill A . Shutemov" , luto@kernel.org, john.ji@intel.com, susie.li@intel.com, jun.nakajima@intel.com, dave.hansen@intel.com, ak@linux.intel.com, david@redhat.com Subject: [PATCH v3 kvm/queue 10/16] KVM: Implement fd-based memory invalidation Date: Thu, 23 Dec 2021 20:30:05 +0800 Message-Id: <20211223123011.41044-11-chao.p.peng@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211223123011.41044-1-chao.p.peng@linux.intel.com> References: <20211223123011.41044-1-chao.p.peng@linux.intel.com> X-Rspamd-Queue-Id: 0DDD820021 X-Stat-Signature: qq3o7p8ue1qcbt94ptxcbgwzcg7rajeq Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Fw3nkNZf; spf=none (imf31.hostedemail.com: domain of chao.p.peng@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=chao.p.peng@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspamd-Server: rspam10 X-HE-Tag: 1640262724-357909 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KVM gets notified when userspace punches a hole in a fd which is used for guest memory. KVM should invalidate the mapping in the secondary MMU page tables. This is the same logic as MMU notifier invalidation except the fd related information is carried around to indicate the memory range. KVM hence can reuse most of existing MMU notifier invalidation code including looping through the memslots and then calling into kvm_unmap_gfn_range() which should do whatever needed for fd-based memory unmapping (e.g. for private memory managed by TDX it may need call into SEAM-MODULE). Signed-off-by: Yu Zhang Signed-off-by: Chao Peng --- include/linux/kvm_host.h | 8 ++++- virt/kvm/kvm_main.c | 69 +++++++++++++++++++++++++++++++--------- virt/kvm/memfd.c | 2 ++ 3 files changed, 63 insertions(+), 16 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 07863ff855cd..be567925831b 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -233,7 +233,7 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIER +#if defined(KVM_ARCH_WANT_MMU_NOTIFIER) || defined(CONFIG_MEMFD_OPS) struct kvm_gfn_range { struct kvm_memory_slot *slot; gfn_t start; @@ -2012,4 +2012,10 @@ static inline void kvm_handle_signal_exit(struct kvm_vcpu *vcpu) /* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +#ifdef CONFIG_MEMFD_OPS +int kvm_memfd_invalidate_range(struct kvm *kvm, struct inode *inode, + unsigned long start, unsigned long end); +#endif /* CONFIG_MEMFD_OPS */ + + #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 7b7530b1ea1e..f495c1a313bd 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -468,7 +468,8 @@ void kvm_destroy_vcpus(struct kvm *kvm) } EXPORT_SYMBOL_GPL(kvm_destroy_vcpus); -#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER) +#if defined(CONFIG_MEMFD_OPS) ||\ + (defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)) typedef bool (*gfn_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range); @@ -595,6 +596,30 @@ static __always_inline int __kvm_handle_useraddr_range(struct kvm *kvm, /* The notifiers are averse to booleans. :-( */ return (int)ret; } + +static void mn_active_invalidate_count_inc(struct kvm *kvm) +{ + spin_lock(&kvm->mn_invalidate_lock); + kvm->mn_active_invalidate_count++; + spin_unlock(&kvm->mn_invalidate_lock); + +} + +static void mn_active_invalidate_count_dec(struct kvm *kvm) +{ + bool wake; + + spin_lock(&kvm->mn_invalidate_lock); + wake = (--kvm->mn_active_invalidate_count == 0); + spin_unlock(&kvm->mn_invalidate_lock); + + /* + * There can only be one waiter, since the wait happens under + * slots_lock. + */ + if (wake) + rcuwait_wake_up(&kvm->mn_memslots_update_rcuwait); +} #endif #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER) @@ -732,9 +757,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, * * Pairs with the decrement in range_end(). */ - spin_lock(&kvm->mn_invalidate_lock); - kvm->mn_active_invalidate_count++; - spin_unlock(&kvm->mn_invalidate_lock); + mn_active_invalidate_count_inc(kvm); __kvm_handle_useraddr_range(kvm, &useraddr_range); @@ -773,21 +796,11 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn, .may_block = mmu_notifier_range_blockable(range), .inode = NULL, }; - bool wake; __kvm_handle_useraddr_range(kvm, &useraddr_range); /* Pairs with the increment in range_start(). */ - spin_lock(&kvm->mn_invalidate_lock); - wake = (--kvm->mn_active_invalidate_count == 0); - spin_unlock(&kvm->mn_invalidate_lock); - - /* - * There can only be one waiter, since the wait happens under - * slots_lock. - */ - if (wake) - rcuwait_wake_up(&kvm->mn_memslots_update_rcuwait); + mn_active_invalidate_count_dec(kvm); BUG_ON(kvm->mmu_notifier_count < 0); } @@ -872,6 +885,32 @@ static int kvm_init_mmu_notifier(struct kvm *kvm) #endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */ +#ifdef CONFIG_MEMFD_OPS +int kvm_memfd_invalidate_range(struct kvm *kvm, struct inode *inode, + unsigned long start, unsigned long end) +{ + int ret; + const struct kvm_useraddr_range useraddr_range = { + .start = start, + .end = end, + .pte = __pte(0), + .handler = kvm_unmap_gfn_range, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = true, + .may_block = false, + .inode = inode, + }; + + + /* Prevent memslot modification */ + mn_active_invalidate_count_inc(kvm); + ret = __kvm_handle_useraddr_range(kvm, &useraddr_range); + mn_active_invalidate_count_dec(kvm); + + return ret; +} +#endif /* CONFIG_MEMFD_OPS */ + #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER static int kvm_pm_notifier_call(struct notifier_block *bl, unsigned long state, diff --git a/virt/kvm/memfd.c b/virt/kvm/memfd.c index 662393a76782..547f65f5a187 100644 --- a/virt/kvm/memfd.c +++ b/virt/kvm/memfd.c @@ -16,6 +16,8 @@ static const struct memfd_pfn_ops *memfd_ops; static void memfd_invalidate_page_range(struct inode *inode, void *owner, pgoff_t start, pgoff_t end) { + kvm_memfd_invalidate_range(owner, inode, start >> PAGE_SHIFT, + end >> PAGE_SHIFT); } static void memfd_fallocate(struct inode *inode, void *owner,