From patchwork Fri Oct 27 18:21:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13438930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CC3AC25B6F for ; Fri, 27 Oct 2023 18:22:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0EF18000A; Fri, 27 Oct 2023 14:22:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9664900002; Fri, 27 Oct 2023 14:22:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC1FF8000A; Fri, 27 Oct 2023 14:22:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A4220900002 for ; Fri, 27 Oct 2023 14:22:34 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7D22D140B02 for ; Fri, 27 Oct 2023 18:22:34 +0000 (UTC) X-FDA: 81392061828.12.3253DCE Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 9704EA0004 for ; Fri, 27 Oct 2023 18:22:32 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YHXXoRVw; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 35_87ZQYKCAg0mivrkowwotm.kwutqv25-uus3iks.wzo@flex--seanjc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=35_87ZQYKCAg0mivrkowwotm.kwutqv25-uus3iks.wzo@flex--seanjc.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698430952; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0ecI9u+NU7x+hIBqOXm0c4fKiWu/1/+75d/SeCcbF3U=; b=3kZM7tZhYpUF1NFPtdXiOAftKOKA77/5WFNjNgsfWvbWPkAPHgEVuf7SVs6mDYtT+BZgAR iAxG9kD34F/1p1GG+TMZ7VZlbJjkwT1JFAyt6yUSbaqdPzBJyx3kLgtDNVGoBVAtHCjF26 phrDaffM2NEgwNgdR4P0WAQ37zsWWj8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YHXXoRVw; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 35_87ZQYKCAg0mivrkowwotm.kwutqv25-uus3iks.wzo@flex--seanjc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=35_87ZQYKCAg0mivrkowwotm.kwutqv25-uus3iks.wzo@flex--seanjc.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698430952; a=rsa-sha256; cv=none; b=ileP1ubFkWifESprH6UYIzT0qgrX8ooUwxVRGY1y69Wf2Rbcd+OBiqZ3djc1T27Efz7k9s 3z2TKUEfonoebizbOxKMkUzJzSqcwWIVgZF0H/WLe78YQK26Ng+NFsagCGoQVYlw44fTKH sh16YhipKzSegoTmnj9D9Rzk/g7o2qA= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7b53a48e5so20845747b3.1 for ; Fri, 27 Oct 2023 11:22:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698430951; x=1699035751; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=0ecI9u+NU7x+hIBqOXm0c4fKiWu/1/+75d/SeCcbF3U=; b=YHXXoRVw2ttKo+YISB5qiv1Ls0vwOs/wlUw3bcBwlmqup5LgOFwayyvWcdML/fTx8P CLik0E8gtyjd6bkgqKsavz40dvlsoS2pEjDJODy69XcslBHhQNHDoQuwOjygAjoTylv5 yLtWJsUhyDmWo6Ml0SU5WIdxJXoWGPQDDKES5ZPlOPJ0fpVvbhYKctpuSWRMGhGsQEWn 5KIF8+kxkxzl6tqzf5Q+8rBCsEWTZxet9Oztnv1VLfvW4nDGNrcIdXgaQZPzGGiKvuEK TG4vo9/mNxaEqpgcTU9rnMYZSOJ4rlwecXRdoMz0U1vB7pjQ3uEpY9F+fjrLp4N0Rtrj 2BNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698430951; x=1699035751; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0ecI9u+NU7x+hIBqOXm0c4fKiWu/1/+75d/SeCcbF3U=; b=o3rr7j3heTGyRmafwGPywz7Arx35RrhJTp28STLklzxngdarewrmcZP0mAXsEFma2z Fvr2M2gLDD5Xkxs7piilNDpppMMPfGSc+Ze2Ov4eH7WWwJ6eW1qrDnldm+snLUVUOwnP +B82wA6wPfXKuVnFkDQ0D/59Jyx4sXob0orl5aautB9vjGAtbDNyJjFCHLma/ZKi5YKJ zxDfzFnClOSx57sXBOe2Yq39C2yMgiZyOiLrFwlm8SQgsGJu6VdpjzEvAnejgVX+kGDx PcF5UdFgDwGk5Gfrxl+xF4uzrWBrb9nP0RwTzXF9sKI8VIQahsZ2EFVKnVV6CKpSLleV 2s8A== X-Gm-Message-State: AOJu0YyYqxKM8DHiC4zn559cETSgsjEIABH63sv+f/VZttA3xdHg+E44 hHwhurop1cml1Yk2JlQk+s5Tvr3LDxw= X-Google-Smtp-Source: AGHT+IEVX0PyerXEbWK1UOBgKc7BZmLynzYtP989xU7q9rVAiqIPcpbzc0WQAlmV6uCIwfdxpJ3o/vng1sw= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a81:4858:0:b0:5a7:be3f:159f with SMTP id v85-20020a814858000000b005a7be3f159fmr70939ywa.5.1698430951577; Fri, 27 Oct 2023 11:22:31 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 27 Oct 2023 11:21:45 -0700 In-Reply-To: <20231027182217.3615211-1-seanjc@google.com> Mime-Version: 1.0 References: <20231027182217.3615211-1-seanjc@google.com> X-Mailer: git-send-email 2.42.0.820.g83a721a137-goog Message-ID: <20231027182217.3615211-4-seanjc@google.com> Subject: [PATCH v13 03/35] KVM: Use gfn instead of hva for mmu_notifier_retry From: Sean Christopherson To: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Sean Christopherson , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Xiaoyao Li , Xu Yilun , Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , David Matlack , Yu Zhang , Isaku Yamahata , " =?utf-8?q?Micka=C3=ABl_Sala?= =?utf-8?q?=C3=BCn?= " , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9704EA0004 X-Stat-Signature: amnx5goqbohzqruru5m4ks5b5g1e5boq X-Rspam-User: X-HE-Tag: 1698430952-657721 X-HE-Meta: U2FsdGVkX1+sD3U4OSZzxMjFOAvcLeu2NzU6GwlsC4Lmmofk2XBP9phtKAkfYcWUG06txeBIMrjxPj/AqhYNkvuknKtT28ENW1Vyi5VcsXKEF53QnF0hqMW6zh9OlrnLcyBFARQhcg61Ux12e7ZDZxHPxxKwsTIoVkeYkNhHrZ4q2ssJqrGMOvHJ2tkWRQiiVBu/uDLwmmAzzkLPfEsM3KmSwpUdHDaCcr/7HPKb+YGbVMG+GIct3rNX7zzp90fXzRz1SlJWzP4peoEBNiCU4GSiKBgq/ehGIdESUWth69Jli0Uk0fRcDN6LaOZEHsReBbKCxSuwDul3NrYuEQK7x2eYGJaAet+JT5xMixSw6IEtGNjmFH8pwEfFOQEtobExQikGLJUABjxBLnr8fMdtNiDYfhfVC6qcUOpBzvkVgKr33ob3uXtBctDa9p1sRKFwrnTSd/g3ZwFLWwgyST0utgoqvvKMzEjx+p3kLhFmuC2nNGXyUiybwhd7YkmdM2PQCtJVFIH5//jk45FheOyUVIWZRyzb3GRXLbLla/w6Os/Bwouzgd4NZhqE7FCCWdM5se7CXU7mDrjnaYKmc0Ui+26o4ifARJtsK1sWITynz6RftFsMARFb9cJh+IIWMeFd04V8Y6Dii8yIrpV9L6rzGRRhbzJ//NavBRBGEhkgjDPA4y/03CmthVJw5KZSf1hp0147U5JReFAIZkv4vWb3jQVHzkXmHBFkMpAoxHpdq5xMqlhB2w3wH2y0ygBSwfLJcMgVPitBa7wWvFCgw973pwYvDQsNhRn3Io1LQpexzwhxRxRMBdQKGcMlP1q1LG5sQBbhnNaYfjbY8qKsBMIoAcx1iUr6RFadgdLiKwTNOKrPFEnOw191EqKpb53y28aSjG+UX5YsskXN/UXLftUDrfeTQB1o/9BFLAf1CgUnrgC2TVlpMPxY09QybGKCs6xpXG7aMRb+4z31KyrMiLv LP7EtCpF LXiFKdNg+/ETIEXFFbXSSrlik30Y4VDOaA9b5MVIBN6+lT/1QyMw+YPuFx9k4WTFKgF/Wl58pD6a1zvGag0tnQkf4ylKKPcYiASnh8+1bliZacbPv3F2MfO6i4HAC99X2Pv6uoaKBVScZ1N2u3ftoiWZDW+TLCtajWUgxVUSq05Y0NqU2Hg77WCmq2opGmnyw8tHZVbo6l+nd1As4smlXueHQS2hCJD6jRsvGukuVjbM9ZhzJh1vIp5PXQqRGQ5sqt8PZkcf9oyYHfavYiResfNEQ3F16wLuJvxu9G0ZoXviPDtIlQGJgUQw+m5lrW7UVNm+pDs+ggPThuG//leaVEk0DIY1zQeGr1lIyHr6dcOsAyeevv5SY2Gx+KBD4AZp/FgLIXodO1CBEZ2gppKxq8GXi3/5XFX9EzVvsaAQ29rYC+eKvdifIIkXnH9MigSlRtDRXhdwXFe2ITNtoLAMy3g2DrRDuJ9FW4DsOcpN+7N3drd7Ir40KtYur6PJQUA+ttwsFeHAABs82gepyRGmFRc+pw30snogS8sn6r+lEn2SO0uUb/8LqUUK2jxq5OnyQXehAetAboC5TTgrYxrSdV52EJxoCkqFRGro5GJrgPiPKiUfxoQqDWykJzQGF5L2IhTeLI1hsWzcIiGN104O6DJhl7dNvTug/Pl8t1fyhFqtlBc2D01N+/gZsFTQMXxJ/LafRviDhrbjC0hqhwHdmGfhtpyM6fWoPdtUkLi5xAh2mFAjIwRcdVRuQqVADBBKOPr+8J9DkCtdjvbcIJdHWnYWGXKYTmWC79uO+0baO9AIkq2HUpWSq9uUN7gCZU0+uXWmyEqV9UWohA3C28yA607Mjgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chao Peng Currently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense. For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small. Suggested-by: Sean Christopherson Cc: Xu Yilun Signed-off-by: Chao Peng Reviewed-by: Fuad Tabba Tested-by: Fuad Tabba [sean: convert vmx_set_apic_access_page_addr() to gfn-based API] Signed-off-by: Sean Christopherson Reviewed-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 10 ++++++---- arch/x86/kvm/vmx/vmx.c | 11 +++++------ include/linux/kvm_host.h | 33 ++++++++++++++++++++------------ virt/kvm/kvm_main.c | 41 +++++++++++++++++++++++++++++++--------- 4 files changed, 64 insertions(+), 31 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f7901cb4d2fa..d33657d61d80 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3056,7 +3056,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) * * There are several ways to safely use this helper: * - * - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before + * - Check mmu_invalidate_retry_gfn() after grabbing the mapping level, before * consuming it. In this case, mmu_lock doesn't need to be held during the * lookup, but it does need to be held while checking the MMU notifier. * @@ -4358,7 +4358,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu, return true; return fault->slot && - mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva); + mmu_invalidate_retry_gfn(vcpu->kvm, fault->mmu_seq, fault->gfn); } static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) @@ -6245,7 +6245,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) write_lock(&kvm->mmu_lock); - kvm_mmu_invalidate_begin(kvm, 0, -1ul); + kvm_mmu_invalidate_begin(kvm); + + kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end); flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end); @@ -6255,7 +6257,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) if (flush) kvm_flush_remote_tlbs_range(kvm, gfn_start, gfn_end - gfn_start); - kvm_mmu_invalidate_end(kvm, 0, -1ul); + kvm_mmu_invalidate_end(kvm); write_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 72e3943f3693..6e502ba93141 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6757,10 +6757,10 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) return; /* - * Grab the memslot so that the hva lookup for the mmu_notifier retry - * is guaranteed to use the same memslot as the pfn lookup, i.e. rely - * on the pfn lookup's validation of the memslot to ensure a valid hva - * is used for the retry check. + * Explicitly grab the memslot using KVM's internal slot ID to ensure + * KVM doesn't unintentionally grab a userspace memslot. It _should_ + * be impossible for userspace to create a memslot for the APIC when + * APICv is enabled, but paranoia won't hurt in this case. */ slot = id_to_memslot(slots, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT); if (!slot || slot->flags & KVM_MEMSLOT_INVALID) @@ -6785,8 +6785,7 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) return; read_lock(&vcpu->kvm->mmu_lock); - if (mmu_invalidate_retry_hva(kvm, mmu_seq, - gfn_to_hva_memslot(slot, gfn))) { + if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) { kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu); read_unlock(&vcpu->kvm->mmu_lock); goto out; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index fb6c6109fdca..11d091688346 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -787,8 +787,8 @@ struct kvm { struct mmu_notifier mmu_notifier; unsigned long mmu_invalidate_seq; long mmu_invalidate_in_progress; - unsigned long mmu_invalidate_range_start; - unsigned long mmu_invalidate_range_end; + gfn_t mmu_invalidate_range_start; + gfn_t mmu_invalidate_range_end; #endif struct list_head devices; u64 manual_dirty_log_protect; @@ -1392,10 +1392,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc); void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc); #endif -void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, - unsigned long end); -void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start, - unsigned long end); +void kvm_mmu_invalidate_begin(struct kvm *kvm); +void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end); +void kvm_mmu_invalidate_end(struct kvm *kvm); long kvm_arch_dev_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg); @@ -1970,9 +1969,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq) return 0; } -static inline int mmu_invalidate_retry_hva(struct kvm *kvm, +static inline int mmu_invalidate_retry_gfn(struct kvm *kvm, unsigned long mmu_seq, - unsigned long hva) + gfn_t gfn) { lockdep_assert_held(&kvm->mmu_lock); /* @@ -1981,10 +1980,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm, * that might be being invalidated. Note that it may include some false * positives, due to shortcuts when handing concurrent invalidations. */ - if (unlikely(kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) - return 1; + if (unlikely(kvm->mmu_invalidate_in_progress)) { + /* + * Dropping mmu_lock after bumping mmu_invalidate_in_progress + * but before updating the range is a KVM bug. + */ + if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA || + kvm->mmu_invalidate_range_end == INVALID_GPA)) + return 1; + + if (gfn >= kvm->mmu_invalidate_range_start && + gfn < kvm->mmu_invalidate_range_end) + return 1; + } + if (kvm->mmu_invalidate_seq != mmu_seq) return 1; return 0; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 5a97e6c7d9c2..1a577a25de47 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -543,9 +543,7 @@ static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn) typedef bool (*gfn_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range); -typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start, - unsigned long end); - +typedef void (*on_lock_fn_t)(struct kvm *kvm); typedef void (*on_unlock_fn_t)(struct kvm *kvm); struct kvm_mmu_notifier_range { @@ -637,7 +635,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, locked = true; KVM_MMU_LOCK(kvm); if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm, range->start, range->end); + range->on_lock(kvm); + if (IS_KVM_NULL_FN(range->handler)) break; } @@ -742,16 +741,29 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, kvm_handle_hva_range(mn, address, address + 1, arg, kvm_change_spte_gfn); } -void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, - unsigned long end) +void kvm_mmu_invalidate_begin(struct kvm *kvm) { + lockdep_assert_held_write(&kvm->mmu_lock); /* * The count increase must become visible at unlock time as no * spte can be established without taking the mmu_lock and * count is also read inside the mmu_lock critical section. */ kvm->mmu_invalidate_in_progress++; + if (likely(kvm->mmu_invalidate_in_progress == 1)) { + kvm->mmu_invalidate_range_start = INVALID_GPA; + kvm->mmu_invalidate_range_end = INVALID_GPA; + } +} + +void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end) +{ + lockdep_assert_held_write(&kvm->mmu_lock); + + WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress); + + if (likely(kvm->mmu_invalidate_range_start == INVALID_GPA)) { kvm->mmu_invalidate_range_start = start; kvm->mmu_invalidate_range_end = end; } else { @@ -771,6 +783,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, } } +static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) +{ + kvm_mmu_invalidate_range_add(kvm, range->start, range->end); + return kvm_unmap_gfn_range(kvm, range); +} + static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *range) { @@ -778,7 +796,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, const struct kvm_mmu_notifier_range hva_range = { .start = range->start, .end = range->end, - .handler = kvm_unmap_gfn_range, + .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, .on_unlock = kvm_arch_guest_memory_reclaimed, .flush_on_ret = true, @@ -817,8 +835,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, return 0; } -void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start, - unsigned long end) +void kvm_mmu_invalidate_end(struct kvm *kvm) { /* * This sequence increase will notify the kvm page fault that @@ -834,6 +851,12 @@ void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start, */ kvm->mmu_invalidate_in_progress--; KVM_BUG_ON(kvm->mmu_invalidate_in_progress < 0, kvm); + + /* + * Assert that at least one range was added between start() and end(). + * Not adding a range isn't fatal, but it is a KVM bug. + */ + WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA); } static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,