From patchwork Tue Jun 11 00:21:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13692708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8165BC27C4F for ; Tue, 11 Jun 2024 00:22:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=hNr/uLQzft9hWVC2M0p44IjSNJxAIdX1Ks6q4Mnl9m4=; b=CRT2gSHldDIlwBCQVdhRgAREne O/V9VSa5KjBlac+uq09yGbEEVIPz+ckWs7ftEaTYgZQVbkkIMTfn5C2Vp3FM2ET/VavJenPZJMRZb hXlUZJ9aZx0fUmlG07Jbsn4JlGi4DKFMxS+/pSv8DHCL5iuI/JoodQN6my2ptlTl2sGGDuLA30zb5 BlftTUTp9XYvdHszSvEdoE/pyn9oS1CkEY9xYHIzp49b5Mt0m9WmKbscVXv2xMc2LvEAJM5NyJ1IE 3xdsDIaH71rCF02ZiGgRRU8uEKKbm0LNEID8eM8UtjnDAUFUpBet+a8vjv4di/FAugbQMOFsPf5Va KiJojuWA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sGpHR-00000006w2S-0bF3; Tue, 11 Jun 2024 00:22:29 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sGpH4-00000006vlH-1YjW for linux-arm-kernel@lists.infradead.org; Tue, 11 Jun 2024 00:22:07 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-62834d556feso95965257b3.3 for ; Mon, 10 Jun 2024 17:22:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065323; x=1718670123; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JyxqE8JQbjAibPP1L18H/WB5ae5upDraVd7kSoejAF4=; b=Wov5sLnt5apJob+gRhoZbEBOmb5TxZOEq8YgK+GkG/2YSHLHJ+JXQ0Y1kzs0f7FJ83 7+6AuVEu2L+EPW3ooAXS1Fozr3mH3/D0UohYHKefyaH3pS7khXoH/lSPk0HrzVo5oT1i 4YZB34wVhk3MwhbMcc/N0CTn7yksNtXi5STe3dL2j520QZO+gcfZyqXZwwfFwHOGizAl 46+SgNix/K29Zy28pullj5v8OZscNnWaQD5rSwkXCoge6taeQI21YZwYUveLUtTW0JPB hMOoXL+kWiKFspPyzm7QCYoNhvffX86SGfYYZrd49nUzKQiDptM3fSiAp3brjuJCQxCO ExVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065323; x=1718670123; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JyxqE8JQbjAibPP1L18H/WB5ae5upDraVd7kSoejAF4=; b=I0wlOB0ZC4hIz/GjjjY2QSclL3tbEHYE1K+yod93HObBiUBwr9AInkUN40Wi0d56QO SOd6dF4/XZIBBXcVjTLh/6WBLsroWxeDOv5R31gGtJ3+E7D69Y9FW7isfaCpCfliqADQ 2WK0U5uiCUUct2DbWDUsWbajZ7AjSST2wi59s0FGwjiLBiv0raaa05ydK1WPtO9SMrnS ORYtuzu4ROrVODcy4y1zrTeg1C2kyaUhngQbaf0HiRl8UZzILt+urOq6PMCD2dBsXEmW sm8dNbAHLkm9rhLqkJ+CmAiUrtw/Z/bIrp+AUYIWCHH0FbVezj+kaZ8r8eme9LlMNTY1 CEXw== X-Forwarded-Encrypted: i=1; AJvYcCVERNurw6E/UVAKHs7ey2Q3e9md4WKBcYrmD55I/6/+NwLyYTrPTBgiZYESXtrTp3qjodI0aL/GrST558J2jpIDFuFSDnh45GBBGY6Ogkxr6rRdgIc= X-Gm-Message-State: AOJu0YxTuYTSiJ7mkS9rm+CC7Mu/cTTmVwqDGveOtv95ZkfTrvtDpRol QWI9pWFNL+TYjvVtnF8QjRl46xdtg21mLDIoPbPRAWWPwIXi9DZH5rNIinPsu1Rb/oKZbVfH5mW lOVkxg5I1J2vLKImDIA== X-Google-Smtp-Source: AGHT+IE2AKcYsstInOqgudHKTQp6Jdm0LGHMNfnLB4IVzasoq6oW0Y69RVWgTVatpoFZnw0nHcf4J+vZvg80MMrE X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:6101:b0:62a:2a39:ccd9 with SMTP id 00721157ae682-62cd5663caemr27714337b3.6.1718065323356; Mon, 10 Jun 2024 17:22:03 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:41 +0000 In-Reply-To: <20240611002145.2078921-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-6-jthoughton@google.com> Subject: [PATCH v5 5/9] KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240610_172206_521911_030AC0E8 X-CRM114-Status: GOOD ( 16.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Provide the basics for allowing architectures to implement mmu_notifier_test_clear_young_fast_only(). Add CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER that architectures will set if they implement the fast-only notifier. kvm_fast_age_gfn and kvm_fast_test_age_gfn both need to support returning a tri-state state of: 1. fast && young, 2. fast && !young, 3. !fast This could be done by making gfn_handler_t return int, but that would mean a lot of churn. Instead, include a new kvm_mmu_notifier_arg 'bool *failed' for kvm_fast_{test,}_age_gfn to optionally use. Signed-off-by: James Houghton --- include/linux/kvm_host.h | 7 ++++++ include/trace/events/kvm.h | 22 ++++++++++++++++++ virt/kvm/Kconfig | 4 ++++ virt/kvm/kvm_main.c | 47 ++++++++++++++++++++++++++++++-------- 4 files changed, 71 insertions(+), 9 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 4d7c3e8632e6..e4efeba51222 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -258,6 +258,9 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg { unsigned long attributes; +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + bool *failed; +#endif }; struct kvm_gfn_range { @@ -271,7 +274,11 @@ struct kvm_gfn_range { bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER +bool kvm_fast_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_fast_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); #endif +#endif /* CONFIG_KVM_GENERIC_MMU_NOTIFIER */ enum { OUTSIDE_GUEST_MODE, diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 74e40d5d4af4..7ba6c35c2426 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -489,6 +489,28 @@ TRACE_EVENT(kvm_test_age_hva, TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) ); +TRACE_EVENT(kvm_fast_test_age_hva, + TP_PROTO(unsigned long start, unsigned long end, bool clear), + TP_ARGS(start, end, clear), + + TP_STRUCT__entry( + __field( unsigned long, start ) + __field( unsigned long, end ) + __field( bool, clear ) + ), + + TP_fast_assign( + __entry->start = start; + __entry->end = end; + __entry->clear = clear; + ), + + TP_printk("mmu notifier fast test age: hva: %#016lx -- %#016lx " + "clear: %d", + __entry->start, __entry->end, + __entry->clear) +); + #endif /* _TRACE_KVM_MAIN_H */ /* This part must be outside protection */ diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index 0404857c1702..77ac680af60c 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,10 @@ config KVM_GENERIC_MMU_NOTIFIER config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS bool +config HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + select KVM_GENERIC_MMU_NOTIFIER + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d8fa0d617f12..aa930a8b903f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -699,7 +699,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, unsigned long start, unsigned long end, - gfn_handler_t handler) + gfn_handler_t handler, + bool *failed) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -711,6 +712,7 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .may_block = false, .lockless = IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), + .arg.failed = failed, }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -901,7 +903,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, * cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn, NULL); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, @@ -911,9 +913,32 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, trace_kvm_test_age_hva(address); return kvm_handle_hva_range_no_flush(mn, address, address + 1, - kvm_test_age_gfn); + kvm_test_age_gfn, NULL); } +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER +static int kvm_mmu_notifier_test_clear_young_fast_only(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + gfn_handler_t handler; + bool failed = false, young; + + trace_kvm_fast_test_age_hva(start, end, clear); + + handler = clear ? kvm_fast_age_gfn : kvm_fast_test_age_gfn; + + young = kvm_handle_hva_range_no_flush(mn, start, end, handler, &failed); + + if (failed) + return MMU_NOTIFIER_FAST_FAILED; + + return young ? MMU_NOTIFIER_FAST_YOUNG : 0; +} +#endif + static void kvm_mmu_notifier_release(struct mmu_notifier *mn, struct mm_struct *mm) { @@ -926,12 +951,16 @@ static void kvm_mmu_notifier_release(struct mmu_notifier *mn, } static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { - .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, - .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, - .clear_flush_young = kvm_mmu_notifier_clear_flush_young, - .clear_young = kvm_mmu_notifier_clear_young, - .test_young = kvm_mmu_notifier_test_young, - .release = kvm_mmu_notifier_release, + .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, + .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, + .clear_flush_young = kvm_mmu_notifier_clear_flush_young, + .clear_young = kvm_mmu_notifier_clear_young, + .test_young = kvm_mmu_notifier_test_young, +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + .test_clear_young_fast_only = + kvm_mmu_notifier_test_clear_young_fast_only, +#endif + .release = kvm_mmu_notifier_release, }; static int kvm_init_mmu_notifier(struct kvm *kvm)