From patchwork Mon Apr 1 23:29:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76A99CD1288 for ; Mon, 1 Apr 2024 23:30:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12FE86B009A; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 090546B009B; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4B3E6B009C; Mon, 1 Apr 2024 19:30:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C4E1D6B009A for ; Mon, 1 Apr 2024 19:30:03 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 909751A01C3 for ; Mon, 1 Apr 2024 23:30:03 +0000 (UTC) X-FDA: 81962558286.30.1D4426B Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) by imf04.hostedemail.com (Postfix) with ESMTP id CBBC440002 for ; Mon, 1 Apr 2024 23:30:01 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Ms/iRFzr"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=PuuFLRErzlwF/IUg1ddMPVb0ueaCVfV6di174ggyvzrqQ8TmeaQQHtx3UYKwLfUPex2Ouu SzMVtrkxucqlFWrNHqEcT3CPgzgR0H3qF7UbJRkosrpNqdCprGvgs/cWX5pZODwCoyrCUW hVaXx7kpwAZ3O2Sa+WW+t2NA9/OyQr0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Ms/iRFzr"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014201; a=rsa-sha256; cv=none; b=JrRyRLKks7R66yDbkgxuVi641EvLC2KLK5QZ6UwEZ+exRgXDY4Z+wg3wWlyfAb+hYsJJqE jJh0A3ICfH632pT7K6ej1x2nHwLboSNWH7cixUx5RE2GSmgWpd5WDrT0xNf+NeIGwGOKEl 4Z561dvzMZ+MabTKtArRdOnCvSlQmr4= Received: by mail-ua1-f74.google.com with SMTP id a1e0cc1a2514c-7e03a706222so2050493241.3 for ; Mon, 01 Apr 2024 16:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014201; x=1712619001; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=Ms/iRFzrsqWVvFeDD99Z243gyY0mUoqfTdYXoypDXFxHXSk2VgfGv0sB4JrAVs2R8x n730B4Qu7PQbcS60kSuaaQN2s7lwa/s60H5xGEaa09F/ZSWEEqI58x/CRtj0yWC/JQBk zRytFRVEPzKw1EQe3zn+grsl0aNPb4lh5MpPCxPq/MO+6zru5Zxclqe79FW+sxQOm2Fe Mz5dUoAnNSBWqG42Unc76iSpbSL0ZHL9EabjbC7I3X5sY/s4E6aBLn/AF59Usf8rIJF5 IBOcR1iWMx42+CQWfW0sv29MnpkUlsB5YtSyeWgFODCAfHYcYmkWRIdfobUoqVST5gLa GxyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014201; x=1712619001; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=l+zJAcxrwBY2TUIKH6RtlLOZf8iIwAyTQPfHTfvNsGcEXspe+xFG76G0NdDL0160W8 Epext39/89zK6OxaseI2vMR8fLgJLkCbTcEx/A20pw13OvdE3pR+YGp/qak45/WQ0tDa MWkUAPGZxeZuq9GCK1he5dXVQLQKF5a9VSc8OPNGEIv9vd1kk7wsIjpCJPqS8etVYi7r DmBdheqTPt1WULWE7iIAe8cSxjoOV2BrT7XuQCZS9TeJPECq7IRFokKnotwdwcDxnjFP CUpc1BD7QHiYANXANW2YAJEweIXbq3FQ2hBFFbnqa9Un4phYECvSv5d7DoNdQKqyt1G1 sH3Q== X-Forwarded-Encrypted: i=1; AJvYcCVkUQFb5uAq1B1u77UkfLPfjxdvi3j3pPZo8LQSJ5UfQB2NHLAXHi7Cp2f70LcxCBlfz4DNBygOE6czcxjBek7RSPc= X-Gm-Message-State: AOJu0YyXaNTnbcObRydy6rhIOksFSO6uZ7Np6RZ9a2kt43mhi+1ecLa4 wzPptE8COOH/UccrzFNCC7TmBECDg1fFJtPSLr80b53yFCBafZqr/pKs+THgCZHC09BYYzW2uwH EKlT6kRtORiNHej26Xw== X-Google-Smtp-Source: AGHT+IHiAeh9QdQ8JIxLLHU+30Xn5sQK+tQdEgkzewt+NNIBZBtyidDAUXciX6U3q2UCp3x2qiBYdkguFZKdqu9L X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:ab0:3792:0:b0:7e3:297b:9df1 with SMTP id d18-20020ab03792000000b007e3297b9df1mr67902uav.1.1712014200934; Mon, 01 Apr 2024 16:30:00 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:44 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-6-jthoughton@google.com> Subject: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: CBBC440002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: n9iu9punrfaz7816hruw1e4ak5b98uex X-HE-Tag: 1712014201-402313 X-HE-Meta: U2FsdGVkX19Q/Ova7zkBIOpod2gLDR7sqKo7oH1V27bBzkDsquJ80qroTuDeGqzdofe+ucX3v+nOWUaIbgxTCE8sv8jOgBxXMiuWiOi8vPsGa7kGP5w/2Anu7D/M6JSPuFoE+nWEyy4T+KPOBDqRrrCMCWYrN9pW/r9n4nq4X3i+xD9KI2QeV2v5/7jpy5zQfgdw8yEXl6Gdn2j6UuuUKk1MiwN9p0w3ODtqnHpbZiRXysmUd0xjUkRFee4mcEnCagMLgEFFUNTXQ1mt5+R0Hdd1SZ5Y1A5Badorlek2bG5vQyJpDKqes1WwDag+n4IYV92RxSskm1uuB+aAAqy6fB9kcwpH/lkF9TaFz/HUgMsVMOROg5y4rbIRGo5h8sL/+txuXIgB3z0SD4+DUH1iSjCCJzvPlfe9GsD3jzogWWiOuZd42+Kk1VkyzDxxWAp9nhWz5Tdry/gWYDu2GYuMbMgCrhm6fVxJP65kDiQTHDgkVZoa5d0Vh3GHp2w9R44td57Zt0qNR7HUoH7NKvS2b2NQZgBEP1TR7GLYseSRHlvNy+RchzZvTdcsMZZaWVfyiZDws0m5u9KvJrqRO/t46kRl54ki+LfF5TQwAmxSeaVZBrdmiEOZ1MLAPQrmbdPGnCXIXCSp1Op+Xtl7RrUsgfivd4Ob+4APXRCoh1rtgNzwkyZ+q4e/R8hX9Sb79AKC7PrrqgMzhdue6pruC3j1QvtD4uKDKVm4juYrv/n5OilbEjbI/fe2vfU0ZTy1xZotbezpB42AnnhCTBi+6p0QjP6EeSd0ofo15GGaYtmom1GzwVwc2gVIe+EGYWFAgtGIo9RFyMY9rCP99JuP45T0iFEXELb3TyVHUZ2+ZAy68Y05+4+yvIpcMr9c75SDUM16cAeEdatuucCMjz5zq22BZqPitMqA4nTkTLD+52TGdReP5bKP6hZdlZKrW7JId+0bka3fdErjwQ497hMij+Y Pf9ixXLb ASdK5IMawXOWyZXMLhlFFrRMSCfUoxWMheEQSMhQ+z9i3krjYJXfCoI9ReuXUlLivpBa4/u0umrYJwryuN1C4qe6o99dzOOmKhWXRZvvmJTlcj6Gj3uqF9+FHR0xnTW+BfwSBMqvEqIQHFrvbUmONSxoKowsm/ZVu71N3UV+H7DQO9AwrUySs36LnCCcJw/HHP3+mWGX2HzweB4lB3kv5Ibf8UObEy/q0VzJvEdPOySxkT5UUXuHDQCsmW1BoIEW719Ga/EUVv2q0lWKlFAVTOZEOBZ/a2lTw9lNAzdRnJOMlkYzhYNJaKYHRHNCdlMkqFEr5rwtqXiuN2OYb0oj/UQOUPiYWvOoQZu6LxJ90vsKvt0qsl0IezUyChcxIuPVN1OfVc4P50EmvdjKMyfMOKB67tPg3VMMrY8BdUxhUWv4l3e+XGfUKg9xWn/mbUAyf3zMe+QI/8TvvhfChMLyhHoKhvvwgyxA99S5aoFyi0OSARrzYFjE32/c/hH4hbcPiQkWjMKY1hWpwb0ruG8VBowfHN+BWuTTUvNngXr4DIHdNo9hX3uDW9StxwpCX+JUYpnM9clJyjUKB/Gj/HXf1Io2j0Cxy8LWrWqGPX/sTuiNQjIS9iiWE9etCE1ddO1Tvwt2soXLXZw1GDBs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 3b58e2306621..c30918d0887e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); */ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + /* + * Indicate that we support bitmap-based aging when using the TDP MMU + * and the accessed bit is available in the TDP page tables. + * + * We have no other preparatory work to do here, so we do not need to + * redefine kvm_arch_finish_bitmap_age(). + */ + return IS_ENABLED(CONFIG_X86_64) && tdp_mmu_enabled + && shadow_accessed_mask; +} + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 992e651540e8..fae1a75750bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1687,8 +1693,14 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d078157e62aa..edea01bc145f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1217,6 +1217,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, if (!is_accessed_spte(iter->old_spte)) return false; + if (!kvm_gfn_should_age(range, iter->gfn)) + return false; + if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,7 +1253,12 @@ bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - return is_accessed_spte(iter->old_spte); + bool young = is_accessed_spte(iter->old_spte); + + if (young) + kvm_gfn_record_young(range, iter->gfn); + + return young; } bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)