From patchwork Thu Sep 26 01:35:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEDCBCCF9EB for ; Thu, 26 Sep 2024 01:36:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AE616B00A5; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85D5E6B00A8; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 661D96B00C8; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4734B6B00A5 for ; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 09547120717 for ; Thu, 26 Sep 2024 01:35:36 +0000 (UTC) X-FDA: 82605172272.10.258289E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 4581D20006 for ; Thu, 26 Sep 2024 01:35:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bA+pQUzg; spf=pass (imf13.hostedemail.com: domain of 3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/c4iHe6I2dqFJ8L4xEL/wcuXYs+QQdS3mQ0rZKnPlRM=; b=ZNiF4SnEr4hCiBbNctiu2Zq21UZAy9KyBbA08ErHFfXN39QXg3n6xGrc3ybv0WkdMSE3QU v+sj+jebdxPbmC9dTKSMGOmzMag3SBpqBKVwQ1H9F2w0+4pVIwJJ548n1geGMFPXX55ZQa 0q3NIMFkBD8XWDfdTxiv9NpEaBhoeIU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bA+pQUzg; spf=pass (imf13.hostedemail.com: domain of 3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314473; a=rsa-sha256; cv=none; b=qz9I5u1pZE5P1HdzMWtkpxgXWcUGXJAMlHXmOU5Tvg8RZGPSB7u9zqpvwCQjOcfRdIit/I QrgeyvAigeXKey9Qc5/I9iHvfdlrKJIHWzzjDFsLd3vVn4lhudWmXA71E8x2ZOIcMTKqOP I0eISskcwUBBOiBkiMfsrlgXf9e/5Ak= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e02b5792baaso797092276.2 for ; Wed, 25 Sep 2024 18:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314533; x=1727919333; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/c4iHe6I2dqFJ8L4xEL/wcuXYs+QQdS3mQ0rZKnPlRM=; b=bA+pQUzgNSm2/Qa7NzLpKDnuviGD14U/8XvC8CsL2yHxL0mgenK2SNYFVemU6DyuhR E0P8YIAKuszhYK3uAXUD/IikJ0UvlxQP3+g51pffoKdp6EFWbZ885vcRvJg2FQHsCNQN CQtiib+/BLdTLclIk+Tuut91ycSCr1bxP0xDN68vkWFfiAx7HsSNFHn0iYnClHMbcuPr GnAX6oVPbDIC7fBpb4wfxT2pmHF1LNm2QT7tsuCuNJglzzVk7KGImZ0rDn4eBFMsBV/D idodOL0fzyBlg6JVGIgU89ckumr7JSn+sY60INUkYjenaLbQoerFGlOHVlt/3eSsiAxU J64w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314533; x=1727919333; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/c4iHe6I2dqFJ8L4xEL/wcuXYs+QQdS3mQ0rZKnPlRM=; b=BOyBBarGkH8ky4NRRdnbjGMojxZfIJft5r64tO58eNNKNZRXpkMmcou3XWA5mOn9sw p4Emw/hODKwPF+DPX1LXj7obbx8N5Ut6iji//SsK2r2vscC5ZUrSvgo3DBm4pfuvCn7l qOA6XXHEUig2E/Eif/t89+WmzcdUanO+HXU3gJUiU8+Q1yB4iX+t4iPGFaaf+KRFCMMA /P9ci3eDj9ogUwkYVcibL/gX1/AbwYOyfhAHJLCM0aVk5clKCyfBVhKI5Y3vLjvCItoh CrMeTeDUn/mS0BX5TPdJ/cN1Gz16fRgQbNCxj6fhLJTKt2cgsnzAtollr52exattR7MH 1PFQ== X-Forwarded-Encrypted: i=1; AJvYcCXwiD6pdHQVwJgLufTgZqGZbareqF2+xhmhp2iOhz8/cPMXim1sdyp+tXoyDoTxAkD/9nKTjS3bRg==@kvack.org X-Gm-Message-State: AOJu0YxRqjkqZWKZnDxpfkC5I6LTISnV4bvg56LknpOGjmjv1+Rhy3nT Wq+UKQiHGgYL7yiCEFPFTbH3blEXvoMcyypIAEzoGNgYM4d/roxHHtED7PHAo/g1FlpIm5EEbSu kCiHsZ39ggcS/l8+RWQ== X-Google-Smtp-Source: AGHT+IGK4m8FSya1x5h1yKPn0SlYZqL+0YnNSUJKqrhSqJhacVQdW7OQ9a4614HCOKTuoSQjtwwz7rbSxAQZQC1c X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a5b:a:0:b0:e25:cf7f:a065 with SMTP id 3f1490d57ef6-e25cf7fa247mr2793276.8.1727314533354; Wed, 25 Sep 2024 18:35:33 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:03 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-16-jthoughton@google.com> Subject: [PATCH v7 15/18] KVM: x86/mmu: Locklessly harvest access information from shadow MMU From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 4581D20006 X-Stat-Signature: ya4qcitj8gzazj6ibrryhpw8s88858be X-HE-Tag: 1727314534-324583 X-HE-Meta: U2FsdGVkX1+4QLOXFPl6No9IwJbIETj7and4w78vom8jt0TtR5x4geZbR8iVjR6A5ZfufQy1rYJLJ5wFdRoxd0xhfyQ9vftV7yqyNTZ/qhciZutNNePjNOUHdpwy5nhpUXGUKh7TvANhmRYLEeFHNN5m84TEdXZsK8LerJSXYsM9iWIQbHmL7rx1V4v4GP6TKD36dq+qB9QNN4aArNrjtwF539IX8vub6u0SqktiM6dnv8jSgdAyvqImoEaU+s/ZZyp6wZmoAzOwFeTXVvbyHVd9JcKh6dc1ajshXdQNjct2rZX3r7cJlLjFKUj75APEhdoVm+VW9/FhRmOHv2JBp3qeyZVUrz3H9x04qpdi3YQ5eMKrhpAG93OG4cYKcGggewZ20CO4aSfUWlc4wGlsA3oSYZ9lD2DUnMd5dlM8EtrlsylHqEnyBRMYqL4e9U1/y32P633pDMyvn5rpFZG3PkDqqrYd8BWorKFEypt0/KIyoOWEKtUJYLdkFPBgP63JSmaEFhBruj1hHs7ya2k410l3VczOJ9YnWXmLJTH2KrynXlg76u0WPM9n4xJjKG560hCkWcohkZekKGPMCmu06aYuwyGVdEbgL6pRNrB4X5fxv+5Dlf7d/mP2RPMB48TsfytJJ3KoOV/afzo/v3Agza1Ffwk8fl/UiSF1kNV764HqpTCj1DsJNL5QF+DBu5NFdafx3Wn10pBmwR0z43FhR/0zNEd4mPp9QCstnkylkYCORmcVnTm+qMCb2ofbTfTG/AvNz3cu6z4qZpynmsqRYoVzI+4HT2B6aBOLlHHsgH/28Hsvp/vNFoCt6xxuLr9VD9TEJtnFNhummkTapvFnYGuII7333uRhhV8QfXIMnJRtMHZYttqU61cWZ362mcvnaObKG9N7RJLvJzuL0TZmZgE9b/ZtnlKiHFg4i3nvcC+Wi6TsBzkpPFChoZPN/PD6C7Zoj7ZARhEFugU8Ly5 MJu0N/jc DboyYYL3RfiE5inYg7+6zFt2itZKCZHjocSVYGBwM2akdMKWHTO1TWgpzVmxrSyulJv9URO8IGhdV4reSSsNzx2wk9WXA7Rskq3YgZYxdk0fkTUecOjbRSQD1fHKcXdcQOhDdMH3vFbFnDO2UfKtdjECnnm/DMqoAiT7RSz+B3ZntpXyfCLOQE2uW47R1Hvaf0/X8mikAX7PSHqFaGqNVMAojsELBPMavrGYgdXo1FKdQSrr5v5FwXPvEKamTthT510z2v6ddGk8ppwbehAfkEmhJZRkDOZNwMpc2AtnfdpvlTrECxiVIFYwjWgqJq4vzwRjxCxKawgaybpTq4/KOhjPHxKzMJCh7vj4BwLsxQdwUCjx812sYbMlztRq0OSMJlVXmv0/1f4iLDaDD888YH02rTgRl3LK3eG+qptnEyDpzlyvrqEw7TMbG6PNDmhQqxH3T6j6hii+uvDPrtAF5BXhUVkr80aTcN7JSnhgZXd19oLhwa1cLFDWyvhxVAQPjUtAHMEWss4HbV7u86NJjFP9sTffqBbCU3+6PCPu38DbQfq7Nba44qy67CjBcP+9qS73R/nei3sXwQdU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move where the lock is taken for the shadow MMU case to only take the lock when !range->arg.fast_only (i.e., for the non-fast_only aging MMU notifiers). Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a63497bbcc61..f47bd88b55e3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1822,16 +1822,24 @@ static bool __kvm_rmap_age_gfn_range(struct kvm *kvm, static bool kvm_rmap_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range, bool test_only) { + bool young; + /* * We can always locklessly test if an spte is young. Because marking * non-A/D sptes for access tracking without holding the mmu_lock is * not currently supported, we cannot always locklessly clear. + * + * For fast_only, we must not take the mmu_lock, so locklessly age in + * that case even though we will not be able to clear the age for + * non-A/D sptes. */ - if (test_only) + if (test_only || range->arg.fast_only) return kvm_rmap_age_gfn_range_lockless(kvm, range, test_only); - lockdep_assert_held_write(&kvm->mmu_lock); - return __kvm_rmap_age_gfn_range(kvm, range, test_only); + write_lock(&kvm->mmu_lock); + young = __kvm_rmap_age_gfn_range(kvm, range, test_only); + write_unlock(&kvm->mmu_lock); + return young; } static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) @@ -1846,11 +1854,8 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) if (tdp_mmu_enabled) young = kvm_tdp_mmu_age_gfn_range(kvm, range); - if (kvm_has_shadow_mmu_sptes(kvm)) { - write_lock(&kvm->mmu_lock); + if (kvm_has_shadow_mmu_sptes(kvm)) young |= kvm_rmap_age_gfn_range(kvm, range, false); - write_unlock(&kvm->mmu_lock); - } return young; } @@ -1862,11 +1867,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) if (tdp_mmu_enabled) young = kvm_tdp_mmu_test_age_gfn(kvm, range); - if (!young && kvm_has_shadow_mmu_sptes(kvm)) { - write_lock(&kvm->mmu_lock); + if (young) + return young; + + if (kvm_has_shadow_mmu_sptes(kvm)) young |= kvm_rmap_age_gfn_range(kvm, range, true); - write_unlock(&kvm->mmu_lock); - } return young; }