From patchwork Thu Sep 26 01:34:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62554CCF9EB for ; Thu, 26 Sep 2024 01:35:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 380076B009C; Wed, 25 Sep 2024 21:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32A426B00C1; Wed, 25 Sep 2024 21:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1316D6B00C2; Wed, 25 Sep 2024 21:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E27236B009C for ; Wed, 25 Sep 2024 21:35:29 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A3D6740716 for ; Thu, 26 Sep 2024 01:35:29 +0000 (UTC) X-FDA: 82605171978.07.9E1214D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id E96C416000F for ; Thu, 26 Sep 2024 01:35:27 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cVk0OA25; spf=pass (imf08.hostedemail.com: domain of 3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/OnVwOE5c+4OTD3Iw61tcspb9ex9OyShqju36BWGlvg=; b=pqW9272RgntcOqKF2Z+M2hz/eZlc+YaUZvpQXY9OgPO6mm+/9li7zDTV+nMI1e0/It5GZD EC7K+U/qu9jQlZcYxmXRB06rFS4pdNhbO/NIAGc2R4oU7T4Z5Z4rNir3QOJAhSdZxSkAn0 A0KLeov+8mEbfADNCfih1W6q7fDsUH0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314408; a=rsa-sha256; cv=none; b=eIHlulFrIsFPFzIubKm+tZOhR/91O0jLo5lGyHcP4xQDR72FXJ7mKYuhJIakH/cRgA9gOc qUIerQ8nQVGElPgsB09Ic9Kq5nBOzxVx0ZLlYdeY1nr0DB1WACCukMOFCeMQtNP+D2nzm5 QC3xSTe4Vdds9xuj5VxQOZmJf+1273Q= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cVk0OA25; spf=pass (imf08.hostedemail.com: domain of 3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e25cfc1c8e3so103101276.1 for ; Wed, 25 Sep 2024 18:35:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314527; x=1727919327; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/OnVwOE5c+4OTD3Iw61tcspb9ex9OyShqju36BWGlvg=; b=cVk0OA25UoyVMWClZdLqpHnp4Tw2wwD36Ha9qPXOsmt8yu17CTRtm8H6qvlOBfJIqp Oo9C2pnBiIiOS28uxUVJiigGrZqpqSV+nYVO+7V7VKb6n0wUWOSixEehc/QbFBrLCm0R uePGJ1455Yk/xhaEbbGIgrERRIN7RCRnFUa0Afx1IFj/NBMcY2iRGJCXUOhKD1Ee1otR FzAAE/6Xka//HwbpDxTLLIUsoN0bsOdh+4+KZ6QY/fiTCR0v6K9OjxtEY4Mx/PaJ+cY0 hqQ83+cUseQfGT/fwXyG/Fr0dajoY7sNTiw3o/ejPFUOx071D0/jX1ucll90GZCF9RMO wiBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314527; x=1727919327; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/OnVwOE5c+4OTD3Iw61tcspb9ex9OyShqju36BWGlvg=; b=Q8QHPcKjj27wrhhYexcxBd/NIsrBsNW8ujE8VJSGI7+Lgqpu7mfg5rHNMNKG9vAlR9 0bzJEGBVMQKCsP7a47WtkLawLW1Jl6Fg3/n5kbpVrmAgheoyrv52sy4bMKYvqo2e7b4q +aKTRzQ/rIPZPj+j3+wxw5KyddadiX/HZN5TbJ38+Wwf4Vm8C2Uq76quJqHCErW8+ILk nu+m4yDeA2tOpSet2nXGXRceL7wUVUPdT7VVN/ib254AYL30CTVPYU2GIuOgCMyFsAC7 H9/1dfsBmlIVUXkULylu6NzudDrFGwLZuDVjciYVXN8cDZ/x56iGaorf05Qj+SIP7CK1 gqTQ== X-Forwarded-Encrypted: i=1; AJvYcCUKSpe3PLJ/gUVY5Au1npOnCHjjnHivwm02Dzd/knB34LGv0W4U3hnrHi1tek0RsBy1EfwQFIHrPQ==@kvack.org X-Gm-Message-State: AOJu0YxRiLSmSEw1UuLI9ffi3k9S8i1LLZDXQP8PXwfMYzapw244bdHL qRR7fowlatwPTzLjC9SCJN6sSCNepk4qfabrXu15YjfrXwYb28LsJlVhGS22rSmXJYgKX81Un5e Lu0YcSnLDpT7bGhwzOg== X-Google-Smtp-Source: AGHT+IFP6wbu5d5xFxibAPH1VEJb74N56TVg2vvH80n5yLYaLmdn9VnaHIq1SmzVQiXoW0zSRxLWmsPWaG4W/Pg1 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:74cc:0:b0:e25:cced:3e3f with SMTP id 3f1490d57ef6-e25cced3f41mr11837276.4.1727314527042; Wed, 25 Sep 2024 18:35:27 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:58 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-11-jthoughton@google.com> Subject: [PATCH v7 10/18] KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging gfns From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: E96C416000F X-Stat-Signature: 6jrdbs3idthp6ttf31iqrgdn4zxfi78x X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1727314527-1532 X-HE-Meta: U2FsdGVkX1/w25qc44ASM5cgckTt4qDTHp6ukImIBhkF7VygPb5LKLvkXn5ZVSk32sjxVKP+O+bN68Z3aHwxV+8ssoT3G2rkzEYd3xsa2cJ+ot8/LwYAadMK2VKcZzcU8HMlksBZepKjR40qh2HiMG3aTbTVaSkd0lMTUkqVJhYHesKEw+6IUjtIqEJ9L24ezPNzwXTDsqDdvGChCVTJ11OH+rLF2IJ2zQCmg0CaDlPmS7MkWnObzKSOoOUV9yYzDxrQX+t3RpIoRp9cBqsbrY+0e4trxjPYrm4ywK9NVJ4OtBFF56S6zfqL/7oCAo++x/jBWRIgvCB9xAuyrerpjSBw+JOYGjaXjmvGNHmLFAZc9Ig0VexYO2yk1AUGxLf8rR/vOpABG8bwK04n1466cLE09Hf4fWP482ead1FkFN5KSuUeWCg+F88pP6sPtyQsb/VIPNKFK0k+1OSo2ZysqHykYJ+6RBVTrklyvvzbF6ujWaJjoln3/qtoat7sJ1oxlXEhtf9y2RtOamDlYiXEw1qmjDn9UZq5iJGr+b9YBSXqPhWjtBVASgltlaS+AhqBmMwFpWd4Ki2qcMP4XP8sPAV8aRzo21rlglL8XwhHwdG1Jnab+MfwPc5Ycc2avAo30U1o8q8qJmvxMmYcdFqfunI0/489vIUzU48NWa0Ay9Rk9c4p+fcUYfU8zoRQcGkzggZF7E7QzVACE04bNA7NNus8s48qSoWn5FcSvyoz5UQjcxGDq2AusDbdxssF8lHf1pmyyizqlODqzFnCgynuctpUXQ7FGJmOvYGDUPTZURtTn6QZUqgPrVT6eGmnQhUJYtVgtjdmr5pFEGMwdT5wR0K2kEvFf15Gp+JPV3k07BcEfSPkLen4ZqBIWzQVc5hGAFJVnlWmpHoetUueBWe05LeHhofjwO3QhfgE4r4sjQ6Ewb7JlJNVbiXK6PLd2ip86TjsYQf+g9rSdMdeCK3 6jJbejFn IeuB5dWzJTo8rVYNj7JLr8whlpCyV0skPeEWQpiTkDFHvmg+VCuf7m7+UaQT2sSvn7xeH2yQkDPfYHDLci5NUSVdAer41S6pcEw301gyM1NyGFrvD8qeUAVfihIBxaF7k7Pd//DJJHit7X9bI5+jglGiUwu7eJXfgtX2p+Hc0bvEK17MWhjyWDETfQN4VXFC39zsWdZPaJ3K8IPmXsAR8csHkTZ3WIlwURf0u4NCPMoaJFPVcQIsrLqUP107vBJ9bBZTGsWR6IhK/eV99GvUEofT03/hxEIgdbVKyvqgZToUkwM2ks6yUAiirOyeyql9Z5RtEbo5tgN45R/s/YOfJVwWoQpWbPrcQcbDasz9SQNCo7X2rH18OKn6gGCIF89bezdxUIYFSecM3kH8CejWTpeJp0pIhx2dreDrYhDKafjp1gr/yJcRnxDYF9ydOa7JHtranaBh7CQfBoxQrda4n3hcedk1kL/PtjOw11Hk/sN4N9J1J1uOMgrYLWpCBlMPxIg1WaFnZ65HkQQ8GDkhS9k3l41V2N3A3FjOb4X4Yneef5jPr/6faqTxLmeOWxqjKVJJzsfeKbgVim1WsafxEi7HSZvrECmLcQyPQgokqymxmVh6fh4WYfmQd2zj9HDWm/aHKky78hhMfEu9HunBnLrD90g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Sean Christopherson Because an L1 KVM can disable A/D bits for its L2, even if kvm_ad_enabled() in L0, we cannot always locklessly age, as aging requires marking non-A/D sptes for access tracking, which is not supported locklessly yet. We can always gather age information locklessly though. Signed-off-by: Sean Christopherson [jthoughton: Added changelog, adjusted conditional] Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 66 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 72c682fa207a..a63497bbcc61 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1014,13 +1014,11 @@ static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) * locking is the same, but the caller is disallowed from modifying the rmap, * and so the unlock flow is a nop if the rmap is/was empty. */ -__maybe_unused static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_head) { return __kvm_rmap_lock(rmap_head); } -__maybe_unused static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, unsigned long old_val) { @@ -1736,8 +1734,53 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } -static bool kvm_rmap_age_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, bool test_only) +static bool kvm_rmap_age_gfn_range_lockless(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_rmap_head *rmap_head; + struct rmap_iterator iter; + unsigned long rmap_val; + bool young = false; + u64 *sptep; + gfn_t gfn; + int level; + u64 spte; + + for (level = PG_LEVEL_4K; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) { + for (gfn = range->start; gfn < range->end; + gfn += KVM_PAGES_PER_HPAGE(level)) { + rmap_head = gfn_to_rmap(gfn, level, range->slot); + rmap_val = kvm_rmap_lock_readonly(rmap_head); + + for_each_rmap_spte_lockless(rmap_head, &iter, sptep, spte) { + if (!is_accessed_spte(spte)) + continue; + + if (test_only) { + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + return true; + } + + /* + * Marking SPTEs for access tracking outside of + * mmu_lock is unsupported. Report the page as + * young, but otherwise leave it as-is. + */ + if (spte_ad_enabled(spte)) + clear_bit((ffs(shadow_accessed_mask) - 1), + (unsigned long *)sptep); + young = true; + } + + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + } + } + return young; +} + +static bool __kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) { struct slot_rmap_walk_iterator iterator; struct rmap_iterator iter; @@ -1776,6 +1819,21 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, return young; } +static bool kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) +{ + /* + * We can always locklessly test if an spte is young. Because marking + * non-A/D sptes for access tracking without holding the mmu_lock is + * not currently supported, we cannot always locklessly clear. + */ + if (test_only) + return kvm_rmap_age_gfn_range_lockless(kvm, range, test_only); + + lockdep_assert_held_write(&kvm->mmu_lock); + return __kvm_rmap_age_gfn_range(kvm, range, test_only); +} + static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages);