From patchwork Wed May 29 18:05:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13679366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B313C25B7C for ; Wed, 29 May 2024 18:05:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A90BE6B00A1; Wed, 29 May 2024 14:05:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3F6B6B00A3; Wed, 29 May 2024 14:05:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B9796B00A4; Wed, 29 May 2024 14:05:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6B7436B00A1 for ; Wed, 29 May 2024 14:05:23 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1F1EA120C41 for ; Wed, 29 May 2024 18:05:23 +0000 (UTC) X-FDA: 82172210526.16.C48C1B7 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf15.hostedemail.com (Postfix) with ESMTP id 21C0FA000D for ; Wed, 29 May 2024 18:05:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="hx2/clWL"; spf=pass (imf15.hostedemail.com: domain of 3YG5XZgoKCNM8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3YG5XZgoKCNM8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717005921; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9IVaEi5zQfJ05wKpIy/u5NLNm+GzTdT+Fqa7TW9318s=; b=DvEghJMfdGUKA6YJ1oi5vv5vIkfrFY9P30rxKI0ije82+chSttpTfkFh043t1ddMoOK6WA t+KQTW1NxCYEKZMF0I8YyHXiftnnyLX53sTKJwHtIpTBVs3ICJR1mvoY/ajBtQjunJgG+8 3Gf/CMpTMnvYSOxFtjAhUxKHAkUmPTo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717005921; a=rsa-sha256; cv=none; b=Kxkgf8FvKvGj9gdXJpo5GT+xVTELP+yJaWu5fOQQXLMMEKhQMVm/NkbKGpMNCf/ijrUsxA KgAQzlV/TTlXx0sBqpzxWk+gT+HbHPqejhaRZVdKEaXBLBRp2FsFKIieK8uPsCTve9T7KE wZS8QXTBPpyV7pbfRYANonl8kvs4vqk= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="hx2/clWL"; spf=pass (imf15.hostedemail.com: domain of 3YG5XZgoKCNM8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3YG5XZgoKCNM8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-7fc2273536bso21341241.2 for ; Wed, 29 May 2024 11:05:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717005920; x=1717610720; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9IVaEi5zQfJ05wKpIy/u5NLNm+GzTdT+Fqa7TW9318s=; b=hx2/clWLr4bAQw+U9EiKdxcQJ4hD9gqrOMukcOfUMfZBxf2GJdw6pzscurzacSbA6o OigPpJBUd56hZn9/GjmJaTRUJQMwBaosixN6yrWuYZrECm0YEEyS6NKvF9efSO+Z0puw aS1PEA/ab3qC5Qbg1XhxR7WKfYsAVjRgZ29cpr7p6y8Utw7KhAs1mNOFIZDTh8SsyoG3 dD9C4ICyBLRJR8pS8Q+/ekxNFyykrztYnM0msnDAFeJllUJw8Z4YI9JrZE2iKTvHz27u oL0NwB01PNoCy1MeC7xL8QiT2V5TdDTLLW8rXu4SC7Qc6v048twKhpRVR1wGdWXvHVmN FSIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717005920; x=1717610720; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9IVaEi5zQfJ05wKpIy/u5NLNm+GzTdT+Fqa7TW9318s=; b=YJpp5I6KtAaQP5p02he5xwMAtabp0YINXthY/xfGmQkCdx4Z+73RDVdksMfXJDwvoY DJ2wL5b3d2LgyQqhhM5igecoUCe+Iv4xQtPVeGNUWg01hv2/6qJn6ZySF0gtLkf6i1QO IOEZ0lcrPgmna7z7CTPChdodFDc7p5AdT4UdWBZL91Iitjd69ClECI4ksXQ/8NjWzzjf liIHrcLEAuXB28/mczcNzXoD67zjCAOviUZ1DGotQwkcUT9yZ3UM8PLrCz1NrUVj5HZR dSYAy1yHcmF5EmHuEynrMlnpUXJxw91t/CVJjyw+stZtnoITxhtM510JpgshkD4P9sp8 R89A== X-Forwarded-Encrypted: i=1; AJvYcCUAtIg91xNev7EVLS+LefGHi8WDvb1+IFRq1Hh+5bdOhrBpy44YxmVrpW+hivSMc06T6IOdYgJM75mBAI/oe47eu6U= X-Gm-Message-State: AOJu0YwL3ZneecSbzWHkYMDV/kSfE4Fm53MKj7mCDY8DI96B012JF8do hONkNeOv7kn2chi4R8UfK3iiN3CP3qp4uYwkgFtMu1vpCm94fLT4H0bS8ygam27npB6zHokUzUy 0Y3Ku52OGa9FINxPWIA== X-Google-Smtp-Source: AGHT+IHvckbiyzzys4cYuqrWT42OTG48lfJUuChD2cJY5KHOUyTU+EmIW59KYswmWzGhXx+WNvqLQXCKjQe35xSx X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6130:2289:b0:80a:5330:87c5 with SMTP id a1e0cc1a2514c-80a5330931fmr5289241.2.1717005920232; Wed, 29 May 2024 11:05:20 -0700 (PDT) Date: Wed, 29 May 2024 18:05:06 +0000 In-Reply-To: <20240529180510.2295118-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240529180510.2295118-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.1.288.g0e0cd299f1-goog Message-ID: <20240529180510.2295118-4-jthoughton@google.com> Subject: [PATCH v4 3/7] KVM: Add lockless memslot walk to KVM From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Oliver Upton , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev X-Rspam-User: X-Stat-Signature: mxxzyduh58o36thmpxhjskxh1euf9azk X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 21C0FA000D X-HE-Tag: 1717005920-371421 X-HE-Meta: U2FsdGVkX1+1s875Q3kS3X06L4OyDqC7B5fXISsAaRMM8Ds/9STGtDa8gI72KGTv9GAQsZkjXqcAwABPX43vAF3/rCV8alZyX0Y2vMQZn1Qd4OwMltXOKn487pqxIavLQCwno9Xr+9Mix5R+u9oJxCaip/Rm3cUX6ATxGHncagp9pHy8WiZt7FiIZtpuLTqnNqdPah71tcMKBnDX2/06vC6/f0h9zv4R8P7vDAfU9lDqCNPBLXEEJAMsVhFKnxPnmjQfM8tW/S28lHJfx8G71ZFlHGV+uf9VFPqw/5G0NwtmJw3bD+AzRJXm7wDvnRwBpi7ng0xxYql+RDMcd6J5TcJis9v1Mv2Qiz2UuXJu2faCayKuTXcUljOT+5i8VPTfaUlQC89wZHDpFkcjWcPehY2qWjYEjTl4Kxgvf35/IOYAnRScEm8pVu39j1CkQ0oCF0S8KJiEMNP4iX4yt6HyAcgJ2pTe/q1mg3nkOubAiHNVch8uGZL9i6o4dXSBCbG37YvY/QfuZ0fAqdwHLKN3L2+gfTBIfgbPPpHmYIy7D+ik7/ZNaLwS/A3NaheYTNAxzNxcwf5InMZq/am7J5ZwiORaF+v2ZL4NEgstD0QU+heBMzidFab6ACvXLO9eYJfXcYIWoAyGuq39gt2wBZNjoBEP4aQbxLjxnZoiFLLeit0BLaIz6kqqxrdYvySEJdfNt4QHe8JjAmPRnNnmux67AgaRWRLmdbRUHxSC+9GLOX9XM1AswBrNjFw/Gj9LJ6h5JRFRUj9USDOAzSi7a71B1wv8hl3bM7BChPX8z5cWCbpjj1K1aLLuK6rkzH0LWJs7j+6PeIkdtGAL+bEZJFxzhvC2Ghd2H9vEEcO9I2iTuUngCc9Md05Pfxk5yGnJfG1CrjU9PHlaSsWX4Y2L/V637uaQJjcKPLfvLRc+bSyy7TVy/tzDpKKYgtgmchlk4oHYcbCCSzyjT8ENfo1z6Ob 91kZT3TP efKWzBHcwe5rozR1CEQomcjhp/i3r1pLhX4KneqJvysiPkOyyC21Yo7tsI4BYvXplLHOz/Msp2qFvVYIxSlznDSEG40CNdF3IKJ+3Bc5UPvc4qDKoimYPFStcNtfXaz0abhP/2exQ2Kmivc8KeL9xfAIbRoLCx4PntzDMYcKTHN0hkX8Hh/1ygSPhTw2OgLqxaHx39lLjza5YbNOpJiQdo8TAoWq2bdbwS5oOkn12PU+L6L9/TP16Aq7ggOtMiwrxdxGzeraJ0oWhR9OYrfvfapgDAsbmY6X3XrQrGYwrZB1N5Y8k2NfAnKbG9FwmnAjdoNHfPdCJ8QgiP+2J2O2EUNuat3/Ifl5AN70IRdYIcdqG2TqOufPKjGv0l9COLF5x/LQEWpJPuhF3+/s8cbVFKsVnOw2HNQ9ev1Fj8q/6O2CrUY/jPfQ7dZVmRnciBj7Ghr56p/me2Ky9Q06ERx1COpQkZP55PEKap9UkMDZRyiVtxEHSQHxpuSn06lTfJkXzssJudlfjWnQZ7ZBUDmomRaXVM0aW8AMl/rxDG02ZNj/MuC5Xh+cxq4ual5/UnFoSzBMWwHAgbxZwrP/TkXRn6RIZ9sAY3FJW4jIlna9e3Sb089z3kXkBaFG2yVmPlE/KdBEthxUBoyuQqqQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing. The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++------------- 2 files changed, 26 insertions(+), 13 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 692c01e41a18..4d7c3e8632e6 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -266,6 +266,7 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 14841acb8b95..d197b6725cb3 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -558,6 +558,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -612,6 +613,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler))) return r; + /* on_lock will never be called for lockless walks */ + if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) + return r; + idx = srcu_read_lock(&kvm->srcu); for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { @@ -643,15 +648,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - break; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + break; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -660,7 +668,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, if (range->flush_on_ret && r.ret) kvm_flush_remote_tlbs(kvm); - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -686,10 +694,12 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, return __kvm_handle_hva_range(kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - gfn_handler_t handler) +static __always_inline int kvm_handle_hva_range_no_flush( + struct mmu_notifier *mn, + unsigned long start, + unsigned long end, + gfn_handler_t handler, + bool lockless) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -699,6 +709,7 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = lockless, }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -889,7 +900,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, * cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range_no_flush(mn, start, end, + kvm_age_gfn, false); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, @@ -899,7 +911,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, trace_kvm_test_age_hva(address); return kvm_handle_hva_range_no_flush(mn, address, address + 1, - kvm_test_age_gfn); + kvm_test_age_gfn, false); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,