From patchwork Wed Jul 24 01:10:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13740525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 273C8C3DA63 for ; Wed, 24 Jul 2024 01:12:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cbTYml8WJPEGh8qSfIdK1VudaJalVARhAdPc06J+ALw=; b=qXq45A6ghoN9Z0cMXFPVnQ7M+Z XW8yPrDPrEoBOGex+J0VGksluz6LJcYfsEvaZRyNq7dWp/fYUona0NHQQJ45uPMW+Lhp6xeeK8qUD xx8MS6x8L/J+xbIH0NX3X0rTEyubPUxh00fq3f+tmXPmoxuHT4xTwxLujrrMyOrM6F13jVW0k31jz Jy1ubP+SabHy3eH7Bp63ICQXPr5A61g1JOaHuGUcWWz6oGk7piFVGHDkj33dfqBytmSaulkHSl8W5 Z3ni0/3FIpWYaXahJYCLL2S3D4d1y2xS7TJPfOQPaL9EyRK31ihWXPXXCfPeZJTHFVSM+iO1mi4cI K5RxmKPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sWQYN-0000000Dxim-3fcF; Wed, 24 Jul 2024 01:12:27 +0000 Received: from mail-vs1-xe49.google.com ([2607:f8b0:4864:20::e49]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sWQXE-0000000Dx8K-37wo for linux-arm-kernel@lists.infradead.org; Wed, 24 Jul 2024 01:11:18 +0000 Received: by mail-vs1-xe49.google.com with SMTP id ada2fe7eead31-49296cd959dso1464849137.1 for ; Tue, 23 Jul 2024 18:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721783474; x=1722388274; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cbTYml8WJPEGh8qSfIdK1VudaJalVARhAdPc06J+ALw=; b=BbcaPBYq8LV5JCzVZm+EgQpAXuBTYuErXyFQqxTLUT06wvvhfqDuvnbOEKc3u9akSR 2URySOk8LG7RckhR8ysmVDe9bTe8xxVh2B59PYcIhMy9O99zKdOB457Z14a9a9HOr4QL yGzlh6ufTxCXP5lxMf4ZQipty0WwtXJdjAJVUijYl6y1PvQL6meAjgme/PC49RNVC8KI /H5N1u0OqNYKKAQ757/A1mT9e7llpJEStnlQrTcs4XPEOwwrdQHI2Vu2MiKJsM6MABQe +dctPDm3T6v4QQ+LgCY7DkApX/a6Lrk3lJQLDMzoOaKpDNcBE14F4RJB1u01QbZyunNG FqvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721783474; x=1722388274; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cbTYml8WJPEGh8qSfIdK1VudaJalVARhAdPc06J+ALw=; b=Ha/fHVw+OPzhCbXlbQRhxRMmcoVjEp93naPu4udABOnWSXRjXt1d63l9XVVcwVDVpk J1vMYix6Z8l074V9SErwjEUfVQNcDDOVwt18O/hP5E/UzIT8oA8tJVEIOePlRAK6ZnL9 VLNOLE0s1SxOtJ+pnSVRfVhCMu0NxRVDuSks+fhEicz2b9n4+e9KUpVix3G3v3xXQMzM MAVazQ9aqZlwsSGp8/Zn6Nupo7O2AofDw6xdfC/F3tQ7aIiT3YNazAhzjvX7laxtcJaZ 8+D2Oh631rE4jmHp3WxeZlaY5p0xYtesgBEH9AlMsMt2Fjr/jARlgG/8tz01K+7Ib+Xj yFMw== X-Forwarded-Encrypted: i=1; AJvYcCXT7zyLfRl1aDTIZmxwtF72gNpnTiKUYxHqKJz/xhd82MXI6BAq16VVe33ZqGcKIxsn34mD6bqujMAzWyWpcoJ0bsFgUcYtQh+xOWM5lI2jatelT1s= X-Gm-Message-State: AOJu0Yyw8ApjfkW+3KwhAY1M/GxPTiquYQbySXixQtRsmXzq7Z13PBoj fhN8IruKTpf/pG6KLJDwJ+XLPB7cMahxx/rhKmCatS9JVLsqc9VToSoEPIibJfQpjO4EcOWwNNt YrVOUBiE9l/t3p2oA8Q== X-Google-Smtp-Source: AGHT+IFU0dcKvF1oJPyuhM+g7ff3d/ALiIFFaW2tweOW7lf82u7TeCdfrHDEq+idZRCqJG0SfkAoO+hAK8jfDIhe X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:54a5:b0:492:a760:c94c with SMTP id ada2fe7eead31-493c199efb7mr42818137.4.1721783474197; Tue, 23 Jul 2024 18:11:14 -0700 (PDT) Date: Wed, 24 Jul 2024 01:10:26 +0000 In-Reply-To: <20240724011037.3671523-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240724011037.3671523-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240724011037.3671523-2-jthoughton@google.com> Subject: [PATCH v6 01/11] KVM: Add lockless memslot walk to KVM From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240723_181116_807635_F8757083 X-CRM114-Status: GOOD ( 16.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing. Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS. The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply. Suggested-by: Yu Zhao Signed-off-by: James Houghton Reviewed-by: David Matlack --- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 689e8be873a7..8cd80f969cff 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -266,6 +266,7 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index b14e14cdbfb9..632334861001 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..33f8997a5c29 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler))) return r; + /* on_lock will never be called for lockless walks */ + if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) + return r; + idx = srcu_read_lock(&kvm->srcu); for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { @@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - goto mmu_unlock; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); mmu_unlock: - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; return __kvm_handle_hva_range(kvm, &range).ret;