From patchwork Wed Jun 21 17:49:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62990EB64D8 for ; Wed, 21 Jun 2023 17:50:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=iYaeFcPHCQHoMemBl3KO4TxhFWqv/SmUQNbD07szCAw=; b=G0IwnfGKyNZrMDcNmBq1yBxm81 Qo/V4CkdOpEB7Oj9hr6bHYGKMheY3SQUZv0/jfjoXBC1vQooAazwQgZ6O9Gljhb6zo/bJzGGtaUUX rVJu5mvRy5HE1bs6fE7lIVsVTmQqsCFVJya2P5i9bm3V5nMqWH++O1Aun7ODfKhPXXX9/yqxxDQvI 9DKMiC+fQOsUmBn1W3I4vDV2ucvJDSU3iIJH83LKDYUi5rtX6xoXJvm4AWl0m5O6avjhQZjEoM3/j pFinlmP/IWwUW+l3EmyLurjUD8+LB3FEWgFaViI5uKBA1stVpA8gV0f6lYW2Wq5IWxXYVCoKA/u9Z zkML75NQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yM-00FH0f-22; Wed, 21 Jun 2023 17:50:26 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yJ-00FGvt-08 for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:24 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-77b25d256aaso655256739f.0 for ; Wed, 21 Jun 2023 10:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369822; x=1689961822; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=USXt3yGC1UEnXzlOA0lpUH2cberOhCHcVj/G1UKw8uU=; b=htGRA8XsqkJrd7igMfp73ZnJI8ZnXqn5sVSOTVNEVIB4tUa7HjEtsq9K1TNwEWBQsZ iDOZEPs3hF4+SE1VlTpHs1e9075DFw2w1NxTW11a6VVlR85nhLPHPmsBM+hgg8Z+rb0v II0tNAKY+fHC9iIz/PJJ8ARzdhexhmsBz1G6CqF0pgf5j7kxhGxHA0upZGmk71oKGioD OzbKasi88kGaeo6pbuBIVPXGDNqhWuA59ev0wFwRhPndfE97GpTHxkduhySd6cPVassm Jct3UqX/i+W/8PKDj4IKUsYC7ZXrvkQQuxbtVHUvV3o1zpBmawLU9LNgOeMOXIwWtk7T NZvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369822; x=1689961822; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=USXt3yGC1UEnXzlOA0lpUH2cberOhCHcVj/G1UKw8uU=; b=lPBr1Ia/kXsiVUDPXohnDvDuuWyjHFIcsym1gMOuaUezI6KfDPpz+ihnmF8hCMBjY2 fHOADdkOisJifjDwgR+sv6c5XjdJXbGopoeffHOPkhDso1ME3IcNtXQP/y0RfLAyx1XP 6a3HZn5hCQX+kXY8vE/11IVUW2Os+uCG0wNHtT4CpYApGZbjy7ABVpH3MOdd/OBnFZpM zOyuJhs2IWpsZwffNxfLesmzbPnr5IZlJE5AlJE1NCamcZwOwdJTaY70UlnEnFKiBhVN F5j12tRmlZPR/wwpzr7F9LAvic3VdLnCGKaeygraNa9eXdIufkjdKfNVXS06XCKH3jb/ E6Zw== X-Gm-Message-State: AC+VfDz9KVN0hWHOYffSMRIhwX7OasYPghw+RggiVOmu1dx2SjZ0WdBp 8QGcyTDVkZKZQasEHuZSjtm7ciJSnI54 X-Google-Smtp-Source: ACHHUZ7UjdRIorpK912qJ80B+WNmvLoEWYcVQQqyYoZ178c0q9AvCI1tPLlFq83IvaiADYrIiHXzSYOljIBZ X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:95a3:0:b0:41d:9cf2:f41d with SMTP id b32-20020a0295a3000000b0041d9cf2f41dmr5841982jai.0.1687369821976; Wed, 21 Jun 2023 10:50:21 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:52 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-2-rananta@google.com> Subject: [RESEND PATCH v5 01/11] KVM: Rename kvm_arch_flush_remote_tlb() to kvm_arch_flush_remote_tlbs() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105023_077092_89DF0584 X-CRM114-Status: GOOD ( 10.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: David Matlack Rename kvm_arch_flush_remote_tlb() and the associated macro __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB to kvm_arch_flush_remote_tlbs() and __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS respectively. Making the name plural matches kvm_flush_remote_tlbs() and makes it more clear that this function can affect more than one remote TLB. No functional change intended. Signed-off-by: David Matlack Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/mips/include/asm/kvm_host.h | 4 ++-- arch/mips/kvm/mips.c | 2 +- arch/x86/include/asm/kvm_host.h | 4 ++-- include/linux/kvm_host.h | 4 ++-- virt/kvm/kvm_main.c | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/mips/include/asm/kvm_host.h b/arch/mips/include/asm/kvm_host.h index 957121a495f0b..1473cf923ee37 100644 --- a/arch/mips/include/asm/kvm_host.h +++ b/arch/mips/include/asm/kvm_host.h @@ -896,7 +896,7 @@ static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {} static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {} static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} -#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB -int kvm_arch_flush_remote_tlb(struct kvm *kvm); +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS +int kvm_arch_flush_remote_tlbs(struct kvm *kvm); #endif /* __MIPS_KVM_HOST_H__ */ diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 884be4ef99dc1..6e34bbe244462 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -981,7 +981,7 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) } -int kvm_arch_flush_remote_tlb(struct kvm *kvm) +int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { kvm_mips_callbacks->prepare_flush_shadow(kvm); return 1; diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index fb9d1f2d6136c..f6e3aa617d8b8 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1794,8 +1794,8 @@ static inline struct kvm *kvm_arch_alloc_vm(void) #define __KVM_HAVE_ARCH_VM_FREE void kvm_arch_free_vm(struct kvm *kvm); -#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB -static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm) +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS +static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { if (kvm_x86_ops.flush_remote_tlbs && !static_call(kvm_x86_flush_remote_tlbs)(kvm)) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 0e571e973bc28..95c3e364f24b4 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1477,8 +1477,8 @@ static inline void kvm_arch_free_vm(struct kvm *kvm) } #endif -#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB -static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm) +#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS +static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { return -ENOTSUPP; } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cb5c13eee1936..a475ff9ef156d 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -366,7 +366,7 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) * kvm_make_all_cpus_request() reads vcpu->mode. We reuse that * barrier here. */ - if (!kvm_arch_flush_remote_tlb(kvm) + if (!kvm_arch_flush_remote_tlbs(kvm) || kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH)) ++kvm->stat.generic.remote_tlb_flush; } From patchwork Wed Jun 21 17:49:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62AE0EB64D8 for ; Wed, 21 Jun 2023 17:50:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1HOgLesv0ohdjVyj/4QJa7g1zhfgk77P9jjjpdx/WLo=; b=tBtTIXM7xC8DyEfEoU2yXqcLpg aypYTbBFwGX9zQOJwryjEPwJGLcsv+hzgMfoZHWfMvp1WzIuwfCzQ9QFs9Tbgtc+RCyRr3GqWr8eh vPVY1YEWwoJTxDCJDZPEFTwS9fNC1y7iMA9d781Nxi95aWhpL6mc6VVysZsfPaYVqB4YVj34vR/aE lfwhS2MR8MHnuXodqcNwtZCPx2LbMtKGuJVYwICTxSnt8NLSchx3tQBku+L+wWnRUpks7HCc5BVS+ 1Pc5oOgcFqEDV3n1KfTOoBdOPfGlsP+KzvLFDrLBIbcmVH+nEwOd9RaknddVRlW++3Dnqdnogr11J MtMMZbkA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yZ-00FHCE-29; Wed, 21 Jun 2023 17:50:39 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yK-00FGwr-0v for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:27 +0000 Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-77e2416ef1bso617547839f.1 for ; Wed, 21 Jun 2023 10:50:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369823; x=1689961823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=G0J4nFC2Gn0XupCqfzQ5AeLw9Bc/hm7+Rw/GcGHeXZY=; b=uDTcZy6WksIxTQoNUyBmNmSYdCye+OOyjLtqei9t+gdrikwCGERe5gniH/1WPexqfV S3ddWaWSCcwkePrKdofhK7s43nwKPFXZAsn/aW7QKGEpHYiWe2/4Xk1ZLXtY0+16TJh1 UWtkWtZHnx6F5hFIZI+w20IyK4MPjSVDx0cbNCAXe0hWZhZsWXS5MqBrp+w+ZkHEg0R1 Mqugrmhm6iklvk4fQwfhCflTZK5fd430BIy9H2RKcUcUkfZPZuLzY09rEgZw6I29/jre 0RSo8O9aukpg0nX+yIAhYQC7NT4FBOjIaomVwoTT8v4hh7UcubGQH5ArCBf16Ebv1z4G 388g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369823; x=1689961823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=G0J4nFC2Gn0XupCqfzQ5AeLw9Bc/hm7+Rw/GcGHeXZY=; b=Rimk1AFAP1SHASSJ6o+sTURCaRfE3TnETUCN3gVEJwuwMGP0IJAVpxzdmeNix3tpA8 cNNsCZgHMaiFPte/8/VyGVOq7uY9wIKQsAU7fNNmbWKL6FB2K/0WdmHBHpr5hj33NX5g oCP8Pnt+3MPgU4wtEWIPWD/i7aIsFPWCF9dmAkVaH+ePCQLNwhRCu+67ZvgGOurMIAy+ wxVYVwoWxFWu7dDnNYKKnBxIp14qfLdmgXQUjWcIseFEM8P4TN6BLSl95kuOU/lZe6Of rl1qeZ8a6yZK+N6MfhtL3HTArE1wJWBg4MDjd1wr5Mr0OSStoyMzE25UlhZNCrRNOE8n F1tQ== X-Gm-Message-State: AC+VfDxZMyyg3p0aH5sIzjeobyGOLEz2XesWXqFSfiz2BbqEzHnHJ4N0 fwqIKqcb2y2DHtqCJYA2/ZFW3USON3Rn X-Google-Smtp-Source: ACHHUZ4gtwXMs9gC/Rt6qeSbz3evFOSdeD4CaQ1dSb+oll+qJkTJp5tA3BeUra9pqAn7WPF5b9IewHmx1wSw X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6602:2051:b0:777:b81a:2025 with SMTP id z17-20020a056602205100b00777b81a2025mr6317324iod.1.1687369823025; Wed, 21 Jun 2023 10:50:23 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:53 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-3-rananta@google.com> Subject: [RESEND PATCH v5 02/11] KVM: arm64: Use kvm_arch_flush_remote_tlbs() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Zenghui Yu X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105024_353222_067A4EA3 X-CRM114-Status: GOOD ( 12.58 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: David Matlack Use kvm_arch_flush_remote_tlbs() instead of CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL. The two mechanisms solve the same problem, allowing architecture-specific code to provide a non-IPI implementation of remote TLB flushing. Dropping CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL allows KVM to standardize all architectures on kvm_arch_flush_remote_tlbs() instead of maintaining two mechanisms. Opt to standardize on kvm_arch_flush_remote_tlbs() since it avoids duplicating the generic TLB stats across architectures that implement their own remote TLB flush. This adds an extra function call to the ARM64 kvm_flush_remote_tlbs() path, but that is a small cost in comparison to flushing remote TLBs. No functional change intended. Signed-off-by: David Matlack Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Zenghui Yu Acked-by: Oliver Upton Reviewed-by: Gavin Shan --- arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/Kconfig | 1 - arch/arm64/kvm/mmu.c | 6 +++--- virt/kvm/Kconfig | 3 --- virt/kvm/kvm_main.c | 2 -- 5 files changed, 6 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7e7e19ef6993e..81ab41b84f436 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1078,6 +1078,9 @@ int __init kvm_set_ipa_limit(void); #define __KVM_HAVE_ARCH_VM_ALLOC struct kvm *kvm_arch_alloc_vm(void); +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS +int kvm_arch_flush_remote_tlbs(struct kvm *kvm); + static inline bool kvm_vm_is_protected(struct kvm *kvm) { return false; diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index f531da6b362e9..6b730fcfee379 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -25,7 +25,6 @@ menuconfig KVM select MMU_NOTIFIER select PREEMPT_NOTIFIERS select HAVE_KVM_CPU_RELAX_INTERCEPT - select HAVE_KVM_ARCH_TLB_FLUSH_ALL select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_XFER_TO_GUEST_WORK diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 3b9d4d24c361a..d0a0d3dca9316 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -81,15 +81,15 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) } /** - * kvm_flush_remote_tlbs() - flush all VM TLB entries for v7/8 + * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8 * @kvm: pointer to kvm structure. * * Interface to HYP function to flush all VM TLB entries */ -void kvm_flush_remote_tlbs(struct kvm *kvm) +int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { - ++kvm->stat.generic.remote_tlb_flush_requests; kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + return 0; } static bool kvm_is_device_pfn(unsigned long pfn) diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index b74916de5183a..484d0873061ca 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -62,9 +62,6 @@ config HAVE_KVM_CPU_RELAX_INTERCEPT config KVM_VFIO bool -config HAVE_KVM_ARCH_TLB_FLUSH_ALL - bool - config HAVE_KVM_INVALID_WAKEUPS bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index a475ff9ef156d..600a985b86215 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -350,7 +350,6 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req) } EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request); -#ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL void kvm_flush_remote_tlbs(struct kvm *kvm) { ++kvm->stat.generic.remote_tlb_flush_requests; @@ -371,7 +370,6 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) ++kvm->stat.generic.remote_tlb_flush; } EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs); -#endif static void kvm_flush_shadow_all(struct kvm *kvm) { From patchwork Wed Jun 21 17:49:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB49DEB64D8 for ; Wed, 21 Jun 2023 17:50:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=2wznuZ/vg8vsN3epIKpZMFzHXPWREHMJFckODYhOfGs=; b=kuH4ROPtfPgxcAD92ss9mVXLZI 67HoByGdkXQ8c7h+P9uwSsQtL7YwZvb0y3OjUtZiW7Rx8ms5fp07gymdiCKTcwNvyRNQvuARKVQJd 0uF2rErllc+adIRbPlNjn3J0NcyBR8arsZpc636L9nCJPKouNLtqDUaJF6vISDK9H2ilB6UPjhlhL YWaccazj/2GOvA6ZSU+VXuyhIG1Pd7OUSHJtNCUzoeDpR/qoIXSTx6uXG8z/VB084IvjKxFwdedUu +UETDXkJcVOugY28563byNtGcQGo/ep7JTbLdCGIFXMT9shSz3iznMrtlgmw/Rv+2T93j5Pt2G8Kx LjAAJEig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yf-00FHHH-0r; Wed, 21 Jun 2023 17:50:45 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yL-00FGyB-2K for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:29 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-57059e6f9c7so121600757b3.0 for ; Wed, 21 Jun 2023 10:50:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369824; x=1689961824; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0kDWyMjxvoVHajO1/4l5o4Ja0nzeqDHBQgPru4Ca7PE=; b=W51l0QbnmbjxnmmCtuwRuWnx7YubpJlELQpz6rAW3Ho9+5KqkOCPFt4QfJFhF9nUon swbQ6/awYlUsaGqw5BY1VoyvwAIrabaj2ur+qqkzwJCB/NNcsB2TLmSfUM9bkTT6i+A4 zv/DhDYK49UtrL7Z1+rgptqaq9Q8uY3WCpPrvORTrQzEtf3ZKyR4CgzAaNVPIg8XfIOf xKKHIilvW0Hfi14ytSJqddkZcKN0LZ/Uu1mc6Uy2aICfHITnrDdzhnZMCpDigz4Ko99J zUxuW1o5ZQsYlxMSD8Fc/hkLFRfGl05LWqyGqFNitmFJAcX2vJ9sDvETccbp+2+pgksv Gh+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369824; x=1689961824; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0kDWyMjxvoVHajO1/4l5o4Ja0nzeqDHBQgPru4Ca7PE=; b=FvCEwcqHVUlApULQ8quYEac57ZMG2njpfo6St627tkRUXYgXgUInZEfaC1ILxfWT7G Pvwwu3Rj5rmV0ISSC+XzYoLr6xAp88lrCHpCvbzglXo/a5mrQawndAtG78fk8jX6f2La Rks4oS/g8Fvqycj/ZTnK/WPKw/lJfehbWI5lRXb95Jq03uWXvEssEjo+19vE5xa3Q0wC a+QMekZyAPcEZUH4hkwty9LKnXu81xgCeZDE01CjkmQi0JtndHQisG9Zu1Pb4iCOns0i bin3+NlOI6RadCxGLr6VtyRnmxwSk2ZRGk7ZVYQZ+XxO0fFFtqH6yHJI92kx6vPptLs3 VoUQ== X-Gm-Message-State: AC+VfDwoYfLIBkug8u7Ytnzc1vkqiyQMpPzn+wYdWXvE9aQwJLmlJD1Z ccTrvOGhK41Gwd+0MdOY6uSU7B4OyUIF X-Google-Smtp-Source: ACHHUZ5K+kZNE7AXUViYSzsrB/zmiZFidf9JUwCxGExwegY/ED1rp2aL7aai9sJtvuWZdoNkgZU1AlyUF8Hg X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a81:e545:0:b0:56c:e53d:ae90 with SMTP id c5-20020a81e545000000b0056ce53dae90mr7488843ywm.4.1687369823914; Wed, 21 Jun 2023 10:50:23 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:54 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-4-rananta@google.com> Subject: [RESEND PATCH v5 03/11] KVM: Allow range-based TLB invalidation from common code From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105025_773836_277F5F55 X-CRM114-Status: GOOD ( 13.25 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: David Matlack Make kvm_flush_remote_tlbs_range() visible in common code and create a default implementation that just invalidates the whole TLB. This paves the way for several future features/cleanups: - Introduction of range-based TLBI on ARM. - Eliminating kvm_arch_flush_remote_tlbs_memslot() - Moving the KVM/x86 TDP MMU to common code. No functional change intended. Signed-off-by: David Matlack Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 9 ++++----- arch/x86/kvm/mmu/mmu_internal.h | 3 --- include/linux/kvm_host.h | 9 +++++++++ virt/kvm/kvm_main.c | 13 +++++++++++++ 5 files changed, 29 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index f6e3aa617d8b8..5c4dc547c030c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1804,6 +1804,9 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) return -ENOTSUPP; } +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages); + #define kvm_arch_pmi_in_guest(vcpu) \ ((vcpu) && (vcpu)->arch.handling_intr_from_guest) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c8961f45e3b1c..32f7fb1b66c8d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -275,16 +275,15 @@ static inline bool kvm_available_flush_remote_tlbs_range(void) return kvm_x86_ops.flush_remote_tlbs_range; } -void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, - gfn_t nr_pages) +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages) { int ret = -EOPNOTSUPP; if (kvm_x86_ops.flush_remote_tlbs_range) ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn, - nr_pages); - if (ret) - kvm_flush_remote_tlbs(kvm); + pages); + + return ret; } static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index); diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index d39af5639ce97..86cb83bb34804 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -170,9 +170,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, struct kvm_memory_slot *slot, u64 gfn, int min_level); -void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, - gfn_t nr_pages); - /* Flush the given page (huge or not) of guest memory. */ static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level) { diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 95c3e364f24b4..a054f48498a8f 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1357,6 +1357,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target); void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode); void kvm_flush_remote_tlbs(struct kvm *kvm); +void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 pages); #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min); @@ -1484,6 +1485,14 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) } #endif +#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +static inline int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, + gfn_t gfn, u64 pages) +{ + return -EOPNOTSUPP; +} +#endif + #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA void kvm_arch_register_noncoherent_dma(struct kvm *kvm); void kvm_arch_unregister_noncoherent_dma(struct kvm *kvm); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 600a985b86215..fc4ee20d33cc0 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -371,6 +371,19 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) } EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs); +void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 pages) +{ + if (!kvm_arch_flush_remote_tlbs_range(kvm, gfn, pages)) + return; + + /* + * Fall back to a flushing entire TLBs if the architecture range-based + * TLB invalidation is unsupported or can't be performed for whatever + * reason. + */ + kvm_flush_remote_tlbs(kvm); +} + static void kvm_flush_shadow_all(struct kvm *kvm) { kvm_arch_flush_shadow_all(kvm); From patchwork Wed Jun 21 17:49:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 840EBEB64D8 for ; Wed, 21 Jun 2023 17:50:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uAfBn2ai0hqCngKm2Atz0wlIwNvWZwYjdosDL26bj+s=; b=AXqKVe5cWR4fVQJ+RwEdkO3mFB pL0qdNaHmjr6BhjmrVdq35BmuFnMd+681Vq2xO8ryDee9CgfL6PB9+50qLRXki0XVKsi7lpcQWAQE QqGwSzWsSUKP2cKy4Ec81NAfgQ/bimHjKf+p0gOvm31u+IMry7J8fMILih1SXQNtr3bBXtf9H9woD Qqk7AZFjpkew9bS1+EHM+fUmbwsRppANCwnJuq16ZtXHBoEO6gxHSvQdWKz7/IcE5oNsDocyfra86 OW6KkfwPSfuUE/KgvxTXLWviTi3/B0Z+jUetJ9+qqeTplI+cUEOsPWDfF/y3Syv0+xJk+DzLwMN1e RqnAcciQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yg-00FHJ4-2c; Wed, 21 Jun 2023 17:50:46 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yM-00FGyf-1A for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:30 +0000 Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-77e34503f1eso474964139f.0 for ; Wed, 21 Jun 2023 10:50:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369825; x=1689961825; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ztWgNOK7v5lnofLRY7X9WYYq0GAkm6RZjHrbLvfw42o=; b=sWoqHmfeLY6tnWxUG0IwhdrL+fj762X3tM56nJrP11HBbUnEqoQwbZ5ZpDAh822HJK HP0lTmDlo6XuinGit2v3WKzkusxyKJiy5JFgEYVQjCI+qRcZSOnDcDzLz6EtpX7Xy0nL o02g6No1w07ACOE6SQ5mrdBvJDju0Epw6lEDFnexL+GmmWzUuoA5LFD1LogF4MY4mGYs MMMpcdvlTEp9au4XWP3tS1UEK92t5G93PjNzjTyrEIBw/IOvSVkRoKLa5pKoqvvrAM7E cI0LI3KRII+V21EJOyKHWm2kOx3TQhjZTUVxblEwRl8gzGTNhWrK1356Z6i7tcoWWpBC ThhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369825; x=1689961825; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ztWgNOK7v5lnofLRY7X9WYYq0GAkm6RZjHrbLvfw42o=; b=QnNGthtzkuov89hkmTLmRJXOCFNZ3qEISdlh4Iq5j1KaU83KcyAWmK8tXoaVYnhLH9 cF4NRJYmd0Qy9t/cFDRZ+jCtbUms2RrPTTEqbzgnHPvprwZ/cJvosM1cw+Xi2k4IKTYF W0P2Q6KQO6bo5DZ99m4DT6QmVKAqKd8uBvsybh2xBjmB6yxREiqK5QXAxHEIIeF9vUpU 8JadrFBEIXPdNd6uJS9JWurhI9mpDSdqk+YUoj7YIhwL7Wg9VRLpVamZMkQ1EYGu+FU5 455GsSzc3SqYByGtUPxpn7lOhrKtuFKRaIEIxXHfAWPb7/WPm7/W5SzNcZD4o9QpYjTL Sb0A== X-Gm-Message-State: AC+VfDyMK+HVsQtOAOkRoYQPxekYqpioV7PS+ZhKCHPt4Fl1AWdQz08+ Z+hC13V3Wm3nHXLv+4EPJkEwiPVJem3x X-Google-Smtp-Source: ACHHUZ56hrw34YrnSchn/pE9EXcQ5pUHErzFlq5h9tirOYu/Yc7gt++2hbtJo1T9iGYQFdEcatuknOvOHcMs X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a5e:c012:0:b0:774:9732:324 with SMTP id u18-20020a5ec012000000b0077497320324mr6331365iol.2.1687369825026; Wed, 21 Jun 2023 10:50:25 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:55 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-5-rananta@google.com> Subject: [RESEND PATCH v5 04/11] KVM: Move kvm_arch_flush_remote_tlbs_memslot() to common code From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105026_432246_69BE0232 X-CRM114-Status: GOOD ( 17.66 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: David Matlack Move kvm_arch_flush_remote_tlbs_memslot() to common code and drop "arch_" from the name. kvm_arch_flush_remote_tlbs_memslot() is just a range-based TLB invalidation where the range is defined by the memslot. Now that kvm_flush_remote_tlbs_range() can be called from common code we can just use that and drop a bunch of duplicate code from the arch directories. Note this adds a lockdep assertion for slots_lock being held when calling kvm_flush_remote_tlbs_memslot(), which was previously only asserted on x86. MIPS has calls to kvm_flush_remote_tlbs_memslot(), but they all hold the slots_lock, so the lockdep assertion continues to hold true. Also drop the CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT ifdef gating kvm_flush_remote_tlbs_memslot(), since it is no longer necessary. Signed-off-by: David Matlack Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/arm64/kvm/arm.c | 6 ------ arch/mips/kvm/mips.c | 10 ++-------- arch/riscv/kvm/mmu.c | 6 ------ arch/x86/kvm/mmu/mmu.c | 16 +--------------- arch/x86/kvm/x86.c | 2 +- include/linux/kvm_host.h | 7 +++---- virt/kvm/kvm_main.c | 18 ++++++++++++++++-- 7 files changed, 23 insertions(+), 42 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 14391826241c8..9a75c23351aca 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1459,12 +1459,6 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - kvm_flush_remote_tlbs(kvm); -} - static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, struct kvm_arm_device_addr *dev_addr) { diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 6e34bbe244462..614cb7322b6d5 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -199,7 +199,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, /* Flush slot from GPA */ kvm_mips_flush_gpa_pt(kvm, slot->base_gfn, slot->base_gfn + slot->npages - 1); - kvm_arch_flush_remote_tlbs_memslot(kvm, slot); + kvm_flush_remote_tlbs_memslot(kvm, slot); spin_unlock(&kvm->mmu_lock); } @@ -235,7 +235,7 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, needs_flush = kvm_mips_mkclean_gpa_pt(kvm, new->base_gfn, new->base_gfn + new->npages - 1); if (needs_flush) - kvm_arch_flush_remote_tlbs_memslot(kvm, new); + kvm_flush_remote_tlbs_memslot(kvm, new); spin_unlock(&kvm->mmu_lock); } } @@ -987,12 +987,6 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm) return 1; } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - kvm_flush_remote_tlbs(kvm); -} - int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { int r; diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index f2eb47925806b..97e129620686c 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -406,12 +406,6 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) { } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - kvm_flush_remote_tlbs(kvm); -} - void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *free) { } diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 32f7fb1b66c8d..d8cfad72f2b1f 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6654,7 +6654,7 @@ static void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, */ if (walk_slot_rmaps(kvm, slot, kvm_mmu_zap_collapsible_spte, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL - 1, true)) - kvm_arch_flush_remote_tlbs_memslot(kvm, slot); + kvm_flush_remote_tlbs_memslot(kvm, slot); } void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, @@ -6673,20 +6673,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, } } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - /* - * All current use cases for flushing the TLBs for a specific memslot - * related to dirty logging, and many do the TLB flush out of mmu_lock. - * The interaction between the various operations on memslot must be - * serialized by slots_locks to ensure the TLB flush from one operation - * is observed by any other operation on the same memslot. - */ - lockdep_assert_held(&kvm->slots_lock); - kvm_flush_remote_tlbs_range(kvm, memslot->base_gfn, memslot->npages); -} - void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, const struct kvm_memory_slot *memslot) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index ceb7c5e9cf9e9..2ea17b51037af 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -12745,7 +12745,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * See is_writable_pte() for more details (the case involving * access-tracked SPTEs is particularly relevant). */ - kvm_arch_flush_remote_tlbs_memslot(kvm, new); + kvm_flush_remote_tlbs_memslot(kvm, new); } } diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index a054f48498a8f..1eec1119cc4c9 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1358,6 +1358,8 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode); void kvm_flush_remote_tlbs(struct kvm *kvm); void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 pages); +void kvm_flush_remote_tlbs_memslot(struct kvm *kvm, + const struct kvm_memory_slot *memslot); #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min); @@ -1386,10 +1388,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, unsigned long mask); void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot); -#ifdef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot); -#else /* !CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ +#ifndef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log); int kvm_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log, int *is_dirty, struct kvm_memory_slot **memslot); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index fc4ee20d33cc0..92218edfc6b84 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -384,6 +384,20 @@ void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 pages) kvm_flush_remote_tlbs(kvm); } +void kvm_flush_remote_tlbs_memslot(struct kvm *kvm, + const struct kvm_memory_slot *memslot) +{ + /* + * All current use cases for flushing the TLBs for a specific memslot + * related to dirty logging, and many do the TLB flush out of mmu_lock. + * The interaction between the various operations on memslot must be + * serialized by slots_locks to ensure the TLB flush from one operation + * is observed by any other operation on the same memslot. + */ + lockdep_assert_held(&kvm->slots_lock); + kvm_flush_remote_tlbs_range(kvm, memslot->base_gfn, memslot->npages); +} + static void kvm_flush_shadow_all(struct kvm *kvm) { kvm_arch_flush_shadow_all(kvm); @@ -2191,7 +2205,7 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log) } if (flush) - kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); + kvm_flush_remote_tlbs_memslot(kvm, memslot); if (copy_to_user(log->dirty_bitmap, dirty_bitmap_buffer, n)) return -EFAULT; @@ -2308,7 +2322,7 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm, KVM_MMU_UNLOCK(kvm); if (flush) - kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); + kvm_flush_remote_tlbs_memslot(kvm, memslot); return 0; } From patchwork Wed Jun 21 17:49:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4622BEB64D7 for ; Wed, 21 Jun 2023 17:50:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ILsET/hh8MbzN5t60yGKblMKnUnk43z0P5xg5ZsewVI=; b=f+0tJyiFoNL8jnlk9nSqPJ+WMX mdJlk1mVa/d7dqUINdICKsuGP5ZxDz10bGzuJrS4R0NnBokNjDsgNLAUQLQHZs7y23KG9RiNUOZ0B 58Diz41VnaIQ74fGQ95bntk2iitW301itukhH1goHiaach4VBw83uLkP/xIplM6y+HgAVCtTNTUjf vk4plLeonnccWY5PhGt4m+cKzinMSEoH+btP1cLZnVOdZ5UO+UNfeE4pAUjKjJkgFaOGc588iGC5Y 3dgfGpYEcM6QwOtzDo/epND/djQCQ0TnHHHWJK4mRemwBI3xMlwx9GUeBTd3L8Qbh6msbvYdqGfnW epq3/LkQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yk-00FHMv-2D; Wed, 21 Jun 2023 17:50:50 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yN-00FGzw-18 for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:32 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-77e2b91f817so455438239f.0 for ; Wed, 21 Jun 2023 10:50:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369826; x=1689961826; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PVNEdexdiDrABO6+9LuBxQhBbC4ryeNsSPli6bB3JDQ=; b=efabCYIR+2QwnoYwa1elPoJcKMg/OaBY/V+2YQUat5Du7vl0v2DN1fxpi9cu7zguZ6 SoIbgIpj4YUXCZQUBYmOQS0Ia78jXi41jheeR4ZE3hwkJinlPeYc46TjrFmjZWbEyrdV g4kh08S0BHT445pDPguNGJrRc45udjiqmfvkhihldsKnZT5gc42U6NAXCXCcfojCldu4 7BG5ZP4tgX6F779y2bEApRIkV6AfYPmkO6tNgnW3Z5DtdebHrsWtG7CWXUTNXqCDo56Z 7uHlb8TDxbdn2yW5CC5LYYU6n7owgjmH7w60H8sG6DkfB5kPev4BjfnTKuW5t01og/HT rcYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369826; x=1689961826; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PVNEdexdiDrABO6+9LuBxQhBbC4ryeNsSPli6bB3JDQ=; b=CXgT1U+Pkdb1wa6GQN5Kzz9FW0egXpz9ca+CtOhvCTFQTKyndf1vTYGwYVjmHCeYVK qmnC3Hwou5QytRbcSp6YbWkljXc2qzlMeLTqhS5BygwE6ti+QcmrATkpXdfdmOqd+vp2 pij25bDLhupUhD+dBxieyTUw1a8EpFhekR1EWWdzfyAvMzbrqI3K4ll4KriOD1AD25HW jKL+HHUPMZFawJEVnqtjQxd3rJIxuXzdifhrXWv17Y6sTqpiIrW0dGEq1HhKbW1zg8yA Ac0osYxgmycy1KRmfyBqB8n27PVTnhNnJ4s1vm7rJWhmOkUBr4VVAZakVknJyZ2n+grq kXXw== X-Gm-Message-State: AC+VfDwXhGPQZXDcLjlm1HJb5C5nRq0c2fRKyeoIJ3BaTzxJqNIcBpR0 v/Jq5mUEVJwvK8MjQJqjjO313gAy/3Nq X-Google-Smtp-Source: ACHHUZ64WC1ts17zqPnEgpIjzBVqL1TeNoFDZWnbctxTBLrlOg9GEHr9y74UvxOzXUcMwrzQ6uJl7o1ymveG X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:298b:0:b0:423:141e:f20b with SMTP id p133-20020a02298b000000b00423141ef20bmr5933346jap.2.1687369826004; Wed, 21 Jun 2023 10:50:26 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:56 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-6-rananta@google.com> Subject: [RESEND PATCH v5 05/11] arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Catalin Marinas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105027_432239_3822A936 X-CRM114-Status: GOOD ( 16.72 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI operation, such that other callers (KVM) can benefit. No functional changes intended. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Catalin Marinas Reviewed-by: Gavin Shan --- arch/arm64/include/asm/tlbflush.h | 108 +++++++++++++++--------------- 1 file changed, 55 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25d..4775378b6da1b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -278,14 +278,61 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLBI_OPS PTRS_PER_PTE +/* When the CPU does not support TLB range operations, flush the TLB + * entries one by one at the granularity of 'stride'. If the TLB + * range ops are supported, then: + * + * 1. If 'pages' is odd, flush the first page through non-range + * operations; + * + * 2. For remaining pages: the minimum range granularity is decided + * by 'scale', so multiple range TLBI operations may be required. + * Start from scale = 0, flush the corresponding number of pages + * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it + * until no pages left. + * + * Note that certain ranges can be represented by either num = 31 and + * scale or num = 0 and scale + 1. The loop below favours the latter + * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. + */ +#define __flush_tlb_range_op(op, start, pages, stride, \ + asid, tlb_level, tlbi_user) do { \ + int num = 0; \ + int scale = 0; \ + unsigned long addr; \ + \ + while (pages > 0) { \ + if (!system_supports_tlb_range() || \ + pages % 2 == 1) { \ + addr = __TLBI_VADDR(start, asid); \ + __tlbi_level(op, addr, tlb_level); \ + if (tlbi_user) \ + __tlbi_user_level(op, addr, tlb_level); \ + start += stride; \ + pages -= stride >> PAGE_SHIFT; \ + continue; \ + } \ + \ + num = __TLBI_RANGE_NUM(pages, scale); \ + if (num >= 0) { \ + addr = __TLBI_VADDR_RANGE(start, asid, scale, \ + num, tlb_level); \ + __tlbi(r##op, addr); \ + if (tlbi_user) \ + __tlbi_user(r##op, addr); \ + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \ + pages -= __TLBI_RANGE_PAGES(num, scale); \ + } \ + scale++; \ + } \ +} while (0) + static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) { - int num = 0; - int scale = 0; - unsigned long asid, addr, pages; + unsigned long asid, pages; start = round_down(start, stride); end = round_up(end, stride); @@ -307,56 +354,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); asid = ASID(vma->vm_mm); - /* - * When the CPU does not support TLB range operations, flush the TLB - * entries one by one at the granularity of 'stride'. If the TLB - * range ops are supported, then: - * - * 1. If 'pages' is odd, flush the first page through non-range - * operations; - * - * 2. For remaining pages: the minimum range granularity is decided - * by 'scale', so multiple range TLBI operations may be required. - * Start from scale = 0, flush the corresponding number of pages - * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it - * until no pages left. - * - * Note that certain ranges can be represented by either num = 31 and - * scale or num = 0 and scale + 1. The loop below favours the latter - * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. - */ - while (pages > 0) { - if (!system_supports_tlb_range() || - pages % 2 == 1) { - addr = __TLBI_VADDR(start, asid); - if (last_level) { - __tlbi_level(vale1is, addr, tlb_level); - __tlbi_user_level(vale1is, addr, tlb_level); - } else { - __tlbi_level(vae1is, addr, tlb_level); - __tlbi_user_level(vae1is, addr, tlb_level); - } - start += stride; - pages -= stride >> PAGE_SHIFT; - continue; - } - - num = __TLBI_RANGE_NUM(pages, scale); - if (num >= 0) { - addr = __TLBI_VADDR_RANGE(start, asid, scale, - num, tlb_level); - if (last_level) { - __tlbi(rvale1is, addr); - __tlbi_user(rvale1is, addr); - } else { - __tlbi(rvae1is, addr); - __tlbi_user(rvae1is, addr); - } - start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; - pages -= __TLBI_RANGE_PAGES(num, scale); - } - scale++; - } + if (last_level) + __flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true); + else + __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); + dsb(ish); } From patchwork Wed Jun 21 17:49:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C42FFC0015E for ; Wed, 21 Jun 2023 17:51:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=FdYNeNc6o6i1f3r8Zwz2DFlMk0scPGASAJcHR+zN/no=; b=EKDMcA7ew/V4SLBnLWDdDpOquV TfBz1XgMA8HmoX+BUW3WXKAnNPHFQ1AF/pKjo1INCy9PWrZ0wa3x03i4Zd2C5i9M0r9L943o0IJHO EiyvLZPOrM65lGC9bz+fFq56XSdOoXkBbldv6HrGckckThec0NpevIUDwZFbpj0oXICieAmWuKPB2 WwsTV86cuMyjeFlx/b9AdqIx53hWgiYQW6RzdnC6SsvVIc4aQiV/q19Quj4Y2z+MWEF5vz+suqc6O 5TbmxE4TmOHbH2m+WHYLHppboMrri5PvzchdWl0eRSpXzKY9PAv++kd8ow8SnGA0cKer7Fr2oNZRk hpOckZAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yq-00FHSn-0X; Wed, 21 Jun 2023 17:50:56 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yO-00FH1Z-0R for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:34 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-77ac4aa24eeso595060239f.1 for ; Wed, 21 Jun 2023 10:50:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369827; x=1689961827; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OXoixw7zylmWvJyeKW2Gpi+PHJj3UbBkieCTdC9BYvQ=; b=TEwOeOygtHoLGKC6cI4EIVXEGk/Jz9nxHQc1scZaPgYHN4dzGy5qyoq2VVaXDB98kb cH1aIyIem827OIJN+ai3rwTx/vyt2DPm8ci81Z/W7D9zu5jj1MdYTmXrY9kvg2ukS4zR O/qJDujh1WoEhxRiexQ/TvuFt3TvJ7sGZiHMqAvpkBlTGoZWlZ5jc7dw4nWsAWEfPdc3 WeH2Wmi8GDHZopWG9K3k0Rx/iyVsqAnH7Fr8dz9TVP/X015RqQN5ZFiPIJ3bRYPEvriV NklyUd+d28zLiEIWDMGobXUxLItjgHhc4o5GF7ectWmIjvyLSTjxwnQSR+xrb/Hceqxl 3RBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369827; x=1689961827; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OXoixw7zylmWvJyeKW2Gpi+PHJj3UbBkieCTdC9BYvQ=; b=Qmgdj8mtmmIw+dA/IJVKGmngyETD6PHSewnnqS0RjdGpEa3iD+Y381J4fUkGDqp1Kc RvQMpULNg6HmgPG6NozlCjIygYbLFGcY+xQezipJPfOLrSBEeVF8FeAs9Rb0KIn1KWsO k4JLbxTFLXgM7nYJ4ehx6DFzg8BltREDApNtiRQE38jkeT0OmC2rfsHmh5rgdQGlj4hG B1d01dX8Cfm8Xn3jmaBdRfX9g9XOVj6LH2jHduVsjWGve7/e1fEjtPZmjyx3qDTT1UvN Jxt9e1Qs9S7TVvPsFWB6RHTvRxubOBmhANVZ6cjwIKeXOC1Kqj/Pf+DwMdz1l+HXhy9v 0TJQ== X-Gm-Message-State: AC+VfDwRvhJUfQNmEKJyLR+VZOTAPFWm1rJYajTRgUr6/M8OLFlbeez1 tew8BB2QjjVpxKA75jbICbGmBZhaE8Nz X-Google-Smtp-Source: ACHHUZ4pwJFMGKjnQpE3Xww4pvTbbOvbAYARZGdpD/M4/4l01hfTpN3+c9+HxNWnwQqKOr2G9K49SFDkZ+i5 X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:a190:0:b0:41c:feba:4e8a with SMTP id n16-20020a02a190000000b0041cfeba4e8amr5942038jah.5.1687369827044; Wed, 21 Jun 2023 10:50:27 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:57 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-7-rananta@google.com> Subject: [RESEND PATCH v5 06/11] KVM: arm64: Implement __kvm_tlb_flush_vmid_range() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105028_214853_8A0C8015 X-CRM114-Status: GOOD ( 12.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/arm64/include/asm/kvm_asm.h | 3 +++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/tlb.c | 30 ++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/tlb.c | 28 ++++++++++++++++++++++++++++ 4 files changed, 72 insertions(+) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 43c3bc0f9544d..60ed0880cc9d6 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -69,6 +69,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___kvm_flush_vm_context, __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa, __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid, + __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range, __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context, __KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff, __KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr, @@ -225,6 +226,8 @@ extern void __kvm_flush_vm_context(void); extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); +extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 728e01d4536b0..a19a9299c8362 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -125,6 +125,16 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt) __kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level); } +static void +handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); + DECLARE_REG(phys_addr_t, start, host_ctxt, 2); + DECLARE_REG(unsigned long, pages, host_ctxt, 3); + + __kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, pages); +} + static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); @@ -316,6 +326,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_flush_vm_context), HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa), HANDLE_FUNC(__kvm_tlb_flush_vmid), + HANDLE_FUNC(__kvm_tlb_flush_vmid_range), HANDLE_FUNC(__kvm_flush_cpu_context), HANDLE_FUNC(__kvm_timer_set_cntvoff), HANDLE_FUNC(__vgic_v3_read_vmcr), diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index 978179133f4b9..213b11952f641 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -130,6 +130,36 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages) +{ + struct tlb_inv_context cxt; + unsigned long stride; + + /* + * Since the range of addresses may not be mapped at + * the same level, assume the worst case as PAGE_SIZE + */ + stride = PAGE_SIZE; + start = round_down(start, stride); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt, false); + + __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + /* See the comment below in __kvm_tlb_flush_vmid_ipa() */ + if (icache_is_vpipt()) + icache_inval_all_pou(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 24cef9b87f9e9..3ca3d38b7eb23 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -111,6 +111,34 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages) +{ + struct tlb_inv_context cxt; + unsigned long stride; + + /* + * Since the range of addresses may not be mapped at + * the same level, assume the worst case as PAGE_SIZE + */ + stride = PAGE_SIZE; + start = round_down(start, stride); + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; From patchwork Wed Jun 21 17:49:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59406EB64D7 for ; Wed, 21 Jun 2023 17:50:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=a2s/2duPwruD1RZwAnW7JTcZDNoSPKnl6t9PZhPnGsc=; b=LboRC8dwfX1ES68CYnNvt55ufI 8jBe+reCvRWR5Ea/zc+ISbzrZBZ/lnhlGGIK6vzuQwgSM5YvkRwreWUYZw2VEiyJHKF7hO/D4T4zs sTBPClExIWSnvuPpFyKhAE68JaDPjMmI6jfwVkonThU0gvrSgDz/3PDkGD7dZcLFYYsP94fbDw8AB m1A+iAeCJq1rvvhnraPj05dIimkzWBtQVIQsF3o2yrSldUoaMfzo/sGM0wY1+YB1guUiRnmD+yA1C aoFoOcCLcHR87xbyJyeChBFBNSt5YW7T03pzqVaxc4mWMDLs8uC6DxCbVrA31ZpEqLePl1egKcx+Q YPzI5ONw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yn-00FHQH-2f; Wed, 21 Jun 2023 17:50:53 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yQ-00FH3D-0b for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:34 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-ba81b37d9d2so6654487276.3 for ; Wed, 21 Jun 2023 10:50:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369828; x=1689961828; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j157PG4NSYE82v2fAItKFhVfRcmmpeiCHUTGpM3SOrs=; b=Gl6wq+mEywBMdam4XfA6Jx3xs02PqlG2630KExwZ+IZ1eaEu2M8C4gqV/l7wcdwgF3 /D1SpZQ0iJjXckKesr45c+8QYgxKeaJkXk4CzLhdqEna8C9zX/I5d6oFGpNmP8iq6sY5 AD2tneZCrgHa95MtCXOU6/zAbzMhBOWLtGo2xx45SVnmLsd1KeV2yL43QUWZzbPYsRE5 0KFNo+sYozd7eoDI93xm/37wexRwY8LUWph38U1kdZ/ZPiRwyt7W5rsyP0dZypbTXy1d v4IZ0o/cb4ZauS58uWHhZOddMD4erO5Pva1AdW/HnuLgUZTI3rDNpeArz5G2OY0RmC2L 7aKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369828; x=1689961828; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j157PG4NSYE82v2fAItKFhVfRcmmpeiCHUTGpM3SOrs=; b=lp6aBk1k+10V2luZf/tjcO/1i3/Qr0tn4CiS4FKTInBImE1IPJ/GuEPvuw3kVp7tnH yxYrIqwgvhpjDfC/gir6tSTdkWPvVQBGC2VnITpJic4fUIwKxgJj9fcDmZ/mN23VDDie lj9ztDgcM8bkl6/QqEaoJCmQF8TZ7zwyBP0eV3DaJkbQKq1pK+pYEgFlIBGhEWB1PNin FcKlxNLC7ht1VykfTBm73LcHAwVOekDfQm/Zw151EWDrGOGQL2DRznt0b3qvkfuO50hR u6qHTzL0aFOrMQHSVwG4d2+yNSizU5VzOnww9s82YjaixbnKXc703W/dmljgJCHRysv3 p6rA== X-Gm-Message-State: AC+VfDw/xuTJx79BFWFEuRQKj8wo3pYp1ujiCqwAGjVcPbQYjpdT7FrG pz5hzYoCffEKkMOR7/QiwoVp3NcUKNfH X-Google-Smtp-Source: ACHHUZ6Ys0yNXltl/QNiOKpOLzit3oOUcal/03jYF6PfCeI/5MRSrcuUGPtaAxFV3KRRmbIadAvYp5jsj7ao X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a25:74c9:0:b0:ba8:4ff5:3217 with SMTP id p192-20020a2574c9000000b00ba84ff53217mr2202294ybc.3.1687369828056; Wed, 21 Jun 2023 10:50:28 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:58 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-8-rananta@google.com> Subject: [RESEND PATCH v5 07/11] KVM: arm64: Define kvm_tlb_flush_vmid_range() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105030_269616_A182B354 X-CRM114-Status: UNSURE ( 9.37 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Implement the helper kvm_tlb_flush_vmid_range() that acts as a wrapper for range-based TLB invalidations. For the given VMID, use the range-based TLBI instructions to do the job or fallback to invalidating all the TLB entries. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/arm64/include/asm/kvm_pgtable.h | 10 ++++++++++ arch/arm64/kvm/hyp/pgtable.c | 20 ++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 4cd6762bda805..1b12295a83595 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -682,4 +682,14 @@ enum kvm_pgtable_prot kvm_pgtable_stage2_pte_prot(kvm_pte_t pte); * kvm_pgtable_prot format. */ enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte); + +/** + * kvm_tlb_flush_vmid_range() - Invalidate/flush a range of TLB entries + * + * @mmu: Stage-2 KVM MMU struct + * @addr: The base Intermediate physical address from which to invalidate + * @size: Size of the range from the base to invalidate + */ +void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, size_t size); #endif /* __ARM64_KVM_PGTABLE_H__ */ diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3d61bd3e591d2..df8ac14d9d3d4 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -631,6 +631,26 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt) return !(pgt->flags & KVM_PGTABLE_S2_NOFWB); } +void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, size_t size) +{ + unsigned long pages, inval_pages; + + if (!system_supports_tlb_range()) { + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + return; + } + + pages = size >> PAGE_SHIFT; + while (pages > 0) { + inval_pages = min(pages, MAX_TLBI_RANGE_PAGES); + kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, addr, inval_pages); + + addr += inval_pages << PAGE_SHIFT; + pages -= inval_pages; + } +} + #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt)) static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot, From patchwork Wed Jun 21 17:49:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26F8CEB64DC for ; Wed, 21 Jun 2023 17:51:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=DCq6+LLXfT8Zs0GZzU/V56OFKt6IuCOBIO7xx9EBs7k=; b=Wsty1VbO7ztIbGNIE9rF+det0T QHD7YukJF6wFbvxVXZInY2OmqDH2/VDY9j0/gQWmgZYFlToj7JPe1DC6lQM2CorHWfPUR/W0gKAq3 U3Ti74vxmlVMz5m33jE2XvxqxL7FQuSCUNMrtg4sFO0cxI8an/67zfXZfb9Ktuq9n1M0UyCpQ/GeU kz7oPDvaxGSZoPNTQvWPo0eeDhJ42P3+V9vl/8C+fJD6EWDI1G/zRrdayeJqRVvFJwqoBZYQtK67C mV+lnAmFMWavvQe1KFWXBijYvE5sEPhB2F2q+pMxlZjZvYSU3i94sXxFF01D1Mjk3fdDGeUWTNJhT VbfUUmtQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1z0-00FHeK-2q; Wed, 21 Jun 2023 17:51:06 +0000 Received: from mail-il1-f201.google.com ([209.85.166.201]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yS-00FH4Z-02 for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:36 +0000 Received: by mail-il1-f201.google.com with SMTP id e9e14a558f8ab-340b21b5927so48736355ab.3 for ; Wed, 21 Jun 2023 10:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369829; x=1689961829; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dtGUI04cZLcqHqfVph3GLMvS0CVYqac6sXI/vA3EN9c=; b=wUCfp8LIVxVyw9rOjJ+xxvqYiNhrMKEQGqZAIrYsflCWYaL7sMIjnn5LDzBF8HHkci fU6aqWxAcgnmVfiz4UwopUTnDAafaCROX3yoC8OfXmrNVemh4RlpXriU2kIUlflYJXHw uMCeKHy/l1FGUBNLLAGrP3KIWF3x8c8QA6lH15hhdrXNCDtmipcd1fYAZABG46exGlSO gFSuHconOaa66m/06jG0Ro0iy0bHzhaKM98M74q46S68x0wg14fu0205JCX28SDxNDoC IeVohMieqPadYlEk/nIWyWNaXHuxsT0P5qiOFBg+yGi/qLEjpv50H3iXe5QJF2vkwCpH B3PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369829; x=1689961829; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dtGUI04cZLcqHqfVph3GLMvS0CVYqac6sXI/vA3EN9c=; b=RKn6KBeosLP/4zFojkaIdhFEnjOyMJlOcGYmgGuHaOic+juQCU9vjp+CQtMYQXqN11 dMf83GdYc5KjtTVJscRaw2ve7oxS5WC/owlrMFN+RXQijD5RTF6gMMdfGqFuVKWxLrIm xed1L6oDmL9hv0vXeJHgX4UMMtWKWkVzvfEmwWEX24KQqi2J0ELsr2j2PLsix0JFirjW CBfzkiU0wm72x+9vviFXV8VxnZH/2CJK7TcIysuv/3qL7iRVWy4itldFjN0X+sgIivHn rvSTqOJu8BNLUJQhbi/S4UCdfiWZnx8FZCtinVsAZXtBYKE9QDHoSnsfqIQgdvYSwj3y TdTQ== X-Gm-Message-State: AC+VfDwepqdis9gY4pof9syPIQWVYMU+1wJhv+GNgo4CPu3EtzFrLRPF 7L+jQdCpVLOedd7QGMuG+5wXmYkZkEdQ X-Google-Smtp-Source: ACHHUZ6E9qc4T7u9+yG0bI7LCSWrNIIprVuEXDN8tL08Gxqy8ruadzAtjVxDCW2tLBKnP2MlbqXA+k7+GwdZ X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a92:c9cb:0:b0:33e:6230:aa65 with SMTP id k11-20020a92c9cb000000b0033e6230aa65mr6242404ilq.0.1687369829258; Wed, 21 Jun 2023 10:50:29 -0700 (PDT) Date: Wed, 21 Jun 2023 17:49:59 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-9-rananta@google.com> Subject: [RESEND PATCH v5 08/11] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105032_047491_9873A387 X-CRM114-Status: UNSURE ( 6.82 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/mmu.c | 7 +++++++ 2 files changed, 10 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 81ab41b84f436..343fb530eea9c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void); #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS int kvm_arch_flush_remote_tlbs(struct kvm *kvm); +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages); + static inline bool kvm_vm_is_protected(struct kvm *kvm) { return false; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d0a0d3dca9316..c3ec2141c3284 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -92,6 +92,13 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm) return 0; } +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages) +{ + kvm_tlb_flush_vmid_range(&kvm->arch.mmu, + start_gfn << PAGE_SHIFT, pages << PAGE_SHIFT); + return 0; +} + static bool kvm_is_device_pfn(unsigned long pfn) { return !pfn_is_map_memory(pfn); From patchwork Wed Jun 21 17:50:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D258AEB64D8 for ; Wed, 21 Jun 2023 17:51:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=++2QXO92MvwnsOgLs7mK9IfRQGVZYrIqlFBSqS2QBBc=; b=hdBNvtg6H5y1FYcj9PSClB430Z 0h7QdG9GdeyhxN9i06Huspy1dDww9aNQMj+x1nLKDAeONFC3FftyDk3RRh8LimqekCyx+0Xxk3ESx oQ8w4SXptHQKc8VgisMhJLW83AEa34ULmbpyujTURAFEsFEymICd8MNTTiJr2DwctwfzEziST//4V idNHVLyS3HXLW/YGVVVljDloIu2AkYn38Qkbkn2+c/cTayg0ULuSmURjvVqD6+zLPFJSgB0G/VniA hMKlKbl0jzjxv0NtxbrBZnYwUfHV5Lvakx0LHhgLCZLnRH5ee2VALb75JO2mxUzNYSRqax7YGjDXB z6uWOHFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1z2-00FHfz-1m; Wed, 21 Jun 2023 17:51:08 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yR-00FH5P-1Z for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:36 +0000 Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-77e3eaa1343so376629139f.2 for ; Wed, 21 Jun 2023 10:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369830; x=1689961830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kpR1MspLCCKbLGu542rFrU99snL12qe0SU5vnENHbnc=; b=obkQNETduI4Wdlr8DqbH6rQDpqF2mUi0lafiiZR1QXrNN2+tKemED10KMGrakG13Nq +F9e7JIuRjiRR9EfoJnua+9jPsENy4Tu90Vrn/2nK11OMp8Aj/golURDqyPfJ94Cfrxi hdWdLoNiohD/PMMDEG0mtQHMd7A92j4tIxNPh8D/cM9gYPH0WwqnOU4OcPsZKdr9HTOs pyGKjpe6+P6Xcb6ByP94N+J+yZmYDJogpjw8kcjVcP5B1YTqtAJdqi8Hp9kutYXKuAUx OCxMudA25zodKFZSyAwbz9vLyvHyzcqnxGuKjUldsdPVjyx7x6uGCQbzc3ZzHDMDOM0D Kg8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369830; x=1689961830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kpR1MspLCCKbLGu542rFrU99snL12qe0SU5vnENHbnc=; b=a0uQzJ5KH6Hut6QPNZijU+epsPDyoHGWDhm6g8YlKezWNHoww22NnklHOMLmmDHWJx MFAJSaxUHIQx3z7N2mF9hAu41zYRiNBy2bP9cooDG13/wmjaurHdAbZVOlqns6yfN7iO 0tjrvK+73pl75K2rj4BtVN/8X0EEWu3jN3EZ72YhUo5T2w88C5rmhei7RuT8CLdw3D+g ZsWKKwE/IXCO50yhEp+OmHq1WK44qrps8V4T1kTojpaogWPsVG7aUCGg39yUAIMriaFU gbxroGbVYfM1Gaj2P4sUbPx+QY/tTzkoSFz0muDVRZ03YdMgadtEDEkD1sgh2bVIQk7g BYVA== X-Gm-Message-State: AC+VfDyAG218m3eMSXkPssK7wmeiwkeSoKd4zw/PJyY3BLIR9M3r86zY zDF+SeoAj26X7HxDHSPnDf8qSHavt3Gu X-Google-Smtp-Source: ACHHUZ6lEejYkvXSZImNPgkcjtxASyr0tUJRYFuyzfo7eVoPr61PQ+mo3lAqnIjMYGuKsT4NfkEe3+3Un922 X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6602:152:b0:777:b38d:2333 with SMTP id v18-20020a056602015200b00777b38d2333mr5713635iot.3.1687369830330; Wed, 21 Jun 2023 10:50:30 -0700 (PDT) Date: Wed, 21 Jun 2023 17:50:00 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-10-rananta@google.com> Subject: [RESEND PATCH v5 09/11] KVM: arm64: Flush only the memslot after write-protect From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105031_538544_43B3064D X-CRM114-Status: UNSURE ( 9.47 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After write-protecting the region, currently KVM invalidates the entire TLB entries using kvm_flush_remote_tlbs(). Instead, scope the invalidation only to the targeted memslot. If supported, the architecture would use the range-based TLBI instructions to flush the memslot or else fallback to flushing all of the TLBs. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/arm64/kvm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c3ec2141c3284..94f10e670c100 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -992,7 +992,7 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) write_lock(&kvm->mmu_lock); stage2_wp_range(&kvm->arch.mmu, start, end); write_unlock(&kvm->mmu_lock); - kvm_flush_remote_tlbs(kvm); + kvm_flush_remote_tlbs_memslot(kvm, memslot); } /** From patchwork Wed Jun 21 17:50:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15551EB64D8 for ; Wed, 21 Jun 2023 17:51:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=2HJQDzH31oeOr553UkgEpKTg4Gr+/iqk/d65nFWnLSU=; b=0I8OGGnr/SNCVDQDzxFBUuGVF8 /0IqNOfjbMUkBhW71n3QAlG5T/i+BAeYUV6mzEOByuGPQFvDN3/sLW9rvqImf2o+b6/0NJjtNMVh7 gRygxHbZICLH99QCIDzwnVzfJie+aMrknY08fbcqGXK6NVxDq8TY54PgodxNznNZiOpDoWF2yTToS uzUnq8Pspc+jxvq9hwUruRt98cLU/Nv3QJ+rnNeS3CHHkFEsiJ9b4OvzqfnJFvKZgRsoWwaL9FYro fURGbsKISQI8YFqg0TcKAJLrI9QOAu6BDahrSic9XGj6HomC9K0KyeIT+R00vbwxzqTyYDvxtclIA 3Dgot7ug==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1z7-00FHlI-2d; Wed, 21 Jun 2023 17:51:13 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yX-00FHAq-2u for linux-riscv@bombadil.infradead.org; Wed, 21 Jun 2023 17:50:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=txSuZoJcjD5pcccV63ogAXj6R+ImdzOqCpjUFBogFls=; b=JUmE7K8m1q/0LRiAxjjaisRY73 aiYFVu3UwPw69Mxlq7wCcqqy/kwckC5lpjBB+acP61m15GhzIFUibXC5oipINuXG2zh85NAvD1ctL +eLmLA6svIEcPVHk8hskOhivD9nfey3YrqWDURfceB69ps6+5QQIQ4zBpX6wQZ9/ETKSMb2CpXDeh yyqsRptS08IKVoKf/L8pfn8oEoxKAuwQ3Zld0tD/4SjA5PjDHYX5qitEwXQs/gLRJ6VEziAtVFmLc rSGAVi+pSYMz2PhmJidpCadsG3qhiriKPaSb/JJ8foUzy3JIpReuHosyt9M9BhMDmGM0zGXQTKyIe hFZwbPpQ==; Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yU-0001Sk-08 for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:36 +0000 Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-77d99de0e9bso573091739f.3 for ; Wed, 21 Jun 2023 10:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369831; x=1689961831; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=txSuZoJcjD5pcccV63ogAXj6R+ImdzOqCpjUFBogFls=; b=tr43CcP/8i342v5/9Ct1YrQsldyO50OPcRiuSq6sRQko5jigiWaQUhv5W5PI7oLr3p sC4wmp8EhlAUZ+KM6y+uwTC1qr1H0WPEZTRKESbsiL9VpbH4vqRV36avGePKj3Nxx7Y8 1NCjmb6o2QsoVVBbkLqu7Hd7yyeHbyGs10Eaul2ycHCksKhyKsW++ZrSeFsO0m3sGcP9 IorLAPs+osOx+RwQUutNQUN5JBQDyYVVUHApYkDisryzVRTt5RItCml0Gj7t46jh9yBn c1L5nKOs2o/qlzL7Vn1WKPUAr4Q9Cug2flxP+m8kS6WYBpop3Qeyht2U29DIWx6nHfNq vDwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369831; x=1689961831; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=txSuZoJcjD5pcccV63ogAXj6R+ImdzOqCpjUFBogFls=; b=A+KelZSfpp12sqOphdleqXU7lu4e1ga5jNFoFTynhuAeOKh4f76mRuqYAGg2SnGJ0P Q03h/QnsaxMSYFNqt0AgOG+sKg8YKn+hzogvvn4uLrr2ETP/EAv07Ml/zAX3oaHzSC8v KJcv7rvFg4phS6bTFz2nxIwiJtsAQW0MuckcCraIj4wyyvrZi5d7WqDacoIXbwSCAhth ywwAdIJFJSEgTFfhuxvxUiSKQWR28k0FrnWEjUscW2aU/8uXpo/wY0dMIvUKbR/nynwq f+cfQcw6Q124DZEu2Y48vuL5q6akftlSm7VVd2tfMGVZ4bXHXQXFoJmZUcGcdGR3VgPv 9i5g== X-Gm-Message-State: AC+VfDwu4sDDiO/pk6du24uAQ/Wo4QnUzmO4YnQqcoBXVwrGXM3ZSS88 u1UBcTxGbuVIU2yINEtRxiBzrJDvpUCW X-Google-Smtp-Source: ACHHUZ68VfUE9LrG5crOF7yNQz1Mp6rxMXDaLofQZc3ceY9MQM5/rteSPwnUZE8mHUTdjrcj+99o5c8geD0l X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a5e:a908:0:b0:77e:4866:5c0c with SMTP id c8-20020a5ea908000000b0077e48665c0cmr3257495iod.0.1687369831367; Wed, 21 Jun 2023 10:50:31 -0700 (PDT) Date: Wed, 21 Jun 2023 17:50:01 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-11-rananta@google.com> Subject: [RESEND PATCH v5 10/11] KVM: arm64: Invalidate the table entries upon a range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_185034_205156_2258AAFB X-CRM114-Status: UNSURE ( 8.93 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, during the operations such as a hugepage collapse, KVM would flush the entire VM's context using 'vmalls12e1is' TLBI operation. Specifically, if the VM is faulting on many hugepages (say after dirty-logging), it creates a performance penalty for the guest whose pages have already been faulted earlier as they would have to refill their TLBs again. Instead, leverage kvm_tlb_flush_vmid_range() for table entries. If the system supports it, only the required range will be flushed. Else, it'll fallback to the previous mechanism. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Gavin Shan --- arch/arm64/kvm/hyp/pgtable.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index df8ac14d9d3d4..50ef7623c54db 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -766,7 +766,8 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, * value (if any). */ if (kvm_pte_table(ctx->old, ctx->level)) - kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + kvm_tlb_flush_vmid_range(mmu, ctx->addr, + kvm_granule_size(ctx->level)); else if (kvm_pte_valid(ctx->old)) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); From patchwork Wed Jun 21 17:50:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13287835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D3C8BEB64DD for ; Wed, 21 Jun 2023 17:51:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kVM46FSRJ81Dt7uk2r8kJzh6hMgTtDtG3GFJGvAadTw=; b=ZV1O9t3iUPAciTaIkNLdgX9Rxb xHc3NflvKWykV11WefQ2BLgjDp1p0XcpKlK8GiNrahQ+RYggYS/gZIQ1DN1chUW17C40LW8jqJyFr mFTgcsOl5bS3aHvwyhwONg23QcEmiiqj3wPZoR1srwk3NJSXSvuSmRK2lEQrZ5qEipH1ay0K0CsoT hdN2IOkkcAJMAXn6ZLKdKMpEacfAtbO6ALMi25MqUewTNEMMNFTlwqbUt26TT1oh5XaxjbVVtu76f JA5YXnCbPkA1SjdF9KDLHZobaJFJWBK+fLtqh2aasa2i8/oc5cxlawrnOZKc1jVK0nlGMcUi8RN/F uIZEbccQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qC1zM-00FHyz-1Y; Wed, 21 Jun 2023 17:51:28 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qC1yT-00FH7A-1d for linux-riscv@lists.infradead.org; Wed, 21 Jun 2023 17:50:40 +0000 Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-77e2421ee83so525717039f.2 for ; Wed, 21 Jun 2023 10:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687369832; x=1689961832; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1k5M++jVJQN3HsAnULjgdeEMSBZOk/cO6/5CPwT5F8k=; b=w6GZsyPqZZib29SWYZEVnZb1y4/Lbstpu1dT+fTCnzTCJENFj5D4vH91+GzI+/bi6H 1oi0xlgidMJoef1Cy+DHvVUNiEPFe0GTcgPM9Vz0WN5MJ/vDDywJ8SAurP3W7Z0E8RV1 bL9yrGC0vSg5SmdyfHRQ/SYavKI9Z8TdIi8UUAeeguYA5s8unRrmN3ckrBjyYDhQ3H/C 6kxbOF2nw+aLZFKsd8D0RuLiMEtzfCulkKG3cg/gg/4NqQ7iNsqHpVsN/CzszFwqzu1T HZd28iVtbf1SIrpe9Y+ZQDrpu1IIvftL7laiPI+V4lrVf1uTFxBYBnD5cSNCRy3C9bk2 Iw5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687369832; x=1689961832; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1k5M++jVJQN3HsAnULjgdeEMSBZOk/cO6/5CPwT5F8k=; b=ZCpz/r2ye2FLx5+D9pdtnkKEkbmiq+FRmcEjau6YgBJsoZUHH09+Lv/t19QhVma5Rx JCco2BQeZ3uvuAbKmVKYoCRWktkRD7qt13Sk1sEJ9enZAE2P8B5S/4blEaVHfS86jYje CjkPwKl7QpBoGXVQZ9Qxsb3yJ6zPEi+QmWencvyifnal3NuhH786t0w+yLlZOdfJOcZt UNl2zOH4TbvYpgXD3qO9kc2gteS0Q3p2c5ES7t6n//FOPi57nbv2/+NMdFFawOh+FaEk zPjGOldPexvaueCySNWB/IcwkC0FW6AbXjJb5DXOJN1d8dIrXBuY4VSd47SJOPBxcIl+ 2zeg== X-Gm-Message-State: AC+VfDwcMCKRI+uvDIMXsMw/k9ryfpFh71mLSV6C2p9I3plzAqVTtF7q B2SgvxtOuOh0Ei4COmzrTCb8VZe/mSZT X-Google-Smtp-Source: ACHHUZ7D/T/4id1CIm/7FILU8FYzyWybEpfQUoMmSgByaqoojlf3IY4EM1ax7sfpQ3tyXmvK1JA9xHULv5Uv X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:2ac7:0:b0:41a:c455:f4c8 with SMTP id w190-20020a022ac7000000b0041ac455f4c8mr5918374jaw.3.1687369832124; Wed, 21 Jun 2023 10:50:32 -0700 (PDT) Date: Wed, 21 Jun 2023 17:50:02 +0000 In-Reply-To: <20230621175002.2832640-1-rananta@google.com> Mime-Version: 1.0 References: <20230621175002.2832640-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621175002.2832640-12-rananta@google.com> Subject: [RESEND PATCH v5 11/11] KVM: arm64: Use TLBI range-based intructions for unmap From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Sean Christopherson , Huacai Chen , Zenghui Yu , Anup Patel , Atish Patra , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , David Matlack , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230621_105033_571752_2CA9230F X-CRM114-Status: GOOD ( 17.32 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck on some systems. Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition deferred TLB invalidation on the system supporting FWB, as the optimization is entirely pointless when the unmap walker needs to perform CMOs. Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/hyp/pgtable.c | 67 +++++++++++++++++++++++++++++++----- 1 file changed, 58 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 50ef7623c54db..1451f9011590d 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -789,16 +789,54 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n smp_store_release(ctx->ptep, new); } -static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, - struct kvm_pgtable_mm_ops *mm_ops) +struct stage2_unmap_data { + struct kvm_pgtable *pgt; + bool defer_tlb_flush_init; +}; + +static bool __stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt) +{ + /* + * If FEAT_TLBIRANGE is implemented, defer the individual + * TLB invalidations until the entire walk is finished, and + * then use the range-based TLBI instructions to do the + * invalidations. Condition deferred TLB invalidation on the + * system supporting FWB, as the optimization is entirely + * pointless when the unmap walker needs to perform CMOs. + */ + return system_supports_tlb_range() && stage2_has_fwb(pgt); +} + +static bool stage2_unmap_defer_tlb_flush(struct stage2_unmap_data *unmap_data) +{ + bool defer_tlb_flush = __stage2_unmap_defer_tlb_flush(unmap_data->pgt); + + /* + * Since __stage2_unmap_defer_tlb_flush() is based on alternative + * patching and the TLBIs' operations behavior depend on this, + * track if there's any change in the state during the unmap sequence. + */ + WARN_ON(unmap_data->defer_tlb_flush_init != defer_tlb_flush); + return defer_tlb_flush; +} + +static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx, + struct kvm_s2_mmu *mmu, + struct kvm_pgtable_mm_ops *mm_ops) { + struct stage2_unmap_data *unmap_data = ctx->arg; + /* - * Clear the existing PTE, and perform break-before-make with - * TLB maintenance if it was valid. + * Clear the existing PTE, and perform break-before-make if it was + * valid. Depending on the system support, the TLB maintenance for + * the same can be deferred until the entire unmap is completed. */ if (kvm_pte_valid(ctx->old)) { kvm_clear_pte(ctx->ptep); - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); + + if (!stage2_unmap_defer_tlb_flush(unmap_data)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, + ctx->addr, ctx->level); } mm_ops->put_page(ctx->ptep); @@ -1005,7 +1043,8 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, enum kvm_pgtable_walk_flags visit) { - struct kvm_pgtable *pgt = ctx->arg; + struct stage2_unmap_data *unmap_data = ctx->arg; + struct kvm_pgtable *pgt = unmap_data->pgt; struct kvm_s2_mmu *mmu = pgt->mmu; struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops; kvm_pte_t *childp = NULL; @@ -1033,7 +1072,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, * block entry and rely on the remaining portions being faulted * back lazily. */ - stage2_put_pte(ctx, mmu, mm_ops); + stage2_unmap_put_pte(ctx, mmu, mm_ops); if (need_flush && mm_ops->dcache_clean_inval_poc) mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops), @@ -1047,13 +1086,23 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { + int ret; + struct stage2_unmap_data unmap_data = { + .pgt = pgt, + .defer_tlb_flush_init = __stage2_unmap_defer_tlb_flush(pgt), + }; struct kvm_pgtable_walker walker = { .cb = stage2_unmap_walker, - .arg = pgt, + .arg = &unmap_data, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker); + if (stage2_unmap_defer_tlb_flush(&unmap_data)) + /* Perform the deferred TLB invalidations */ + kvm_tlb_flush_vmid_range(pgt->mmu, addr, size); + + return ret; } struct stage2_attr_data {