From patchwork Fri Jun 2 17:01:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Colton Lewis X-Patchwork-Id: 13265712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 308ABC7EE29 for ; Fri, 2 Jun 2023 17:03:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=LoFs3X2cygJFOaljut7oBPMoDavXwqdQZC5hI5qrDMc=; b=GARlkAu3nmaEhwyfizDTHvGgAX fYasisNG7iV9tYkhgZ2I0G7VI15Bw7PR9PxnyO/M2zdiKhUxNhS7pxiaAcsXXG2Spi8DSbre2nOzU 4UqG30FFre42mnGTKEPXY/MPmIGQQdooKprxnWLBlG7ZA27LZBumDQA7d45ruu8AHQdtk8y5aMSdd hcq6eYKOGFVSwpRHs99LVKbsoT8SoX28WGBD7LXVPwyfQbsErj0pi68IAKqxop2TTwg1/d3TPo/p+ 7DGfVHv0lfGm6lPIjVjlYzI5u9Nn9mOI6/wftn3vF1fDLSEBJbYHuxTqzAuBAFG12iV7atDM/qjQd yc0iXACw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q58B1-007UMN-17; Fri, 02 Jun 2023 17:02:59 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q58Ax-007UJY-0D for linux-arm-kernel@lists.infradead.org; Fri, 02 Jun 2023 17:02:56 +0000 Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-7603d830533so81079139f.1 for ; Fri, 02 Jun 2023 10:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685725374; x=1688317374; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=46U2EWEjxn55kWeKsj3lDgLNoam846hFcVZO+vAwdLA=; b=WAqGXaWPraSA2daImKoen5B6tdZ+WYHgTjxhUUeeZAsbrniK5QsoJdy6bnYalQom0q MTUHUxan0x428dbLSXOSDczGl+sVACZNDReIhyFVk8qGv8O19J818JqDUV+L39pYvd+O +D0peOAv5LwnSHuhbBKS4on+Spl3k1vhxR2tQ3f+hr+YwUTyNz098zCnNqlglZ99wmXF 03RZCAfLMqzIZPxHZwDc+9cJzrwA0lt7idYGl5Mr8XHmSaECCNan2CtJJnr3I5Ts2P2r magwgDPHCmtfLDNKqCKxBMgHtvBcTUMA++iZxC/lrSlB4KiCVknekWOPC0oT6ZjtI3V8 dGqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685725374; x=1688317374; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=46U2EWEjxn55kWeKsj3lDgLNoam846hFcVZO+vAwdLA=; b=laAxpCYqw+rWirP+kI+xlbxIZtJc+DVtCJqFYzFDjPj/UeX5pCLJ/hlsTExQP2+2f6 QjP6hfg8H7hinqt4EeJ+K1hCUvC/mdUKbNcnRPRS0dfYurF5lczInKLaGkhuVb/XQsRJ 6AZNZdkC+FW59SVi/q/Dj7cg0t4ihvaYqsjqcWSRTIW/FrgL10oKdGEwK12yMEc0z9se oD22b/HQDyQvQaUoACsyij/ZO5W2QtE1itElDF0RndF52MxJnnBnnfnzYJFw9czlE1se tArJaGaDFnfxL1WvYz66dE2GgF5HA6VMf9ltiOP+jLt/LnXJB1qjgkIsphTt+PoVxO2f UKmA== X-Gm-Message-State: AC+VfDyKvO1XWrZ96f+xeoOWIzvnLbVHcAI4oHvO8JXMPkFl3kvNVYm/ 8e3ehEFVvvl8YMDZMZNRoNYyxg/3D87375ovcg== X-Google-Smtp-Source: ACHHUZ4geDJMOIvtQH6zUtgqB8OHUq2dHYHBZAw17LyumZCyIixhJcFdC+UZZW8jvomo19VQYFFEQblUZyGsWdbTIA== X-Received: from coltonlewis-kvm.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:14ce]) (user=coltonlewis job=sendgmr) by 2002:a05:6602:1503:b0:777:2367:a7d3 with SMTP id g3-20020a056602150300b007772367a7d3mr1813249iow.0.1685725373854; Fri, 02 Jun 2023 10:02:53 -0700 (PDT) Date: Fri, 2 Jun 2023 17:01:47 +0000 In-Reply-To: <20230602170147.1541355-1-coltonlewis@google.com> Mime-Version: 1.0 References: <20230602170147.1541355-1-coltonlewis@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Message-ID: <20230602170147.1541355-4-coltonlewis@google.com> Subject: [PATCH 3/3] KVM: arm64: Skip break phase when we have FEAT_BBM level 2 From: Colton Lewis To: kvm@vger.kernel.org Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, Colton Lewis X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230602_100255_100503_C2E9AE35 X-CRM114-Status: GOOD ( 16.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Skip the break phase of break-before-make when the CPU has FEAT_BBM level 2. This allows skipping some expensive invalidation and serialization and should result in significant performance improvements when changing block size. The ARM manual section D5.10.1 specifically states under heading "Support levels for changing block size" that FEAT_BBM Level 2 support means changing block size does not break coherency, ordering guarantees, or uniprocessor semantics. Because a compare-and-exchange operation was used in the break phase to serialize access to the PTE, an analogous compare-and-exchange is introduced in the make phase to ensure serialization remains even if the break phase is skipped and proper handling is introduced to account for this function now having a way to fail. Considering the possibility that the new pte has different permissions than the old pte, the minimum necessary tlb invalidations are used. Signed-off-by: Colton Lewis --- arch/arm64/kvm/hyp/pgtable.c | 58 +++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 7 deletions(-) -- 2.41.0.rc0.172.g3f132b7071-goog diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 8acab89080af9..6778e3df697f7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -643,6 +643,11 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt) return !(pgt->flags & KVM_PGTABLE_S2_NOFWB); } +static bool stage2_has_bbm_level2(void) +{ + return cpus_have_const_cap(ARM64_HAS_STAGE2_BBM2); +} + #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt)) static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot, @@ -730,7 +735,7 @@ static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_ * @ctx: context of the visited pte. * @mmu: stage-2 mmu * - * Returns: true if the pte was successfully broken. + * Returns: true if the pte was successfully broken or there is no need. * * If the removed pte was valid, performs the necessary serialization and TLB * invalidation for the old value. For counted ptes, drops the reference count @@ -750,6 +755,10 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, return false; } + /* There is no need to break the pte. */ + if (stage2_has_bbm_level2()) + return true; + if (!stage2_try_set_pte(ctx, KVM_INVALID_PTE_LOCKED)) return false; @@ -771,16 +780,45 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, return true; } -static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t new) +static bool stage2_pte_perms_equal(kvm_pte_t p1, kvm_pte_t p2) +{ + u64 perms1 = p1 & KVM_PGTABLE_PROT_RWX; + u64 perms2 = p2 & KVM_PGTABLE_PROT_RWX; + + return perms1 == perms2; +} + +/** + * stage2_try_make_pte() - Attempts to install a new pte. + * + * @ctx: context of the visited pte. + * @new: new pte to install + * + * Returns: true if the pte was successfully installed + * + * If the old pte had different permissions, perform appropriate TLB + * invalidation for the old value. For counted ptes, drops the + * reference count on the containing table page. + */ +static bool stage2_try_make_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, kvm_pte_t new) { struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops; - WARN_ON(!stage2_pte_is_locked(*ctx->ptep)); + if (!stage2_has_bbm_level2()) + WARN_ON(!stage2_pte_is_locked(*ctx->ptep)); + + if (!stage2_try_set_pte(ctx, new)) + return false; + + if (kvm_pte_table(ctx->old, ctx->level)) + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + else if (kvm_pte_valid(ctx->old) && !stage2_pte_perms_equal(ctx->old, new)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, mmu, ctx->addr, ctx->level); if (stage2_pte_is_counted(new)) mm_ops->get_page(ctx->ptep); - smp_store_release(ctx->ptep, new); + return true; } static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, @@ -879,7 +917,8 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - stage2_make_pte(ctx, new); + if (!stage2_try_make_pte(ctx, data->mmu, new)) + return -EAGAIN; return 0; } @@ -934,7 +973,9 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx, * will be mapped lazily. */ new = kvm_init_table_pte(childp, mm_ops); - stage2_make_pte(ctx, new); + + if (!stage2_try_make_pte(ctx, data->mmu, new)) + return -EAGAIN; return 0; } @@ -1385,7 +1426,10 @@ static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx, * writes the PTE using smp_store_release(). */ new = kvm_init_table_pte(childp, mm_ops); - stage2_make_pte(ctx, new); + + if (!stage2_try_make_pte(ctx, mmu, new)) + return -EAGAIN; + dsb(ishst); return 0; }