From patchwork Fri May 26 23:44:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6A64C77B7A for ; Fri, 26 May 2023 23:44:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 233A2280003; Fri, 26 May 2023 19:44:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E32D280001; Fri, 26 May 2023 19:44:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05C36280003; Fri, 26 May 2023 19:44:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EBE48280001 for ; Fri, 26 May 2023 19:44:50 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C1126AED8F for ; Fri, 26 May 2023 23:44:50 +0000 (UTC) X-FDA: 80834038740.09.9392CCC Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 7369640014 for ; Fri, 26 May 2023 23:44:48 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=A+5tnxly; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144688; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gdtrmeQ7r8fSzyqqoeow0Oco6w18CVCYcKcBQXKvLk0=; b=NPmAOQobc7iY+qicdMu2Nal46l0MoZWenUtxvWC3I8u/vTvaRvcxbIrkp5p2loihZEMat0 yUKTFdTWtOBzcbZSvwayrFQzqG/i7gTXlFvcY0jyFiiMhFk/CxzWyNWhRi6GKavcVCx8Dr dlFse8XK2/n6/NJp5NM+Y/lV1D6bizg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=A+5tnxly; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144688; a=rsa-sha256; cv=none; b=CSO5jqQttPE1dU5fzbEfsZG+UWaWQp3bMytoLslzFgeZr1Shbroa4Ranqh+c+p6fUvkVSu Q/DRSGojXyAHxW2xSYIbHYkUW0A4mxXpS2PwddxPLmS7jxyy2zXDbJ6fBHDHJCP3UtOih0 aVFHumHhe6dXbEW9IFNMYlFI+mbGiEk= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-babb78a3daaso1692967276.1 for ; Fri, 26 May 2023 16:44:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144687; x=1687736687; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gdtrmeQ7r8fSzyqqoeow0Oco6w18CVCYcKcBQXKvLk0=; b=A+5tnxly4jAziM7PCgC2JSUrvZIuWUL9UuTqWd7F/s/V7U9ZvBSUtbCMn4PDjXhEkh xPkXA8ijRXU8kfBAAJh45kvENT/cQ+0BqqVDKv4CzNn3FeibDZ4esB2AS5+kj2xWW1A/ 5f+rTWaRryOWr0eLpQP68cwlR3USUfp7Ely4h3ulPbhNc/8a6O9EbhRXY0Hc8yu56mx8 LQMCwDasuefCJVI+rIDEYvr3/19bdInHAEUvVq6phC2Jc21wzdoWQlzM7WDhIq/i18eK BEwyErOKi0xkhmnwWH0N/n1UzNpwg5kRKa2pc24aeVs5XXh0p7GOcbD7gzDkKgQaeHXr C7IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144687; x=1687736687; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gdtrmeQ7r8fSzyqqoeow0Oco6w18CVCYcKcBQXKvLk0=; b=g8/JufTnF2NTynxa84YKtQwp5eluAh9OXZOMtk0HqEqqq3rt7Aku96vY4DK5vK7SyX O0XO/6yZgNK0LrlG+BdHCl5zmXEqvZMWk5Xew6tJNe0L5jzuu24zhgLiVRHXPD1uTISM 9gITp7W6tc/MU4YBdAZ2vSANfZwXhmMcu+jIGY901m4KhKuzYQlZvkPQp9nDF8ZZGvPA q+sGWeaZU2FU4Qdppdq4hhlyHBmdM9F4HBu7GEIz5aO54BUCH4f6W3klnvbE3xmofLzT 1r9T64vopMPDr0CoVDyrpeAVvEOPq7Zf4ggXxmwEHHsHBfTX8/0OFhoLbV9G+wY4zpV0 a+2g== X-Gm-Message-State: AC+VfDySL9sIGUS6OAtGwMuM4qcOh00E7lIvbYh4KuKolq+iTVV/uXg5 sAgbgMFBx6yQ0PNw74BIxNzyfL1UemI= X-Google-Smtp-Source: ACHHUZ75roT4ZouTjM2lH4DZFyTBmS3QLSjPTFGxgUaEGtzpG1LtCtXvcTQXcsV4/+3guLPQ6FLc+WkYaOc= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a25:b28d:0:b0:bad:155a:1004 with SMTP id k13-20020a25b28d000000b00bad155a1004mr1830575ybj.2.1685144687512; Fri, 26 May 2023 16:44:47 -0700 (PDT) Date: Fri, 26 May 2023 17:44:30 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-6-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 05/10] kvm/arm64: add kvm_arch_test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspam-User: X-Stat-Signature: miudf7xkzcik9j8g78t1w1nhsf8zcdkq X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7369640014 X-HE-Tag: 1685144688-465266 X-HE-Meta: U2FsdGVkX1/4yW44XV5X+AfA7SSAlqq8quhpfnZ7tRZGWf0PAL/RMDWvq1B6i2jvstzTM3TbynPNb2KMuQsawX3+jqagfR7RXcz66wgD6aK4+SDb2jpYVB8CDM68rkDBAarA473IFZo7rJlPZQoHL3gyrjm0asIqmGlD8SlLgpjySPxb8nBTESB9BWv8ijpV3Dyj7QkehkOo82tTcSn1dekrpBR606Yl/sJ6Kw/9ilRJgmF9OZnjp3uh8RtZM2WKLGxOrTyJ/BLOkYshF1mkscMz2/Nfx9O2tvA0hs/fBWXB9b885p0BWjG6IgX+lbpFP4v3cN20PPlwtc/fTxqFdbWUJnwe2JwRzGmpzEDK2FAl6Eeg6Ra08h9Yzv18yV/z+JcMVtpgjokmpT47XLPjMBdqOqvuEwcOoa8h3tYcRjZrvhoOvKqmpc0f7DQplHyLBidxFumRjGFVq0CkmMPh/NRfpIkXytlKDyOeqJa7YC5v7sDKyT0KtDzFRKfW7ywYemrCSRw1vTQRK8hoILmLbD1MufNUOMDeRMa9axH99c5LCmuLFmoNca35ed/T5XaDUPidJDMd09ClDr2tc8jAs4pVmiOgHbJmOSzFDEHWi3UdSCncIIETva2DXH2sTIhHJ8r4uuaPwIBmetL0KZbBSBIJb191Q9btAUBDxe3TbWLq755gxHdlKq81xjHV5mhUBUB3njZu0KFpaz35GFURgYLewasLVT2g3AhovHq+zzSuCD5xVGFubZxmiXkPMoTBPwTV34wkBCwMQlXY/ukwoKtDD0SNiOJ5+EBwM0muIC2veSEmavMRiwlCKJqktOLGLUs04IMw8naV44ueOPqoltiltZHONeJpAq4IvSGgiE2XAxiKWXh7oX3PBOfUI1Szqcgbw0Ii4y5Y0YNuW/bnUTHPEPT1aZBTT+cP1jSKTxGe06IMR/WKPRR2ct+A3PgiyE69lugxw5LwlIn0j66 tGiX2T6y gKxdp2uSJ6mF7S1ul0IijZQwuPoUXtBL2fFRUki3suuD8sN45CD7OJymvFr/06M4OT48d9WLAFLzA2ipX18/9ZykrDDvUXYjloZgysb0QpIieySXYjX7hRv8Ak0Q5e31upxAUBwG/ydg/YE0Nv5GSCDGMRypU3TI6JSlxXoi/7T+B9BJl0lXNgybfYYHp5GdUny1mPanVzz1zEhCGIZ9wBmljPxIDbKoVm20v0OvKl+xXLEe5KvTZcq1uWgjFl0X5SPsCtfnSiZHiPzuF+hkyUC8m4ar5PTsjtEE0oPM/aswUDh5CKMEzWZTB7T6UcXtFMX6m7uxNgzzVHpDhqjOnbxqNupTyes9p8qfulA6OlUw5QB8ONKE2AQfdByhLWjnpnjMLtN9XZ1IfxZELMXTIPWZCVoE3TeeElU6EgwfYdmi6gk6zA4UFQu/Bxcwa+K4jnM2Hfcul3eRN6axH4n/mXk3vZEO3RAjveb1OigDshU7UpdrFTBX9jd6jjq+d9QfQIJo99nKs25dIEd71ge/5vPh2AXiZsQqxE1n72tiOEkEWDVHWd2VR6reN55OkbqWMRdK2PT7ZNAl0OagoCJJyKntTdPiKrQQAN3oFZ4b6GBKo7V9R8I1Dx1AYOumLykeW5GJFZo2WVzO5cpZrbWP6ObccBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement kvm_arch_test_clear_young() to support the fast path in mmu_notifier_ops->test_clear_young(). It focuses on a simple case, i.e., hardware sets the accessed bit in KVM PTEs and VMs are not protected, where it can rely on RCU and cmpxchg to safely clear the accessed bit without taking kvm->mmu_lock. Complex cases fall back to the existing slow path where kvm->mmu_lock is then taken. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/kvm_host.h | 6 ++++++ arch/arm64/kvm/mmu.c | 36 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7e7e19ef6993..da32b0890716 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1113,4 +1113,10 @@ static inline void kvm_hyp_reserve(void) { } void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu); bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); +#define kvm_arch_has_test_clear_young kvm_arch_has_test_clear_young +static inline bool kvm_arch_has_test_clear_young(void) +{ + return cpu_has_hw_af() && !is_protected_kvm_enabled(); +} + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c3b3e2afe26f..26a8d955b49c 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1678,6 +1678,42 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) range->start << PAGE_SHIFT); } +static int stage2_test_clear_young(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags flags) +{ + kvm_pte_t new = ctx->old & ~KVM_PTE_LEAF_ATTR_LO_S2_AF; + + VM_WARN_ON_ONCE(!page_count(virt_to_page(ctx->ptep))); + + if (!kvm_pte_valid(new)) + return 0; + + if (new == ctx->old) + return 0; + + if (kvm_should_clear_young(ctx->arg, ctx->addr / PAGE_SIZE)) + stage2_try_set_pte(ctx, new); + + return 0; +} + +bool kvm_arch_test_clear_young(struct kvm *kvm, struct kvm_gfn_range *range) +{ + u64 start = range->start * PAGE_SIZE; + u64 end = range->end * PAGE_SIZE; + struct kvm_pgtable_walker walker = { + .cb = stage2_test_clear_young, + .arg = range, + .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_SHARED, + }; + + BUILD_BUG_ON(is_hyp_code()); + + kvm_pgtable_walk(kvm->arch.mmu.pgt, start, end - start, &walker); + + return false; +} + phys_addr_t kvm_mmu_get_httbr(void) { return __pa(hyp_pgtable->pgd);