From patchwork Fri May 26 23:44:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B582EC7EE2C for ; Fri, 26 May 2023 23:44:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C389900003; Fri, 26 May 2023 19:44:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8239C280001; Fri, 26 May 2023 19:44:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69DD3900004; Fri, 26 May 2023 19:44:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 569BF900003 for ; Fri, 26 May 2023 19:44:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 24F8F40FBD for ; Fri, 26 May 2023 23:44:46 +0000 (UTC) X-FDA: 80834038572.06.E821FBE Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 4F2F710000B for ; Fri, 26 May 2023 23:44:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=HaeURiP8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144684; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=74b92Wk/fESh4Us3jjAt9ER/dw3pjjcYNf6aM3EASmw=; b=Epe3oOD1HOfFasKpFMWMz7K0+Q1Nm9GTiYYEnGRQWfMD0vEog6ruhOJr4I7yqFZHYaHRgf Z58DQMB1EPSE/FRWJX/5ds81b//ZmlHi+mjysTXHVF5b/7oEkTtjoAA7CL5ns/pCCQcp15 TV+W2aBJUDkC1JLKrmR5GkGnD409b7o= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=HaeURiP8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144684; a=rsa-sha256; cv=none; b=ftQ/Z1o2SxHIfGTDdoTCRPNgz3I7CHEQZZV3yVH4YLtJ4CCKdpvso7mXQrIOl0bZyq9Fws EqrNBDbbZCTUQTehkd4yHHLV4+iv+yf5vWL5TdeAMhrxnLVQEBHJFf/HY18wCU6pBGwbxw TFe+kBeN5wPXviO1wYx0XvukM5snJi0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-565d1b86a64so352957b3.3 for ; Fri, 26 May 2023 16:44:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144683; x=1687736683; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=74b92Wk/fESh4Us3jjAt9ER/dw3pjjcYNf6aM3EASmw=; b=HaeURiP8Vv+S5DfSxjNudO8jlQJ3LkpTBadHgzP5bqFZBs2b3foI2By3BUQ7HvtvPw W7QXj6uSkaA/QUNh4WGZeI/+nH65GkUhIUrM4lyZvIxfASIS2yU+XXbU6dLJN4ghefvO HpvDwxabNs2VhwI1FmYErw8clASpd/3GLJz6qPYetihMIY0W6vexsuS/yVGVZpRWw3il an2SbsIcj+SL6k3rb4VmBY4XZ/nf3i0WPF6L2efKh0GQc6ZIGr6GiKv97NKXrU9K5c92 qSgR2Y5nDu/cn+wzryf1LV/5CRQgKNqjczW+qgHgoPm+B6opAwXo+9yxImX5iYvtFZ8h Yz0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144683; x=1687736683; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=74b92Wk/fESh4Us3jjAt9ER/dw3pjjcYNf6aM3EASmw=; b=Gw/OcgjcKPVyHW3HjO4rz2FkW9F5Xza1R4rqbLlgmKrZyy9WSR0IHRjwfjuOw/wiBm GXtbXm6e6HYit2Xhd6fxmcemyA5YWOx8j1OxlFaeoesSlI8Yc3TAzEiir2zD5Tn+X7O6 T5mqujl4rXbxFRfIpuAaQ3lIv8mKgmoTdJTJULrd4zaGdbI1TvaWJ8VzpUK7pco7JPES +MyxVDCNXRih0Xmh/cMj66nmh0fUWlbzcayN4KjkskCA4QvT5Ofk+gAt/bCht1ohNqvi 02mddZ0TTp2WM3nLjC3FJgxgukK1ktMFmNBGSa07Lir1i5KDBvvZmvJUxUp4HymmqRLR Oz6w== X-Gm-Message-State: AC+VfDxUnldYvOW/ylkSj5HLZ4eXzG/OH5BjPYhkrD7MNOghkZX4oJC2 osBdr1R03oQ6B9h2ID2MdEZsHc0mZfM= X-Google-Smtp-Source: ACHHUZ77t8VJ8lKvCLJIppCq4GXisPABEajtkBmVe9W7YGxSZPGguPwTv3qLn5OBBx4lDI9spcGEUr4VdcA= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a81:e608:0:b0:561:c4ef:1def with SMTP id u8-20020a81e608000000b00561c4ef1defmr2027484ywl.0.1685144683190; Fri, 26 May 2023 16:44:43 -0700 (PDT) Date: Fri, 26 May 2023 17:44:27 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-3-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 02/10] mm/kvm: use mmu_notifier_ops->test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 4F2F710000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ggmzopx9hyecmkbrhkethei9sgarna8t X-HE-Tag: 1685144684-830906 X-HE-Meta: U2FsdGVkX1+mEQ3Fsskp/VW0KwdDjEKRySZ/snTVjMWwdS17uWAFj7t5/UaocLBlr+Kfa5F4PaHNcwz8hxq0a9Ce+bgdalshkocwX9sMOBWkxxzVzX+rhwVl//VoPcbFyRk0oluPtdrHy+JY1ll8Lc+LtvjPi3b/4H3bltQcSg6zxuuWjvWbv04oG4t54I/0unareSjFWw0KL2dLUQHbDZsOWswXRT1ZxqOY9PLBPrsCSFczwFY0uH17DwLeoFxDOLVckNfJ2OntAUHJjdjn+8dlNobFEK6Jpz11uBDHW17SJWXjgZ2KhPMLFfyfLKaCpBcjKQzURDPe7/eiaysHV4H9r/qkyIM1eimT8KnQS9ofuVjaGMHL++MZ5K/1vfxC0h6Sc5dP4Bi1QYVjBlCsJA0H8ZMMaocU50nd3jZCHKi/zNsUB6UQT1UFAsCu/4Zc9uLo3ozxVssSNAbit5qWtVWtt6MLCavSKekd9f3yLzv/lwUYqVCpJYETIHqW0y7UMMMSFTQVafoL22lfnmCyRpb04zgha9TzxOJG9ZInw4V9JOlx/sP7Vq8ArYg1idJ/P9MVczqD1GVm6H0gltXG3fmBRETb3eLvbiSZqvMkd44bBudET5drjYW2v+y4SkrO7a6egw7fx0X1Z6fgiY+K3ur3MJeohRLDUYK8gPsKAALZMQu9TWE+/2nA2HQyAMVQ4wkPh1wcQ5FcZT2yDU9xkhZVrtn5kMAn61xxdqXpPcDxvcfi75bwSyIAgk/uL7LrOEFXzraJyCGt1POSEjHz5lAWivGX/Hm1zA7HoDxXCCoVw12xnkAJ9ndKrqCGT7+rGyRg44wxA+0vKxJIfEjgsV3VYeWyH2L+d9Wf90g8GRxRE+RFqw6lJwALCU9BGsuc7eFDebMIZh2dNM/G+136ymMJnDvCSpE8/X9Ui/At4Mn3+q0uo2+ROLCZgx3HhgGNT87PAeSJSACJGV7YIGH B5khJO9F V/dfdZHtWweiQKz96A1mhLzc9rStRFNdjTlkmwc1gXtI6r03Wd/wi4e2+yPxc11HtOrjIN9KwnmqUNkvVV8J8gexIT9KqKKUh7rZzLIEjuhEACvOmiJSu/7ggnpK5lBsk1heo4Krtn5/ENWjCEpdBainuDWSjUT62G82nEBqJfPlsfBU+rhlGsYU1WT68FfKuKNjpy3R3tK+QXh6DlOE2+NNGx7GRJ3aTdftysqgLLT/U2iv/OuhflU6hr94O9ssb7glemNIPQ5F7mGnDamA83yu9T8PN5t/376LItjo+e0r9FluKRrAdsTZPRXxlvRpJ9ip5gompVI8ut6CxNevFIVORpYp+1adb85PqoyfuywfOg2j2/mueNvZ57DvCXP0oVCV5pQMn3CmBiTEiP/JPlKi56n50lP1aYIa6E4dzA3ucY9hFHdI/TbfNuLclMY08ytT6vwAMmm69LsAiAKVpgYFy+mTopxUUSL5c3Rg/zbU9Hq57qKZAeSIKQ4XPQaDN78JgGtH1Rgd4JvVfMZn55dCoKfofZ2K2zveGbkP5ij1WDX7OBNXNpVvMm7FNUmag9vT4K/Ids1fpADieuxQwra7B5IGvODM5uPk15tcTciRV4Ex0350plop3hhpD+sH/QKIlsdMhHAqTyKz2kAaXHtOK/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace test_young() and clear_young() with test_clear_young(). Signed-off-by: Yu Zhao --- include/linux/mmu_notifier.h | 29 ++----------------- include/trace/events/kvm.h | 15 ---------- mm/mmu_notifier.c | 42 ---------------------------- virt/kvm/kvm_main.c | 54 ------------------------------------ 4 files changed, 2 insertions(+), 138 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index dfdbb370682d..c8f35fc08703 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -104,26 +104,6 @@ struct mmu_notifier_ops { unsigned long start, unsigned long end); - /* - * clear_young is a lightweight version of clear_flush_young. Like the - * latter, it is supposed to test-and-clear the young/accessed bitflag - * in the secondary pte, but it may omit flushing the secondary tlb. - */ - int (*clear_young)(struct mmu_notifier *subscription, - struct mm_struct *mm, - unsigned long start, - unsigned long end); - - /* - * test_young is called to check the young/accessed bitflag in - * the secondary pte. This is used to know if the page is - * frequently used without actually clearing the flag or tearing - * down the secondary mapping on the page. - */ - int (*test_young)(struct mmu_notifier *subscription, - struct mm_struct *mm, - unsigned long address); - int (*test_clear_young)(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end, bool clear, unsigned long *bitmap); @@ -393,11 +373,6 @@ extern void __mmu_notifier_release(struct mm_struct *mm); extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long start, unsigned long end); -extern int __mmu_notifier_clear_young(struct mm_struct *mm, - unsigned long start, - unsigned long end); -extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); extern int __mmu_notifier_test_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end, bool clear, unsigned long *bitmap); @@ -437,7 +412,7 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_test_clear_young(mm, start, end, true, NULL); return 0; } @@ -445,7 +420,7 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_clear_young(mm, address, address + 1, false, NULL); return 0; } diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 3bd31ea23fee..46c347e56e60 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -489,21 +489,6 @@ TRACE_EVENT(kvm_age_hva, __entry->start, __entry->end) ); -TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), - - TP_STRUCT__entry( - __field( unsigned long, hva ) - ), - - TP_fast_assign( - __entry->hva = hva; - ), - - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) -); - #endif /* _TRACE_KVM_MAIN_H */ /* This part must be outside protection */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 7e6aba4bddcb..c7e9747c9920 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -382,48 +382,6 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, return young; } -int __mmu_notifier_clear_young(struct mm_struct *mm, - unsigned long start, - unsigned long end) -{ - struct mmu_notifier *subscription; - int young = 0, id; - - id = srcu_read_lock(&srcu); - hlist_for_each_entry_rcu(subscription, - &mm->notifier_subscriptions->list, hlist, - srcu_read_lock_held(&srcu)) { - if (subscription->ops->clear_young) - young |= subscription->ops->clear_young(subscription, - mm, start, end); - } - srcu_read_unlock(&srcu, id); - - return young; -} - -int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) -{ - struct mmu_notifier *subscription; - int young = 0, id; - - id = srcu_read_lock(&srcu); - hlist_for_each_entry_rcu(subscription, - &mm->notifier_subscriptions->list, hlist, - srcu_read_lock_held(&srcu)) { - if (subscription->ops->test_young) { - young = subscription->ops->test_young(subscription, mm, - address); - if (young) - break; - } - } - srcu_read_unlock(&srcu, id); - - return young; -} - int __mmu_notifier_test_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end, bool clear, unsigned long *bitmap) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 31ee58754b19..977baaf1b248 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -674,25 +674,6 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, return __kvm_handle_hva_range(kvm, &range); } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - hva_handler_t handler) -{ - struct kvm *kvm = mmu_notifier_to_kvm(mn); - const struct kvm_hva_range range = { - .start = start, - .end = end, - .pte = __pte(0), - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .on_unlock = (void *)kvm_null_fn, - .flush_on_ret = false, - .may_block = false, - }; - - return __kvm_handle_hva_range(kvm, &range); -} static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long address, @@ -854,39 +835,6 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, return kvm_handle_hva_range(mn, start, end, __pte(0), kvm_age_gfn); } -static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, - unsigned long end) -{ - trace_kvm_age_hva(start, end); - - /* - * Even though we do not flush TLB, this will still adversely - * affect performance on pre-Haswell Intel EPT, where there is - * no EPT Access Bit to clear so that we have to tear down EPT - * tables instead. If we find this unacceptable, we can always - * add a parameter to kvm_age_hva so that it effectively doesn't - * do anything on clear_young. - * - * Also note that currently we never issue secondary TLB flushes - * from clear_young, leaving this job up to the regular system - * cadence. If we find this inaccurate, we might come up with a - * more sophisticated heuristic later. - */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); -} - -static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long address) -{ - trace_kvm_test_age_hva(address); - - return kvm_handle_hva_range_no_flush(mn, address, address + 1, - kvm_test_age_gfn); -} - struct test_clear_young_args { unsigned long *bitmap; unsigned long end; @@ -969,8 +917,6 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, .clear_flush_young = kvm_mmu_notifier_clear_flush_young, - .clear_young = kvm_mmu_notifier_clear_young, - .test_young = kvm_mmu_notifier_test_young, .test_clear_young = kvm_mmu_notifier_test_clear_young, .change_pte = kvm_mmu_notifier_change_pte, .release = kvm_mmu_notifier_release,