From patchwork Thu Sep 26 01:35:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5812CCFA10 for ; Thu, 26 Sep 2024 01:36:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E293E6B00C6; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDABD6B00C7; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0F766B00C8; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9F4896B00C6 for ; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 574B5141193 for ; Thu, 26 Sep 2024 01:35:34 +0000 (UTC) X-FDA: 82605172188.21.7DED5DE Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 9288B40007 for ; Thu, 26 Sep 2024 01:35:32 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MHZomqoV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314443; a=rsa-sha256; cv=none; b=eLUHZIrprDuW1rfFPEa7fMtAnpWVZq468w4fnzQKj/95Ctu98MniRT3oLfsG2nh8o6sktm jEHmK7DuG0XBIbTAygJOhGNY3MyaWq9EBuSzRihZD2nSPKdeEx+upjsJJn35+MUiMeLNha 0YoxL2Q4iD/xfYEBUs4Sarl+XR/iaLc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MHZomqoV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jwY5XxMaAH9vrNcOxKBlxaXWmYulPCq+P3CKqgHldo8=; b=sLuHDWBz4n494+yjXWoo+W3EMAyC6uWQF2pFi5joLvPAwv+P+4E2M2wKF3NUUbKNYK9prz jyENqpzAfZ8m5/dH5m6AjypXvlDPaBl9VKmJJVEwlOmXZzHyd0Crz+orIm+YA6hhF1ctN2 jk/HDL7XhaoN171PMEV4t+EszWfC0GU= Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-846df072591so165038241.1 for ; Wed, 25 Sep 2024 18:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314531; x=1727919331; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jwY5XxMaAH9vrNcOxKBlxaXWmYulPCq+P3CKqgHldo8=; b=MHZomqoVNtdoenfGdp34L1jF7GSFrnX8hAqAr1VldwXfzvtk9MOdev+N3lpF++E9IB FcwmTLHhP3CjdR1anUh/Phy+SWx697Q1CHE3nxH2tTDpHYxaoy3p2gjOaCMyfVsTZZzx 1M+JbOs7OI0ZvIvVdx+TjW78P1aoh5djg5Qy+aLE2qrI/rZciSSVPMmCRhB7KGWCKoZD Wsm5AXeJK/UxPMCz3zSKM3Yyhgf7heeuAlQuXP1F4v5mekERFju1el09QOgfgH2zZabZ 4zSGELOaJn6tW+vEWOREADV+OCckWC2w4PxO7EnzlTa5v//YrGVmh/idicEDrJaJlxNy Mtaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314531; x=1727919331; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jwY5XxMaAH9vrNcOxKBlxaXWmYulPCq+P3CKqgHldo8=; b=fHbs46pYept6Bxb2JzNNSNsTt7i0+mdB2ZLkQiZHuZN3DWraCUlnK9eQd1JP/JXsSS auNp3B4P2+zmFJCVNS1fIPp9Xj9j1a6OOIlkzs0o4SEZ/BhqUlr5nq5wms4HAOa1p0Kf U19s16QKsA3irRpgXPo3WjQXb545syiPNTIA8agY6OtV12/NNcfh+rgR9/oFmwuH9Kn6 J+qyS4dZxfIIXHXTnTcA0HK4cgvLuz9/BqOBx/0VD2DuGg+bU4Ck5IYyqbchXdTpu3hL +kAjq+KTrXpiRXDkdbLLYtLufwwwkFy/nKQneGDDIaSBW1gpYBjCnzSeKH122LCycNP5 bKLg== X-Forwarded-Encrypted: i=1; AJvYcCUm9d3Pt4VVpjdWRqSBy6OpnMJp27sPaEMN6ARYwnk8sVcqhP0sJJugjCV5ezkmBtfOrN7yrvtFGQ==@kvack.org X-Gm-Message-State: AOJu0YyBK2cKLYfm6AoG5yX5Q4TfH9CK/AbpA5RLMwSOql5LtbuSXuT0 BBRrB5hisbTrwa+PYHqy4qy7+rRKja6fgAzI2D2niMSbHM5VpeAUCt6Jw9LWkyPbsLTow+7wH/q kHbsVhirZ8Qpeu9tvCw== X-Google-Smtp-Source: AGHT+IFR+2qF3anIZYWl2APj1JzmMbbMXKjDESaaFvEMKsw92aYBHlJKDywpCEXIkyk1bOavtsxa/wF5vSbBnBSJ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:ab0:380d:0:b0:846:d5f9:2186 with SMTP id a1e0cc1a2514c-84e83c0a7fdmr10563241.2.1727314529698; Wed, 25 Sep 2024 18:35:29 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:00 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-13-jthoughton@google.com> Subject: [PATCH v7 12/18] mm: Add has_fast_aging to struct mmu_notifier From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: 9288B40007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hsui1z1386bcec9f6e3pggaiiy39bagh X-HE-Tag: 1727314532-697958 X-HE-Meta: U2FsdGVkX19lAuPwsAzT+FQ1hc2Q5wFTxxWeaJjTSTsnDHHNpbOVIH9ZuupXgS/TETUYAL08tpHjx6sMOKAyp1S78URPXLVnZopATohutBhQWMC02VDy7IrudkhUnuWim8b+gV9eE4MkGAxJH5i20ILbEimG1YlrAgD1rwhsWTsoEakIgZCrJ2tsWLsDMM8evWjSc7nwIWCuhMFpPMRpOS2+/OniXowfBXnPb8IMak0QwdN2BKPE/SoAYDLmjrGv5eJ9CvTHbZQy9RZseCiAg3vRtG8ysZXJl8TZTMb8ggLRrUS9iXh3CrvGAIzi6fDm5otNaPeHPO57lTUqWjz+44ZoXLCJq8Qcsom9D6gonPTf7DxGG5dOob5Xk+LVpCvjTXj0nsbIeEyqrFfbhVtO1XURBWAXUW9aZGTmiuZzv8eC4/nDLxFWO/0FXw+9ghygBZ24nsmxn6PyJvzVr0v29Va/S4uzuNva4+rKIUHS7T77XGVnlEM7fdCTjo50uVPiObLX3O2tUbpnibH1F+/R/nhpbiZf+D251HN5FaBBPQy7hrav4sDeLztxFOzbSdjNkQrdvWdCXQHPL3QqDwPfikJCOiUzpksdslzwBiiXx1/mfgIoytbPOzi4BV2MGFBQdD8j8wR8tHWE1/UshH09M2/2QV2VfqtM+jFbWjA7HG4pMyHhBb8s9xERy9+1YVwZEvCRl945I0j2IyFxhsTo59QQy804z5kWzZDja3+icPTODBef+hUYg8QRqUvIAIQBaKTxNGBxIzaliMNU/ggOpzU/jjVA4CRM/njFOI3OqqIDfFPCk1KJnkIhCj8IjdYkTt5qi8GV1lVtioF/2NvOKpsaV19QifgrEPFbD63tluK9xBbuv1W97GyFbxK3miUCHkpC8kTqLt9mv/qQ9Kw9w9tG8T+vk0HaoIYaEw9p5DV+ARzhLGm701ynlZnGOZ36Ics0o6SP2m4x4KLrof4 fh3MeGs4 1X6QOcAT8HmeWTFQUjmdtf5oqpuznTFxyhkcGA4zISMFg32O6Utc1D+C6hCb95WNViAsp07f53bdZvzEKzMwwVZynJaHZtpqXDuZoNrD0zsjqWdPhctvdZAQH78qbml1mtF6rdJkmZ92SdfgnAYORffOBkdVYGSJtvBxdWNOEa4fqxMia4UO7zM+6Ko9sZqbNuG7U0nSSVNqlskimtZA6psriZehHbJfDX2R9Qctkw8qBY20td789mZ1TIYH0gGzuDYKjvRHBvZc+zuWdCztS57E+JQjyLkBUIFXSGrMaBq5MBnwvzpErHTRZNbOgLdB7KWdl/yrE6tCHrGvvPQDg/4MzWujSpO2SZOlu28/seHNZPYoDz7qEmXawLrvRl1VYAEJFndGoydDrvkRu8/Bb0nWpAhIMI4BpzjkdhyUEpO5nFbU08YU0iSZnau6IUGXlOq+sX5ZGaAEDmeK5rlouR6laqUgh7YkcM6IRadooca9mgDyDGqbFIYRaby7TEouvXrY1Jv2Bg6WwT8H7WXJfO/OMjM3u29cD+tYCo3SazuEaOzHmhavy7BQqHHBw7vSTp2liMXTfXFpEEMalpwY6Vxfeuwx2XppNJCPe/pTcoVxD/eaBt0afOmmGYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: has_fast_aging should be set by subscribers that non-trivially implement fast-only versions of both test_young() and clear_young(). Fast aging must be opt-in. For a subscriber that has not been enlightened with "fast aging", the test/clear_young() will behave identically whether or not fast_only is given. Have KVM always opt out for now; specific architectures can opt-in later. Given that KVM is the only test/clear_young() implementer, we could instead add an equivalent check in KVM, but doing so would incur an indirect function call every time, even if the notifier ends up being a no-op. Add mm_has_fast_young_notifiers() in case a caller wants to know if it should skip many calls to the mmu notifiers that may not be necessary (like MGLRU look-around). Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 14 ++++++++++++++ mm/mmu_notifier.c | 20 ++++++++++++++++++++ virt/kvm/kvm_main.c | 1 + 3 files changed, 35 insertions(+) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index e2dd57ca368b..37643fa43687 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -231,6 +231,7 @@ struct mmu_notifier { struct mm_struct *mm; struct rcu_head rcu; unsigned int users; + bool has_fast_aging; }; /** @@ -383,6 +384,7 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_test_young(struct mm_struct *mm, unsigned long address); +extern bool __mm_has_fast_young_notifiers(struct mm_struct *mm); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, @@ -428,6 +430,13 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm) +{ + if (mm_has_notifiers(mm)) + return __mm_has_fast_young_notifiers(mm); + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { @@ -619,6 +628,11 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm) +{ + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 8982e6139d07..c405e5b072cf 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -382,6 +382,26 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, return young; } +bool __mm_has_fast_young_notifiers(struct mm_struct *mm) +{ + struct mmu_notifier *subscription; + bool has_fast_aging = false; + int id; + + id = srcu_read_lock(&srcu); + hlist_for_each_entry_rcu(subscription, + &mm->notifier_subscriptions->list, hlist, + srcu_read_lock_held(&srcu)) { + if (subscription->has_fast_aging) { + has_fast_aging = true; + break; + } + } + srcu_read_unlock(&srcu, id); + + return has_fast_aging; +} + int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 7d5b35cfc1ed..f6c369eccd2a 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -930,6 +930,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { static int kvm_init_mmu_notifier(struct kvm *kvm) { kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops; + kvm->mmu_notifier.has_fast_aging = false; return mmu_notifier_register(&kvm->mmu_notifier, current->mm); }