From patchwork Mon Aug 15 07:13:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12943189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B282C282E7 for ; Mon, 15 Aug 2022 07:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E7226B0075; Mon, 15 Aug 2022 03:14:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 86D898D0001; Mon, 15 Aug 2022 03:14:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6225A6B007B; Mon, 15 Aug 2022 03:14:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4F89A6B0075 for ; Mon, 15 Aug 2022 03:14:18 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 29D6F14031F for ; Mon, 15 Aug 2022 07:14:18 +0000 (UTC) X-FDA: 79800963396.05.8C8BBA3 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id AB685C000A for ; Mon, 15 Aug 2022 07:14:17 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id y2-20020a25bb82000000b0068c18dad92aso83440ybg.13 for ; Mon, 15 Aug 2022 00:14:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:cc:to:from:subject:references :mime-version:message-id:in-reply-to:date:from:to:cc; bh=+K6f0/4jX+jKnK8wFp8yh3bewm2bUjE0+0/Xv+vCPNE=; b=qiz6LyvodJhBbvwsZ23v8IyelLyAbuUig5lBBh+S0NmhNx2GvkVG+nJ2wfizF6upPm Ir96pYXZohODg4j+4ncY5LzXhaE630UGQleMABz5UgIwe2hM5LE01eB02XBC9ykNiV5d 6ZY4GCusPmvEWt6ot5fecF9Wj5AGHpa5Yt2hhnrJUp/XAhiXftCe/OgdQuu73p65Nee8 oXTR/1Ph5ZjNsUBM+2wdMXAY1zaOjl13sRpRy2i/fcg2dUyBfN5qcwJL9xdcKSR4ZurP 7zfhadGWpQk50HZuKj9G4tGmAG2tkulbJF7Fg/4miHhwxM8n/k6WvLYGhRSeM81ofS0H Cccg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:from:subject:references :mime-version:message-id:in-reply-to:date:x-gm-message-state:from:to :cc; bh=+K6f0/4jX+jKnK8wFp8yh3bewm2bUjE0+0/Xv+vCPNE=; b=5JGdUDo8Rcy8Yyh/Xp7w2dcLV7fTUVE779Js/q7q1lYe34TgXei6snvyOdsPiJ8528 cy4FfiPnT4o2MgBdgrMAFcvpwRjbBx5zHQ8nx7qIdEKDxkmoLL8NKURgVrErGjnnkvYV w4XHgOW9JVxe2bkeVhwjKtIe7I/sIYyLnWBdIMdCTIzSw9FvGxR3SOdMkO2PeXoH7Ohk 92eedcOr3AoZjWvrM3+OeiXs+w3g/MIN5eFBSxOekFAvovdlLAd41UD1mZmOdMpn2uOg C9jaGapOjKeLRCz/S7EhJ8fCYHQuQA6FMp0I1JgkQrmFcogf1vHot0O7L9Z353iS0gpg LklQ== X-Gm-Message-State: ACgBeo03OaG9AzcV4zwXO5qnBk0UcKClvXSwcTt5YflISEqdgdU71g7A +9DtyeDExelnYeXbwPW4A2LHbJgwKEo= X-Google-Smtp-Source: AA6agR70hzrTklSq7wn9nioaQyvoT/xV0PaM8wHQtwVNUey7E+zWJrPZ9O9d7RYylUL3mYq7qjVywbsIe9A= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:d91:5887:ac93:ddf0]) (user=yuzhao job=sendgmr) by 2002:a05:6902:1206:b0:676:e465:24b1 with SMTP id s6-20020a056902120600b00676e46524b1mr10764993ybu.323.1660547656945; Mon, 15 Aug 2022 00:14:16 -0700 (PDT) Date: Mon, 15 Aug 2022 01:13:21 -0600 In-Reply-To: <20220815071332.627393-1-yuzhao@google.com> Message-Id: <20220815071332.627393-3-yuzhao@google.com> Mime-Version: 1.0 References: <20220815071332.627393-1-yuzhao@google.com> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog Subject: [PATCH v14 02/14] mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG From: Yu Zhao To: Andrew Morton Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Barry Song , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , " =?utf-8?q?Holger_Hoffst=C3=A4tte?= " , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660547657; a=rsa-sha256; cv=none; b=3gEU2ux3ORlrdlvFf4eB8Z/0OOyuO6tOQgjdLoE3M3oLWPaK8kHCrlbivchxffTDUCuK0J 89/Ujc9eMJpH5K4/kGVQfhBvV/naQthoqtq8zOGI2WSlIw6mMP5M0zbQutLA9gSNLl99WN tRB+xkXPCexJ+SKTX8dW0tZ1Jo9oSxg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qiz6Lyvo; spf=pass (imf22.hostedemail.com: domain of 3SPL5YgYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3SPL5YgYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660547657; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+K6f0/4jX+jKnK8wFp8yh3bewm2bUjE0+0/Xv+vCPNE=; b=DQ5iu8AjWu6k1WAXE+PILW0hR8Gha2zV6FpwNj8HCqtwmcBKqR8FmeGAXsJt63YVKdou/X 17Ue9dcouTDYJYg3n6hZtpPoFrTEajwplQLsqM929y0v4szbZNFLWpy7YSZ0D31Ojh8hRM 4fIuzinqOHBVOObuziLSMiIwWLyEhhA= X-Rspam-User: X-Stat-Signature: adfkgp3dccqqyotsdgh4ztq1m9wgkdu3 X-Rspamd-Queue-Id: AB685C000A Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qiz6Lyvo; spf=pass (imf22.hostedemail.com: domain of 3SPL5YgYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3SPL5YgYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam01 X-HE-Tag: 1660547657-697987 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures support the accessed bit in non-leaf PMD entries, e.g., x86 sets the accessed bit in a non-leaf PMD entry when using it as part of linear address translation [1]. Page table walkers that clear the accessed bit may use this capability to reduce their search space. Note that: 1. Although an inline function is preferable, this capability is added as a configuration option for consistency with the existing macros. 2. Due to the little interest in other varieties, this capability was only tested on Intel and AMD CPUs. Thanks to the following developers for their efforts [2][3]. Randy Dunlap Stephen Rothwell [1]: Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3 (June 2021), section 4.8 [2] https://lore.kernel.org/r/bfdcc7c8-922f-61a9-aa15-7e7250f04af7@infradead.org/ [3] https://lore.kernel.org/r/20220413151513.5a0d7a7e@canb.auug.org.au/ Signed-off-by: Yu Zhao Reviewed-by: Barry Song Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffstätte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- arch/Kconfig | 8 ++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index f330410da63a..ebea10a4513e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1416,6 +1416,14 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool +config ARCH_HAS_NONLEAF_PMD_YOUNG + bool + help + Architectures that select this option are capable of setting the + accessed bit in non-leaf PMD entries when using them as part of linear + address translations. Page table walkers that clear the accessed bit + may use this capability to reduce their search space. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig" diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..674d694a665e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -85,6 +85,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index dc5f7d8ef68a..5059799bebe3 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -815,7 +815,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) static inline int pmd_bad(pmd_t pmd) { - return (pmd_flags(pmd) & ~_PAGE_USER) != _KERNPG_TABLE; + return (pmd_flags(pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) != + (_KERNPG_TABLE & ~_PAGE_ACCESSED); } static inline unsigned long pages_to_mb(unsigned long npg) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a932d7712d85..8525f2876fb4 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { @@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp) { diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 95f408df4695..d9095251bffd 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -213,7 +213,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -234,7 +234,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, BUILD_BUG(); return 0; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH