From patchwork Thu Nov 11 04:15:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12614053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 923DCC4332F for ; Thu, 11 Nov 2021 04:15:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 47539619E1 for ; Thu, 11 Nov 2021 04:15:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 47539619E1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E894E6B00B3; Wed, 10 Nov 2021 23:15:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E104A6B00B4; Wed, 10 Nov 2021 23:15:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BED066B00B5; Wed, 10 Nov 2021 23:15:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0085.hostedemail.com [216.40.44.85]) by kanga.kvack.org (Postfix) with ESMTP id A70206B00B3 for ; Wed, 10 Nov 2021 23:15:19 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6D9757FAD1 for ; Thu, 11 Nov 2021 04:15:19 +0000 (UTC) X-FDA: 78795334758.02.7EB319C Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf06.hostedemail.com (Postfix) with ESMTP id 22563801A8A0 for ; Thu, 11 Nov 2021 04:15:19 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id h62-20020a25a544000000b005c5d9b06e57so7484658ybi.6 for ; Wed, 10 Nov 2021 20:15:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=UqQ1CAHDKzNIHLLK4QSZngjkwRJTSigVp7v0+NSGo7s=; b=iCTrg5hHtKFPWug2bcoGi1y0JSa/3ibgMhmXVbdRD8JMGzVbhCPL8gwxwh6Hd6I0m4 cxsZfNdVwtSrDVKdEp3d9cqbPMulbzip+86pdgV4njqKhDOi3BIQnPGMz2aT0ZKPV96n xK+tc14myylCnDee3S0ioGi+PpuCZd9TMCv5IHtHpbPhSglKW+PxgU9vaBXFQI4HSxux t/DwjX34g7G80VP9EDeyeShh1C+RWvkNa6UjlGUFqMsDUdu/+mJdn5Mu1DSLhFDu8QPN mMRo4OdwQjqUK31qpe28XbPEKhUEnh0Yicb9/nDJgtLVccFoGmVcmDKfZD7e9ulZQDEX 4kCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UqQ1CAHDKzNIHLLK4QSZngjkwRJTSigVp7v0+NSGo7s=; b=vPSxAZrDT8O2LiuyxEO61Zq+9QlYtxyoE0JMih/CvYBUZJP0JI0oWmD5gKQ99wrN9l CM8pIpdhSXyZumYI7lxEmgM39rDguMGd51fF05oNY36BGOLfPWLqW35gJ/0fmnEg3jjg JwR1Y0tL6eyZasgtBOEne7L+bZLgE0CC6fjRThdPUpD+98iIzPuKGRyNzBOUVsryXfq4 QrhsjNK8tEciTxxXrBmgDB1Rsc5c/+ZUwgcYiSI4RdtfvsEiELN7a1Trv0bm61tHrkz4 OEXm5o1aJhmh9u/jyJyoi70gvK3TSGWQ0M1f3Szgk9btYmYJxoytwkWyjvtCnFNVo4Rg fScg== X-Gm-Message-State: AOAM532KIMWdK2PWeuQCS4/A23rBQFbh/hkMOfOY0eCUnCwwW/ZApuFW J/8FzhOx2HkU6IMfa4CRARpBt9ufJ7V0YnzeMWH49I3DM4mA18g2Fw0a1k6OmX+DI6ZkQfLzXrO DgenE6Oz+ochpDhVHTjdN+2HubMVxDENxoN8AKW1fm0RSjDPeZQPScA8h X-Google-Smtp-Source: ABdhPJwuPl9PJhP6Upxe5eqWLpjhaqFw0snAdbggMzVCEnomOiwgAM1BKhvsCadbArbOq/ns603ARh86ExE= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:346b:bb72:659e:f91c]) (user=yuzhao job=sendgmr) by 2002:a25:c70f:: with SMTP id w15mr5226488ybe.167.1636604118332; Wed, 10 Nov 2021 20:15:18 -0800 (PST) Date: Wed, 10 Nov 2021 21:15:02 -0700 In-Reply-To: <20211111041510.402534-1-yuzhao@google.com> Message-Id: <20211111041510.402534-3-yuzhao@google.com> Mime-Version: 1.0 References: <20211111041510.402534-1-yuzhao@google.com> X-Mailer: git-send-email 2.34.0.rc0.344.g81b53c2807-goog Subject: [PATCH v5 02/10] mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG From: Yu Zhao To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, page-reclaim@google.com, holger@applied-asynchrony.com, iam@valdikss.org.ru, Yu Zhao , Konstantin Kharlamov X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 22563801A8A0 X-Stat-Signature: 8anc758pan5cdzxj6fh6ryww5gfr7ed6 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=iCTrg5hH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 31piMYQYKCE0D9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=31piMYQYKCE0D9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com X-HE-Tag: 1636604118-385890 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures support the accessed bit on non-leaf PMD entries, e.g., x86_64 sets the accessed bit on a non-leaf PMD entry when using it as part of linear address translation [1]. As an optimization, page table walkers who are interested in the accessed bit can skip the PTEs under a non-leaf PMD entry if the accessed bit is cleared on this PMD entry. Although an inline function may be preferable, this capability is added as a configuration option to look consistent when used with the existing macros. [1]: Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3 (June 2021), section 4.8 Signed-off-by: Yu Zhao Tested-by: Konstantin Kharlamov --- arch/Kconfig | 9 +++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 18 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 8df1c7102643..7392fcc88777 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1288,6 +1288,15 @@ config ARCH_HAS_ELFCORE_COMPAT config ARCH_HAS_PARANOID_L1D_FLUSH bool +config ARCH_HAS_NONLEAF_PMD_YOUNG + bool + depends on PGTABLE_LEVELS > 2 + help + Architectures that select this are able to set the accessed bit on + non-leaf PMD entries in addition to leaf PTE entries where pages are + mapped. For them, page table walkers that clear the accessed bit may + stop at non-leaf PMD entries if they do not see the accessed bit. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig" diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index d9830e7e1060..245742b79be9 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -84,6 +84,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG if X86_64 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index c60b16f8b741..36205ec0acac 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -817,7 +817,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) static inline int pmd_bad(pmd_t pmd) { - return (pmd_flags(pmd) & ~_PAGE_USER) != _KERNPG_TABLE; + return (pmd_flags(pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) != + (_KERNPG_TABLE & ~_PAGE_ACCESSED); } static inline unsigned long pages_to_mb(unsigned long npg) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3481b35cb4ec..a224193d84bf 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { @@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp) { diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a64c6124c137..afb9004b778a 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -211,7 +211,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -232,7 +232,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, BUILD_BUG(); return 0; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH