From patchwork Sat Mar 13 07:57:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12136577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC9A9C433DB for ; Sat, 13 Mar 2021 07:58:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6053564F1E for ; Sat, 13 Mar 2021 07:58:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6053564F1E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DAB186B0073; Sat, 13 Mar 2021 02:58:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D824C6B0074; Sat, 13 Mar 2021 02:58:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4D726B0075; Sat, 13 Mar 2021 02:58:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id A75E66B0073 for ; Sat, 13 Mar 2021 02:58:05 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 663B8824805A for ; Sat, 13 Mar 2021 07:58:05 +0000 (UTC) X-FDA: 77914097730.24.4E29040 Received: from mail-qt1-f202.google.com (mail-qt1-f202.google.com [209.85.160.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 6C036C5F0 for ; Sat, 13 Mar 2021 07:58:01 +0000 (UTC) Received: by mail-qt1-f202.google.com with SMTP id v19so19292153qtw.19 for ; Fri, 12 Mar 2021 23:58:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9FOrNXENvEBq+xjcXdvWYV76Tj/52PeZfpdHZpBdeTw=; b=CZNFSLn1Vr+7g91u2WdSPY2RASOoMeGVnEu8XS9ogwps7Gq+7F5umE3fWsyowJpBWD /BpSgazEV0uTx/142ccxmLjj6Tc5kR7KGsb79Ptj4azaGNuJBT032A7MXAqita6Xkryl 6IanFkwVS4tC+SsZAZtk+kuQzdp1pO5Pnx+cXwdQzEVQmCkWjuUnjKzoEPGf4IlnkcFb QOa5YU3bEwcfmAFIwkc2tWsdZ2h8rKKycxzT/zHBKI265GiGZGNofgaHIdU33DduoJGc Q3y72dvencUMJG5nccrm3sqMC+9s98wb5AiyP7gwQH8cNu/oUBPDF/k1zMnv1QbL0y4q Wkmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9FOrNXENvEBq+xjcXdvWYV76Tj/52PeZfpdHZpBdeTw=; b=t2A5QYSJ+lm09GuTY8d8H1pzkaJ66w5uFl+JmHDoAbRYLAI3ifnP5w4HJMxLuQ9cMB srLxp9a4J26T9PyKKyrayFw2iaic83e6fNkYHvugYL7CGF+LpiI0Uvhf8sWkAbHHcWKE x39Driiue75eeR8Q8yJfu73vsHHDPyDPN+Lceh4GpT+dr0BKNZSfeaJCp+dG9gg7Jdew eVc5TBZNO2PLORFy+X0fJISj6nV28u6g+ba65r1qj76c65V+Qz5DF1prRqwi1r9qheqI QtNEPEUpq59xEjbdgPMGLJmtHJGARVLu2nxKqkllrwFWCQ18fKdNIwY1gEXDRlCHNV+f S12A== X-Gm-Message-State: AOAM5333m0YKur3g80yd3kKR0J90n+nhlVj+0XIMS5u0RZ2N7lu1JgIT +ckFGQWzjER1AUEZ+flM2n0T0JRQJPvm51n99BKLfk9TR3uHrR8EPq8HKYFmL2j9+TLKiDUXndz pAU0QyqQoQWlyBIrSq7vlfNmdBcjEhFOBTfj6IHGpFMHePSkV67xAXqTX X-Google-Smtp-Source: ABdhPJx3v9fo9fGIto8kHzW5LTGjVA6UTKRtTeuo4NwnDLlQop9mIBVZK8pHZLGtADbIHe5kSQ6HeS7hVrU= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f931:d3e4:faa0:4f74]) (user=yuzhao job=sendgmr) by 2002:a0c:cb0c:: with SMTP id o12mr15467752qvk.54.1615622284101; Fri, 12 Mar 2021 23:58:04 -0800 (PST) Date: Sat, 13 Mar 2021 00:57:39 -0700 In-Reply-To: <20210313075747.3781593-1-yuzhao@google.com> Message-Id: <20210313075747.3781593-7-yuzhao@google.com> Mime-Version: 1.0 References: <20210313075747.3781593-1-yuzhao@google.com> X-Mailer: git-send-email 2.31.0.rc2.261.g7f71774620-goog Subject: [PATCH v1 06/14] mm, x86: support the access bit on non-leaf PMD entries From: Yu Zhao To: linux-mm@kvack.org Cc: Alex Shi , Andrew Morton , Dave Hansen , Hillf Danton , Johannes Weiner , Joonsoo Kim , Matthew Wilcox , Mel Gorman , Michal Hocko , Roman Gushchin , Vlastimil Babka , Wei Yang , Yang Shi , Ying Huang , linux-kernel@vger.kernel.org, page-reclaim@google.com, Yu Zhao X-Stat-Signature: 1bqnbgbwsdw9yr9uujbhjze4edyqqm7n X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6C036C5F0 Received-SPF: none (flex--yuzhao.bounces.google.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from="<3jHBMYAYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com>"; helo=mail-qt1-f202.google.com; client-ip=209.85.160.202 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615622281-93532 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures support the accessed bit on non-leaf PMD entries (parents) in addition to leaf PTE entries (children) where pages are mapped, e.g., x86_64 sets the accessed bit on a parent when using it as part of linear-address translation [1]. Page table walkers who are interested in the accessed bit on children can take advantage of this: they do not need to search the children when the accessed bit is not set on a parent, given that they have previously cleared the accessed bit on this parent in addition to its children. [1]: Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3 (October 2019), section 4.8 Signed-off-by: Yu Zhao --- arch/Kconfig | 8 ++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 2 +- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 16 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 2bb30673d8e6..137446d17732 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -783,6 +783,14 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD bool +config HAVE_ARCH_PARENT_PMD_YOUNG + bool + help + Architectures that select this are able to set the accessed bit on + non-leaf PMD entries in addition to leaf PTE entries where pages are + mapped. For them, page table walkers that clear the accessed bit may + stop at non-leaf PMD entries when they do not see the accessed bit. + config HAVE_ARCH_HUGE_VMAP bool diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2792879d398e..b5972eb82337 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -163,6 +163,7 @@ config X86 select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 + select HAVE_ARCH_PARENT_PMD_YOUNG if X86_64 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 select HAVE_ARCH_WITHIN_STACK_FRAMES diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index a02c67291cfc..a6b5cfe1fc5a 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -846,7 +846,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) static inline int pmd_bad(pmd_t pmd) { - return (pmd_flags(pmd) & ~_PAGE_USER) != _KERNPG_TABLE; + return ((pmd_flags(pmd) | _PAGE_ACCESSED) & ~_PAGE_USER) != _KERNPG_TABLE; } static inline unsigned long pages_to_mb(unsigned long npg) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f6a9e2e36642..1c27e6f43f80 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG) int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { @@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp) { diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5e772392a379..08dd9b8c055a 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -193,7 +193,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -214,7 +214,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, BUILD_BUG(); return 0; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HAVE_ARCH_PARENT_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH