From patchwork Wed Jul 14 09:17:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12376353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A36A9C11F66 for ; Wed, 14 Jul 2021 09:21:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 438D4613A9 for ; Wed, 14 Jul 2021 09:21:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 438D4613A9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8EC296B0085; Wed, 14 Jul 2021 05:21:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C2AC6B0088; Wed, 14 Jul 2021 05:21:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73D386B008C; Wed, 14 Jul 2021 05:21:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0032.hostedemail.com [216.40.44.32]) by kanga.kvack.org (Postfix) with ESMTP id 542A46B0085 for ; Wed, 14 Jul 2021 05:21:01 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 562BB1857A0BD for ; Wed, 14 Jul 2021 09:21:00 +0000 (UTC) X-FDA: 78360649080.17.CD4A789 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf12.hostedemail.com (Postfix) with ESMTP id 580A610000A7 for ; Wed, 14 Jul 2021 09:20:59 +0000 (UTC) Received: by mail-pg1-f171.google.com with SMTP id 62so1793359pgf.1 for ; Wed, 14 Jul 2021 02:20:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pXaZpxV+G9LRpUwF+VG87MLwHGWNlv6eVU68Ll1BDlg=; b=g9aguBnwL1920OCwUhqI3Lc2IO3wEKFUYXyt4p/cwHmuxEILVxh9mCSu9o0GwgtZO/ wDxqvMQ1ZbiH6IKJHUiIPV3UpcITtfYmpdPxud8pNaCiUvW4xu36P1GqJ+kJzTmLLVcd RdbstYinUCtwGZBtxTWq4GBGizTXYTsDQD49emXEPQ6A5PHpMQDyKUurFxmjWRe29RnC 1/bok7yNN2gB730SHadWTJnC4V+qr6ewnzyR7j2Gb1qse6sxVDy4syZ7xfv5p50v2NTB ZstpW+Ku5pkS4IAB4iKj2loqbQYjiSEszi1mTN7NAx/O9PQeL6M5hoYhAyTIW/3Gewp8 n2Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pXaZpxV+G9LRpUwF+VG87MLwHGWNlv6eVU68Ll1BDlg=; b=DRfkNnA4dKAD11InvIIKN7qDYrWykFhO+9pJwgBWHWrDWDuU5UyKX2klRV3Ivby7VP S+rQaAi6rzfnqPlauelxqiv46jRssxB5Hr+UtPIUMd38bKadwnfDJxYGhuF+HyJmF+Ys 9QiNfaPupTFBXGDByp84eUplSUrMBYDfCEVANF8q4oVhqWrmBPwMCb7v9HkI6ExhFtpM KXErnug4dBjVGqMyG8e7TiTWxJ6kO3Qx9Inx21jBoBXEPHgBFrkET2yrKziluv/PLomZ j5fbK2RlW1tG+LAxZOxBQH9tGHVeP4PuYeum8LLtruDjnek/K1t86/gDBNcE7d3J+qM9 Z+pg== X-Gm-Message-State: AOAM533dOo6Wz3Rs1Fa0FKa1fH3ahr7yV0hzWfvGkuS9jefADvKjsTdq XjjSHd1BmvtSf7Bb8lirrs67zA== X-Google-Smtp-Source: ABdhPJxjjEP8yL9wM+fk5lNc/bRm4F/w/VpzdXNoCSV1b00VidI9N1OgiQuYPqXD7Wap1+ynN2apBg== X-Received: by 2002:a63:de45:: with SMTP id y5mr8563948pgi.261.1626254458547; Wed, 14 Jul 2021 02:20:58 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id k19sm1742540pji.32.2021.07.14.02.20.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Jul 2021 02:20:58 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH 1/5] mm: introduce PAGEFLAGS_MASK to replace ((1UL << NR_PAGEFLAGS) - 1) Date: Wed, 14 Jul 2021 17:17:56 +0800 Message-Id: <20210714091800.42645-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210714091800.42645-1-songmuchun@bytedance.com> References: <20210714091800.42645-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 580A610000A7 X-Stat-Signature: 3gsxzwibur1ffcq8f9tbygduma6pw8oq Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=g9aguBnw; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1626254459-256401 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of hard-coding ((1UL << NR_PAGEFLAGS) - 1) everywhere, introducing PAGEFLAGS_MASK to make the code clear to get the page flags. Signed-off-by: Muchun Song --- include/linux/page-flags.h | 4 +++- include/trace/events/page_ref.h | 4 ++-- lib/test_printf.c | 2 +- lib/vsprintf.c | 2 +- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 5922031ffab6..358d3f6fa976 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -178,6 +178,8 @@ enum pageflags { PG_reported = PG_uptodate, }; +#define PAGEFLAGS_MASK (~((1UL << NR_PAGEFLAGS) - 1)) + #ifndef __GENERATING_BOUNDS_H static inline unsigned long _compound_head(const struct page *page) @@ -859,7 +861,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page) * alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON) + (~(PAGEFLAGS_MASK | __PG_HWPOISON)) #define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2) diff --git a/include/trace/events/page_ref.h b/include/trace/events/page_ref.h index 5d2ea93956ce..643b1b4e9f27 100644 --- a/include/trace/events/page_ref.h +++ b/include/trace/events/page_ref.h @@ -38,7 +38,7 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, TP_printk("pfn=0x%lx flags=%s count=%d mapcount=%d mapping=%p mt=%d val=%d", __entry->pfn, - show_page_flags(__entry->flags & ((1UL << NR_PAGEFLAGS) - 1)), + show_page_flags(__entry->flags & ~PAGEFLAGS_MASK), __entry->count, __entry->mapcount, __entry->mapping, __entry->mt, __entry->val) @@ -88,7 +88,7 @@ DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, TP_printk("pfn=0x%lx flags=%s count=%d mapcount=%d mapping=%p mt=%d val=%d ret=%d", __entry->pfn, - show_page_flags(__entry->flags & ((1UL << NR_PAGEFLAGS) - 1)), + show_page_flags(__entry->flags & ~PAGEFLAGS_MASK), __entry->count, __entry->mapcount, __entry->mapping, __entry->mt, __entry->val, __entry->ret) diff --git a/lib/test_printf.c b/lib/test_printf.c index 8ac71aee46af..eadcc6458b10 100644 --- a/lib/test_printf.c +++ b/lib/test_printf.c @@ -614,7 +614,7 @@ page_flags_test(int section, int node, int zone, int last_cpupid, bool append = false; int i; - flags &= BIT(NR_PAGEFLAGS) - 1; + flags &= ~PAGEFLAGS_MASK; if (flags) { page_flags |= flags; snprintf(cmp_buf + size, BUF_SIZE - size, "%s", name); diff --git a/lib/vsprintf.c b/lib/vsprintf.c index dd006adfe853..ff7c180a2971 100644 --- a/lib/vsprintf.c +++ b/lib/vsprintf.c @@ -2019,7 +2019,7 @@ static const struct page_flags_fields pff[] = { static char *format_page_flags(char *buf, char *end, unsigned long flags) { - unsigned long main_flags = flags & (BIT(NR_PAGEFLAGS) - 1); + unsigned long main_flags = flags & ~PAGEFLAGS_MASK; bool append = false; int i; From patchwork Wed Jul 14 09:17:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12376355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 720D1C07E9A for ; Wed, 14 Jul 2021 09:21:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 025E761396 for ; Wed, 14 Jul 2021 09:21:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 025E761396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A6276B0088; Wed, 14 Jul 2021 05:21:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47CCC6B008C; Wed, 14 Jul 2021 05:21:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D0AC8D0001; Wed, 14 Jul 2021 05:21:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id 0B4F26B0088 for ; Wed, 14 Jul 2021 05:21:07 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 058108245571 for ; Wed, 14 Jul 2021 09:21:06 +0000 (UTC) X-FDA: 78360649332.34.C0B980D Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf22.hostedemail.com (Postfix) with ESMTP id 58EE5192A for ; Wed, 14 Jul 2021 09:21:05 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id d12so1492639pfj.2 for ; Wed, 14 Jul 2021 02:21:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1wb5iN6C/r/jl8NTvJ6JTzD0t7BIT0w39tfIUzR0c+I=; b=zplQE4G7tcgmBinMSn8w4wPywLrIByLSGqitptn1ZR9FdsRi1QXv75GdAavbVA7JN4 LyVVy0vtZerXQwmXr6nlVQzQ3wBO5qAZAXBObfbOI49LsAX5QlP+g6vJsXVcdvmcZwT6 Gtv+myhlxOWU74E1cjQjT6kd3Y5OOiBtoi7kmyBlIfLms9+pQsxkmrhpzv2F7OGgj/Cg QjRJvLHu25gepnxpMVz6Q15ZrVzwLM4tpLz3sboZo42D7V6oxAwSsJmTab3xWFFXa7Ki mrOks3PQ1/lBeMR5srIeZCq9NCxEM81sIhf5bY2Hz1lS64edYvUANFsmm+YFyDWPIdhV rHaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1wb5iN6C/r/jl8NTvJ6JTzD0t7BIT0w39tfIUzR0c+I=; b=HhyLudDBAjL2LOXgIvqW1pQCGYOz7NFOq/InHgbh9gcDYfGUYv9v+DD/xyiqFXLaZM QYDS6M+BJ/vhU3AhCl1vfziR54ug49h6eb1IaoOPzEYxt2r85xKmxWtpafRPG7/tTJ+K 66gM4RPco83k8+MpzddVy4/owfADcBd8sHUj04k12jsI/j59+x7XX3JmKZjOKAnyiBfq VXJPxB2SW5Nq6tnaGCXmTGkOC462i48ch03G9Z1Znz1I19yt9L1Oy67axChNt/u+Xi1w 0vd93lSM0Su9hmuDeG5CODIKx11yVQiVdJ3XERw+BB7/+5rgsTT93HxV08dKbc5KcuAc /Odg== X-Gm-Message-State: AOAM530wxZo099hV8mnACfKRkhVOAtvfy51GCJ0HkuHyF16qBZkx1zjt 6plR/28bjEhV40TSyLyPOHTx0w== X-Google-Smtp-Source: ABdhPJzEijXZzF0Qf19GMjfpTQOk+y74uZh4grXC65z8xbeHoDsVcjqOTYZMN3z9W/kZBgak8cDqZQ== X-Received: by 2002:a65:564f:: with SMTP id m15mr8717302pgs.346.1626254464534; Wed, 14 Jul 2021 02:21:04 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id k19sm1742540pji.32.2021.07.14.02.20.58 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Jul 2021 02:21:04 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH 2/5] mm: introduce save_page_flags to cooperate with show_page_flags Date: Wed, 14 Jul 2021 17:17:57 +0800 Message-Id: <20210714091800.42645-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210714091800.42645-1-songmuchun@bytedance.com> References: <20210714091800.42645-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=zplQE4G7; spf=pass (imf22.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 58EE5192A X-Stat-Signature: kg7tihpqfijwph56eydnar5ayqca3ic5 X-HE-Tag: 1626254465-51596 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce save_page_flags to return the page flags which can cooperate with show_page_flags. If we want to hihe some page flags from users, it will be useful to alter save_page_flags directly. This is a preparation for the next patch to hide some page flags from users. Signed-off-by: Muchun Song --- include/trace/events/mmflags.h | 3 +++ include/trace/events/page_ref.h | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 390270e00a1d..69cb84b1257e 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -121,6 +121,9 @@ IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" ) \ IF_HAVE_PG_SKIP_KASAN_POISON(PG_skip_kasan_poison, "skip_kasan_poison") +#define save_page_flags(page) \ + (((page)->flags & ~PAGEFLAGS_MASK)) + #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ __def_pageflag_names \ diff --git a/include/trace/events/page_ref.h b/include/trace/events/page_ref.h index 643b1b4e9f27..53d303048d27 100644 --- a/include/trace/events/page_ref.h +++ b/include/trace/events/page_ref.h @@ -28,7 +28,7 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, TP_fast_assign( __entry->pfn = page_to_pfn(page); - __entry->flags = page->flags; + __entry->flags = save_page_flags(page); __entry->count = page_ref_count(page); __entry->mapcount = page_mapcount(page); __entry->mapping = page->mapping; @@ -38,7 +38,7 @@ DECLARE_EVENT_CLASS(page_ref_mod_template, TP_printk("pfn=0x%lx flags=%s count=%d mapcount=%d mapping=%p mt=%d val=%d", __entry->pfn, - show_page_flags(__entry->flags & ~PAGEFLAGS_MASK), + show_page_flags(__entry->flags), __entry->count, __entry->mapcount, __entry->mapping, __entry->mt, __entry->val) @@ -77,7 +77,7 @@ DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, TP_fast_assign( __entry->pfn = page_to_pfn(page); - __entry->flags = page->flags; + __entry->flags = save_page_flags(page); __entry->count = page_ref_count(page); __entry->mapcount = page_mapcount(page); __entry->mapping = page->mapping; @@ -88,7 +88,7 @@ DECLARE_EVENT_CLASS(page_ref_mod_and_test_template, TP_printk("pfn=0x%lx flags=%s count=%d mapcount=%d mapping=%p mt=%d val=%d ret=%d", __entry->pfn, - show_page_flags(__entry->flags & ~PAGEFLAGS_MASK), + show_page_flags(__entry->flags), __entry->count, __entry->mapcount, __entry->mapping, __entry->mt, __entry->val, __entry->ret) From patchwork Wed Jul 14 09:17:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12376357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D155C07E9C for ; Wed, 14 Jul 2021 09:21:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AEDE16128B for ; Wed, 14 Jul 2021 09:21:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AEDE16128B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F06806B008C; Wed, 14 Jul 2021 05:21:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDCBC8D0002; Wed, 14 Jul 2021 05:21:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D56BD8D0001; Wed, 14 Jul 2021 05:21:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id A875F6B008C for ; Wed, 14 Jul 2021 05:21:12 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 987A5256F2 for ; Wed, 14 Jul 2021 09:21:11 +0000 (UTC) X-FDA: 78360649542.12.10A3C0C Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf24.hostedemail.com (Postfix) with ESMTP id 2DF87B00009D for ; Wed, 14 Jul 2021 09:21:11 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id j9so1484857pfc.5 for ; Wed, 14 Jul 2021 02:21:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vHQ3UvG1NNiOhOhJEGNTO4n34RD5RRdHyAt+lnuFQLI=; b=WELaa2RK46XiADUxU3quNuzD9662ygUCVhhbjtda8omNkOohW/DMkiS47ZNM3yIE6T A89US0SlOT+pZExbPgBxQ3r9xCvLQT86D0shi460DJFcxDTR93RiltUyFLuJV5Wb1SSJ wGV5Qj9VhEKeTqWA8LLP/k8FcyfSsnmnt+0loUqa4TxsOq0t/1IfyGfiPxWUKCWejUUC AuHBKMtpCOJeMXXYzxW6hJFeag6+IjKT9ewx+PRMkJyBtszUe365+pBoWILAv5qhJVxd qJAybAwRjJOzBxEW5h3LxDYB0rdsAEBK1y5v+wzzAZMsS0MgRZw+WimvdFH99UTs3eHL fjlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vHQ3UvG1NNiOhOhJEGNTO4n34RD5RRdHyAt+lnuFQLI=; b=TQos8QBo1J2leI0xIy/d3VwuJQ8b6GyyOJdR21zJt4PQF+2/f0PMcyK0BTY2N3qczu 98XH8fUQV5vJsKqO0cLoYnQBB09SuEHGlv0yTqpwvM9Gjy/FprEcJEe5TIdZGypT7YBk 9X/+ZR/TOy+2EvK7uz6Zv9pusdNntJoM7BPe+0T57IOpXJJd1JWHWqFKGZVX4wtMd8O8 EmctSAF/wSnPtaI531YxwTiQexqKUTXbV5tABd0VgWg1mxBUkILjaxt5tHSDFIYgu1n1 tg9xOSNZgBEenTmEpa7mZP3hS9X26SFYy9sqRyHLd4EOuMAyGcqhpBn5KJUX1ZeqDfbD Dupw== X-Gm-Message-State: AOAM530oJzqWBnvy/U39+M02fMPgJk9jTE+HUSzOyJMQB8JpJqejchjv 4QD3pgMyoaVdyCxzkYdOckz4vg== X-Google-Smtp-Source: ABdhPJy9chGaNLVX+RdVyyubqXA+C48qkxYganCEOkLvGOWpbpJLMAFka36uZsYXg21zkcRPGahIhw== X-Received: by 2002:a63:580a:: with SMTP id m10mr8633458pgb.254.1626254470267; Wed, 14 Jul 2021 02:21:10 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id k19sm1742540pji.32.2021.07.14.02.21.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Jul 2021 02:21:09 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH 3/5] mm: hugetlb: free the 2nd vmemmap page associated with each HugeTLB page Date: Wed, 14 Jul 2021 17:17:58 +0800 Message-Id: <20210714091800.42645-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210714091800.42645-1-songmuchun@bytedance.com> References: <20210714091800.42645-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=WELaa2RK; spf=pass (imf24.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam05 X-Stat-Signature: dosz8oggu3spw78y9cp6ds6x7zeaqn5p X-Rspamd-Queue-Id: 2DF87B00009D X-HE-Tag: 1626254471-292092 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, we only free 6 vmemmap pages associated with a 2MB HugeTLB page. However, we can remap all tail vmemmap pages to the page frame mapped to with the head vmemmap page. Finally, we can free 7 vmemmap pages for a 2MB HugeTLB page. It is a fine gain (e.g. we can save extra 2GB memory when there is 1TB HugeTLB pages in the system compared with the current implementation). But the head vmemmap page is not freed to the buddy allocator and all tail vmemmap pages are mapped to the head vmemmap page frame. So we can see more than one struct page struct with PG_head (e.g. 8 per 2 MB HugeTLB page) associated with each HugeTLB page. We should adjust compound_head() to make it returns the real head struct page when the parameter is the tail struct page but with PG_head flag. Signed-off-by: Muchun Song --- Documentation/admin-guide/kernel-parameters.txt | 2 +- include/linux/page-flags.h | 97 ++++++++++++++++++++++++- include/trace/events/mmflags.h | 3 +- mm/hugetlb_vmemmap.c | 60 ++++++++------- mm/sparse-vmemmap.c | 21 ++++++ 5 files changed, 150 insertions(+), 33 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index bdb22006f713..a154a7b3b9a5 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1606,7 +1606,7 @@ [KNL] Reguires CONFIG_HUGETLB_PAGE_FREE_VMEMMAP enabled. Allows heavy hugetlb users to free up some more - memory (6 * PAGE_SIZE for each 2MB hugetlb page). + memory (7 * PAGE_SIZE for each 2MB hugetlb page). Format: { on | off (default) } on: enable the feature diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 358d3f6fa976..d29c16f7b176 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -182,25 +182,102 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP +extern bool hugetlb_free_vmemmap_enabled; + +/* + * If the feature of freeing some vmemmap pages associated with each HugeTLB + * page is enabled, the head vmemmap page frame is reused and all of the tail + * vmemmap addresses map to the head vmemmap page frame (furture details can + * refer to the figure at the head of the mm/hugetlb_vmemmap.c). In other + * word, there are more than one page struct with PG_head associated with each + * HugeTLB page. We __know__ that there is only one head page struct, the tail + * page structs with PG_head are fake head page structs. We need an approach + * to distinguish between those two different types of page structs so that + * compound_head() can return the real head page struct when the parameter is + * the tail page struct but with PG_head. This is what page_head_if_fake() + * does. + * + * The page_head_if_fake() returns the real head page struct iff the @page may + * be fake, otherwise, returns NULL if the @page cannot be a fake page struct. + * The following figure describes how to distinguish between real and fake head + * page struct. + * + * if (test_bit(PG_head, &page->flags)) { + * unsigned long head = READ_ONCE(page[1].compound_head); + * + * if (head & 1) { + * if (head == (unsigned long)page + 1) + * ==> head page struct + * else + * ==> tail page struct + * } else + * ==> head page struct + * } else + * ==> cannot be fake head page struct + * + * We can safely access the field of the @page[1] with PG_head because it means + * that the @page is a compound page composed with at least two contiguous + * pages. + */ +static __always_inline struct page *page_head_if_fake(const struct page *page) +{ + if (!hugetlb_free_vmemmap_enabled) + return NULL; + + /* + * Only addresses aligned with PAGE_SIZE of struct page may be fake head + * struct page. The alignment check aims to avoid access the fields ( + * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly) + * cold cacheline in some cases. + */ + if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) && + test_bit(PG_head, &page->flags)) { + unsigned long head = READ_ONCE(page[1].compound_head); + + if (likely(head & 1)) + return (struct page *)(head - 1); + } + + return NULL; +} +#else +static __always_inline struct page *page_head_if_fake(const struct page *page) +{ + return NULL; +} +#endif + static inline unsigned long _compound_head(const struct page *page) { unsigned long head = READ_ONCE(page->compound_head); if (unlikely(head & 1)) return head - 1; - return (unsigned long)page; + + head = (unsigned long)page_head_if_fake(page); + + return head ? : (unsigned long)page; } #define compound_head(page) ((typeof(page))_compound_head(page)) static __always_inline int PageTail(struct page *page) { - return READ_ONCE(page->compound_head) & 1; + struct page *head; + + if (READ_ONCE(page->compound_head) & 1) + return 1; + + head = page_head_if_fake(page); + + return head ? head != page : 0; } static __always_inline int PageCompound(struct page *page) { - return test_bit(PG_head, &page->flags) || PageTail(page); + return test_bit(PG_head, &page->flags) || + READ_ONCE(page->compound_head) & 1; } #define PAGE_POISON_PATTERN -1l @@ -578,7 +655,19 @@ static inline void set_page_writeback_keepwrite(struct page *page) test_set_page_writeback_keepwrite(page); } -__PAGEFLAG(Head, head, PF_ANY) CLEARPAGEFLAG(Head, head, PF_ANY) +static __always_inline int PageHead(struct page *page) +{ + struct page *head; + + PF_POISONED_CHECK(page); + head = page_head_if_fake(page); + + return head ? head == page : test_bit(PG_head, &page->flags); +} + +__SETPAGEFLAG(Head, head, PF_ANY) +__CLEARPAGEFLAG(Head, head, PF_ANY) +CLEARPAGEFLAG(Head, head, PF_ANY) static __always_inline void set_compound_head(struct page *page, struct page *head) { diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 69cb84b1257e..a0f0fd82e3e0 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -122,7 +122,8 @@ IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" ) \ IF_HAVE_PG_SKIP_KASAN_POISON(PG_skip_kasan_poison, "skip_kasan_poison") #define save_page_flags(page) \ - (((page)->flags & ~PAGEFLAGS_MASK)) + (PageHead(page) ? ((page)->flags & ~PAGEFLAGS_MASK) : \ + ((page)->flags & ~(PAGEFLAGS_MASK | PG_head_mask))) #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c540c21e26f5..527bcaa44a48 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -124,9 +124,9 @@ * page of page structs (page 0) associated with the HugeTLB page contains the 4 * page structs necessary to describe the HugeTLB. The only use of the remaining * pages of page structs (page 1 to page 7) is to point to page->compound_head. - * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs + * Therefore, we can remap pages 1 to 7 to page 0. Only 1 pages of page structs * will be used for each HugeTLB page. This will allow us to free the remaining - * 6 pages to the buddy allocator. + * 7 pages to the buddy allocator. * * Here is how things look after remapping. * @@ -134,30 +134,30 @@ * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ * | | | 0 | -------------> | 0 | * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | ----------------^ ^ ^ ^ ^ ^ - * | | +-----------+ | | | | | - * | | | 3 | ------------------+ | | | | - * | | +-----------+ | | | | - * | | | 4 | --------------------+ | | | - * | PMD | +-----------+ | | | - * | level | | 5 | ----------------------+ | | - * | mapping | +-----------+ | | - * | | | 6 | ------------------------+ | - * | | +-----------+ | - * | | | 7 | --------------------------+ + * | | | 1 | ---------------^ ^ ^ ^ ^ ^ ^ + * | | +-----------+ | | | | | | + * | | | 2 | -----------------+ | | | | | + * | | +-----------+ | | | | | + * | | | 3 | -------------------+ | | | | + * | | +-----------+ | | | | + * | | | 4 | ---------------------+ | | | + * | PMD | +-----------+ | | | + * | level | | 5 | -----------------------+ | | + * | mapping | +-----------+ | | + * | | | 6 | -------------------------+ | + * | | +-----------+ | + * | | | 7 | ---------------------------+ * | | +-----------+ * | | * | | * | | * +-----------+ * - * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for + * When a HugeTLB is freed to the buddy system, we should allocate 7 pages for * vmemmap pages and restore the previous mapping relationship. * * For the HugeTLB page of the pud level mapping. It is similar to the former. - * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. + * We also can use this approach to free (PAGE_SIZE - 1) vmemmap pages. * * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures * (e.g. aarch64) provides a contiguous bit in the translation table entries @@ -166,7 +166,13 @@ * * The contiguous bit is used to increase the mapping size at the pmd and pte * (last) level. So this type of HugeTLB page can be optimized only when its - * size of the struct page structs is greater than 2 pages. + * size of the struct page structs is greater than 1 pages. + * + * Notice: The head vmemmap page is not freed to the buddy allocator and all + * tail vmemmap pages are mapped to the head vmemmap page frame. So we can see + * more than one struct page struct with PG_head (e.g. 8 per 2 MB HugeTLB page) + * associated with each HugeTLB page. The compound_head() can handle this + * correctly (more details refer to the comment above compound_head()). */ #define pr_fmt(fmt) "HugeTLB: " fmt @@ -175,14 +181,16 @@ /* * There are a lot of struct page structures associated with each HugeTLB page. * For tail pages, the value of compound_head is the same. So we can reuse first - * page of tail page structures. We map the virtual addresses of the remaining - * pages of tail page structures to the first tail page struct, and then free - * these page frames. Therefore, we need to reserve two pages as vmemmap areas. + * page of head page structures. We map the virtual addresses of all the pages + * of tail page structures to the head page struct, and then free these page + * frames. Therefore, we need to reserve one pages as vmemmap areas. */ -#define RESERVE_VMEMMAP_NR 2U +#define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -bool hugetlb_free_vmemmap_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); +bool hugetlb_free_vmemmap_enabled __read_mostly = + IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); +EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled); static int __init early_hugetlb_free_vmemmap_param(char *buf) { @@ -236,7 +244,6 @@ int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) */ ret = vmemmap_remap_alloc(vmemmap_addr, vmemmap_end, vmemmap_reuse, GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); - if (!ret) ClearHPageVmemmapOptimized(head); @@ -282,9 +289,8 @@ void __init hugetlb_vmemmap_init(struct hstate *h) vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; /* - * The head page and the first tail page are not to be freed to buddy - * allocator, the other pages will map to the first tail page, so they - * can be freed. + * The head page is not to be freed to buddy allocator, the other tail + * pages will map to the head page, so they can be freed. * * Could RESERVE_VMEMMAP_NR be greater than @vmemmap_pages? It is true * on some architectures (e.g. aarch64). See Documentation/arm64/ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..62e3d20648ce 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -53,6 +53,17 @@ struct vmemmap_remap_walk { struct list_head *vmemmap_pages; }; +/* + * How many struct page structs need to be reset. When we reuse the head + * struct page, the special metadata (e.g. page->flags or page->mapping) + * cannot copy to the tail struct page structs. The invalid value will be + * checked in the free_tail_pages_check(). In order to avoid the message + * of "corrupted mapping in tail page". We need to reset at least 3 (one + * head struct page struct and two tail struct page structs) struct page + * structs. + */ +#define NR_RESET_STRUCT_PAGE 3 + static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, struct vmemmap_remap_walk *walk) { @@ -245,6 +256,15 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, entry); } +static inline void reset_struct_pages(struct page *start) +{ + int i; + struct page *from = start + NR_RESET_STRUCT_PAGE; + + for (i = 0; i < NR_RESET_STRUCT_PAGE; i++) + memcpy(start + i, from, sizeof(*from)); +} + static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, struct vmemmap_remap_walk *walk) { @@ -258,6 +278,7 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, list_del(&page->lru); to = page_to_virt(page); copy_page(to, (void *)walk->reuse_addr); + reset_struct_pages(to); set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } From patchwork Wed Jul 14 09:17:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12376359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DBCEC07E9C for ; Wed, 14 Jul 2021 09:21:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B5D96128B for ; Wed, 14 Jul 2021 09:21:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B5D96128B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4F4F98D0001; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CB226B0095; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36C4B8D0001; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id 15CEC6B0092 for ; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1820018587528 for ; Wed, 14 Jul 2021 09:21:17 +0000 (UTC) X-FDA: 78360649794.22.5C88D28 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf29.hostedemail.com (Postfix) with ESMTP id 9559F9000256 for ; Wed, 14 Jul 2021 09:21:16 +0000 (UTC) Received: by mail-pg1-f180.google.com with SMTP id 62so1794049pgf.1 for ; Wed, 14 Jul 2021 02:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CkdbfyChYU3bqFZgwG+U5i+8HOhTg3puUmTJxwvU+7E=; b=U6HfgZ+T/eT56e1XbZEvevYHqqXpNVMhY4FWPTeZZeapWt4nKyvve9g7cSTGlRaEgx Pz/ACDbeddOp2tDYW6pgVgxNjM3yy76a14fwbXT+0hznC6V0PQAicWAhywPZzR3ZvB+t M2RyXS+z6J8shiA0tuFWYJv+L0F+EjgqHmshUyXV8DH5Gi8HxShdiciPI6HbL7DHkSb+ KxtUS6NJP6NWxkhn+G0mViqf0S6oeZ2DnaaLhSUxg90Z+r/23t6OIUyxKThlnr3ZMlaS Pdd/itKuHLZlLFy4Uc6Z1SK4qnvVculUrR0+MLUJf6lic7Vv2JPnEcAC5a0wlhQgRSjS fbWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CkdbfyChYU3bqFZgwG+U5i+8HOhTg3puUmTJxwvU+7E=; b=R8U+iGRwnrqQGj4VcFBvIfGEMnyQTXeYDn5JYKq2GXDC4dkzU49QJ0B9jEKDS4a3qz wWdMZaLGN8n86e0QPpUOVoYd+1y4ykT+ISghga8WvPxNqB0kCAnUFwSPKjpVpEAWmbx8 D6UCK+44DxOHjXnob8PmqOENRYVV1AVTcvTG69U0l4s234jeBdyiqZbmLuj0VrYPlxyt Tp1T95GhxtOFyZp4N194nq7a3nEYoRk315i64LV2gLJ0MShq7z93CcQg05uQRhGM6WIw xgAM7A7nqKNtKNw2wf4ZUex+cnX1eiYbrJ8hViyfcplSvkAIH3nhkkL3Sf9ftLyOAtD1 aC7g== X-Gm-Message-State: AOAM5319CZPcxVk7J2oG8cla/rqJqZ+pvkBeQ1X2Z8/66li/rPbMYfuG Ah7r0jLSU4bWe0SxnI0GkL2B5w== X-Google-Smtp-Source: ABdhPJzoLfhTDsf9vvjpt27Db4UVjn18L2E3wvZk35t27v/G6ikZT65cAao/86OzOUyqtO55T+nmdA== X-Received: by 2002:a65:690a:: with SMTP id s10mr6674972pgq.99.1626254475855; Wed, 14 Jul 2021 02:21:15 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id k19sm1742540pji.32.2021.07.14.02.21.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Jul 2021 02:21:15 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH 4/5] mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key Date: Wed, 14 Jul 2021 17:17:59 +0800 Message-Id: <20210714091800.42645-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210714091800.42645-1-songmuchun@bytedance.com> References: <20210714091800.42645-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9559F9000256 X-Stat-Signature: e9675jboacarm4kussesmtqro7brp1f1 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=U6HfgZ+T; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1626254476-849452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_head_if_fake() is used throughout memory management and the conditional check requires checking a global variable, although the overhead of this check may be small, it increases when the memory cache comes under pressure. Also, the global variable will not be modified after system boot, so it is very appropriate to use static key machanism. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 6 +++++- include/linux/page-flags.h | 6 ++++-- mm/hugetlb_vmemmap.c | 13 +++++++------ 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f7ca1a3870ea..876a1af73bec 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1057,7 +1057,11 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr #endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +#define hugetlb_free_vmemmap_enabled \ + static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, \ + &hugetlb_free_vmemmap_enabled_key) #else #define hugetlb_free_vmemmap_enabled false #endif diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index d29c16f7b176..f05f806b98b4 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -183,7 +183,8 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); /* * If the feature of freeing some vmemmap pages associated with each HugeTLB @@ -222,7 +223,8 @@ extern bool hugetlb_free_vmemmap_enabled; */ static __always_inline struct page *page_head_if_fake(const struct page *page) { - if (!hugetlb_free_vmemmap_enabled) + if (!static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key)) return NULL; /* diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 527bcaa44a48..a382e12f6ff0 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -188,9 +188,9 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -bool hugetlb_free_vmemmap_enabled __read_mostly = - IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); -EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled); +DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); static int __init early_hugetlb_free_vmemmap_param(char *buf) { @@ -204,9 +204,9 @@ static int __init early_hugetlb_free_vmemmap_param(char *buf) return -EINVAL; if (!strcmp(buf, "on")) - hugetlb_free_vmemmap_enabled = true; + static_branch_enable(&hugetlb_free_vmemmap_enabled_key); else if (!strcmp(buf, "off")) - hugetlb_free_vmemmap_enabled = false; + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); else return -EINVAL; @@ -284,7 +284,8 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled) + if (!static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key)) return; vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; From patchwork Wed Jul 14 09:18:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12376361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB6A2C07E9A for ; Wed, 14 Jul 2021 09:21:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2CE9561396 for ; Wed, 14 Jul 2021 09:21:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2CE9561396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 73E7C8D0002; Wed, 14 Jul 2021 05:21:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 714C76B0095; Wed, 14 Jul 2021 05:21:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58E918D0002; Wed, 14 Jul 2021 05:21:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 36B7A6B0092 for ; Wed, 14 Jul 2021 05:21:24 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 216EE1858337D for ; Wed, 14 Jul 2021 09:21:23 +0000 (UTC) X-FDA: 78360650046.19.36088A1 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf30.hostedemail.com (Postfix) with ESMTP id D99D2E001816 for ; Wed, 14 Jul 2021 09:21:22 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id d1so1217494plg.0 for ; Wed, 14 Jul 2021 02:21:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0sSF06QuAY7kUNXCT+H8hO7WNAzsL+XHoJo4uKapYUQ=; b=hWbxjhh0bB5B6m4rcT6U6l1KYThmR4c2fqxERQD3zKLqXkWJMG/SysrhsRf+SH6gzN JEvfZCrpYNToVVx1FkVeze7w0t73ZfLF5XBrOqO7WP6KJN+q2Tx8Es0D/dBSjxvxJra9 W3IWsb4howhaF1lP3TRBoD8r+btX3cApquQw7dBUSWmYpU+t2e+xBLLJPwv5crFsoxqn QwJKXSegIhAsNoxYnw+UkwIfkHX7fs+RO0mU8Pfyir0agDJugBedGS3RoIyyGn9qYyT5 hggFWsM8Dbszg55bzNtvl5DFHAcfG1OGjoELlI2kE8QKY+CKDEJS5v2xnv37+xjmmq4L HT1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0sSF06QuAY7kUNXCT+H8hO7WNAzsL+XHoJo4uKapYUQ=; b=IByP0BDnEGJcC2sHT+ZdO5nQ1ehocO3NS1sv2U5V15/a/W6IuxuOradt1GQ3eWear5 EywHBIoWM1MlE4JKDsq2E//s1LqZYV974rKBaYXLiKuHRhuCP72jJk6UQQgf1lzgGAn8 5mBvoV1LFkdX2Q55JQPS51N8K5zZ2uKoH+3GhNzNQFpBHkLi1Q3r4ZbbdPtPxeRli8dx 7AR/K1jMCMY+ghAQ7RgUqtroqUDUlW/50YM1SFo9AJZ6+EvbN/beeAQwN1R6FhsvCpFp 9hH2vWv+ruM1vBwuldvfelBG8j1s35u4Kyt6ixoTVPcdfeAgxlaYlfyUiUjuJYF5bRzI RHuA== X-Gm-Message-State: AOAM531BQSG8J/oivV43dSN2sxPIOkOgWorfxxo1YATpdGhMuRHdEJ0C gbHqJtorwpO3HMoBh+ZH74i2dg== X-Google-Smtp-Source: ABdhPJxDHd9SN5fagjQhrWthsjzWABX4QS3GgblMv9qSYLWRITZkU4ZRFkOcw5Ylui6X2JqBjrTSLA== X-Received: by 2002:a17:903:2309:b029:12a:965b:333 with SMTP id d9-20020a1709032309b029012a965b0333mr6943494plh.31.1626254481985; Wed, 14 Jul 2021 02:21:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id k19sm1742540pji.32.2021.07.14.02.21.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Jul 2021 02:21:21 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH 5/5] mm: sparsemem: use page table lock to protect kernel pmd operations Date: Wed, 14 Jul 2021 17:18:00 +0800 Message-Id: <20210714091800.42645-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210714091800.42645-1-songmuchun@bytedance.com> References: <20210714091800.42645-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=hWbxjhh0; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam02 X-Stat-Signature: jcipmtx516idr33oyouiheq5c8y67jrp X-Rspamd-Queue-Id: D99D2E001816 X-HE-Tag: 1626254482-33044 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The init_mm.page_table_lock is used to protect kernel page tables, we can use it to serialize splitting vmemmap PMD mappings instead of mmap write lock, which can increase the concurrency of vmemmap_remap_free(). Signed-off-by: Muchun Song --- mm/ptdump.c | 16 ++++++++++++---- mm/sparse-vmemmap.c | 49 ++++++++++++++++++++++++++++++++++--------------- 2 files changed, 46 insertions(+), 19 deletions(-) diff --git a/mm/ptdump.c b/mm/ptdump.c index da751448d0e4..eea3d28d173c 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -40,8 +40,10 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long addr, if (st->effective_prot) st->effective_prot(st, 0, pgd_val(val)); - if (pgd_leaf(val)) + if (pgd_leaf(val)) { st->note_page(st, addr, 0, pgd_val(val)); + walk->action = ACTION_CONTINUE; + } return 0; } @@ -61,8 +63,10 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long addr, if (st->effective_prot) st->effective_prot(st, 1, p4d_val(val)); - if (p4d_leaf(val)) + if (p4d_leaf(val)) { st->note_page(st, addr, 1, p4d_val(val)); + walk->action = ACTION_CONTINUE; + } return 0; } @@ -82,8 +86,10 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long addr, if (st->effective_prot) st->effective_prot(st, 2, pud_val(val)); - if (pud_leaf(val)) + if (pud_leaf(val)) { st->note_page(st, addr, 2, pud_val(val)); + walk->action = ACTION_CONTINUE; + } return 0; } @@ -101,8 +107,10 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long addr, if (st->effective_prot) st->effective_prot(st, 3, pmd_val(val)); - if (pmd_leaf(val)) + if (pmd_leaf(val)) { st->note_page(st, addr, 3, pmd_val(val)); + walk->action = ACTION_CONTINUE; + } return 0; } diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 62e3d20648ce..e636943ccfc4 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -64,8 +64,8 @@ struct vmemmap_remap_walk { */ #define NR_RESET_STRUCT_PAGE 3 -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, - struct vmemmap_remap_walk *walk) +static int __split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, + struct vmemmap_remap_walk *walk) { pmd_t __pmd; int i; @@ -87,15 +87,37 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, set_pte_at(&init_mm, addr, pte, entry); } - /* Make pte visible before pmd. See comment in __pte_alloc(). */ - smp_wmb(); - pmd_populate_kernel(&init_mm, pmd, pgtable); + spin_lock(&init_mm.page_table_lock); + if (likely(pmd_leaf(*pmd))) { + /* Make pte visible before pmd. See comment in __pte_alloc(). */ + smp_wmb(); + pmd_populate_kernel(&init_mm, pmd, pgtable); + flush_tlb_kernel_range(start, start + PMD_SIZE); + spin_unlock(&init_mm.page_table_lock); - flush_tlb_kernel_range(start, start + PMD_SIZE); + return 0; + } + spin_unlock(&init_mm.page_table_lock); + pte_free_kernel(&init_mm, pgtable); return 0; } +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, + struct vmemmap_remap_walk *walk) +{ + int ret; + + spin_lock(&init_mm.page_table_lock); + ret = pmd_leaf(*pmd); + spin_unlock(&init_mm.page_table_lock); + + if (ret) + ret = __split_vmemmap_huge_pmd(pmd, start, walk); + + return ret; +} + static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct vmemmap_remap_walk *walk) @@ -132,13 +154,12 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, pmd = pmd_offset(pud, addr); do { - if (pmd_leaf(*pmd)) { - int ret; + int ret; + + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, walk); + if (ret) + return ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, walk); - if (ret) - return ret; - } next = pmd_addr_end(addr, end); vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -321,10 +342,8 @@ int vmemmap_remap_free(unsigned long start, unsigned long end, */ BUG_ON(start - reuse != PAGE_SIZE); - mmap_write_lock(&init_mm); + mmap_read_lock(&init_mm); ret = vmemmap_remap_range(reuse, end, &walk); - mmap_write_downgrade(&init_mm); - if (ret && walk.nr_walked) { end = reuse + walk.nr_walked * PAGE_SIZE; /*