From patchwork Thu Apr 18 06:15:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13634195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7319C4345F for ; Thu, 18 Apr 2024 06:16:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E817E6B0088; Thu, 18 Apr 2024 02:15:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF6316B0093; Thu, 18 Apr 2024 02:15:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAD266B0092; Thu, 18 Apr 2024 02:15:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8536D6B0088 for ; Thu, 18 Apr 2024 02:15:59 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 31EE581115 for ; Thu, 18 Apr 2024 06:15:59 +0000 (UTC) X-FDA: 82021642038.25.402CF5E Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id 5560D16000D for ; Thu, 18 Apr 2024 06:15:57 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713420957; a=rsa-sha256; cv=none; b=qX8Uglb6FNqutb8aPSYnW2mW+9HDGDzKCNsirqB8uwj7uH3ZFBXz3WQOVibp5+zXOXri2b baj0MAzp/V2hp4YuVIM6i4WQc8M8Hmw1O3qe7idsE5qI6SXLtklKkEoJxwmrE4OPMfL5vY EC2yjSHrDIWL7IL8BVe6OtMenSALBXk= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713420957; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=udltLnJrzq8A+QOdcyzDK5oKmDtw05sfEaiVfZKA5bE=; b=RbLXQ4BChDi534JTLIXkMYzNp6B7sfHUbanLvQXRrqNMvTgrqzE1HowSRG13UhQ8xHJ2UV A7GPUneHmsng0s9nRzjIc9IYk1DFQaffisSgMa+QEFFExzHtKTDUmbpCy1hm/F6QSSONrg 2nqNgOJuQgDakkhX+6rlx5LKl6ZCiMw= X-AuditID: a67dfc5b-d6dff70000001748-16-6620ba93ce04 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 rebase on mm-unstable 6/8] mm: buddy: make room for a new variable, mgen, in struct page Date: Thu, 18 Apr 2024 15:15:34 +0900 Message-Id: <20240418061536.11645-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240418061536.11645-1-byungchul@sk.com> References: <20240418061536.11645-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnke7kXQppBjvnC1jMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8Xx3gNMFvPvfWaz 2LxpKrPF8SlTGS1+/wAqPjlrMouDgMf31j4Wj52z7rJ7LNhU6rF5hZbH4j0vmTw2repk89j0 aRK7x7tz59g9Tsz4zeIx72Sgx/t9V9k8tv6y82iceo3N4/MmuQC+KC6blNSczLLUIn27BK6M f6f3MhasUqn41DGNpYHxpUwXIyeHhICJxLRfjxlh7L5TncwgNpuAusSNGz/BbBEBM4mDrX/Y QWxmgbtMEgf62UBsYYFsiS+LvgPVcHCwCKhK/O9jAQnzCphKPL94lBVipLzE6g0HwMZwAo3p f3+IEaRcCKjmwt+QLkYuoJL3bBJf/tyGqpeUOLjiBssERt4FjAyrGIUy88pyEzNzTPQyKvMy K/SS83M3MQLDflntn+gdjJ8uBB9iFOBgVOLhPXlAPk2INbGsuDL3EKMEB7OSCG+LsGyaEG9K YmVValF+fFFpTmrxIUZpDhYlcV6jb+UpQgLpiSWp2ampBalFMFkmDk6pBsa5jWeuh3osWhkz 4ysLk9+az42XTvz75vXggUWIyPNNM4++WjUhY/JHplm+7fNMqyeu/7lj/uapyUxXVJ4z+2e+ LDpTMt22f7dT785jBU8vTDuts9zH/nhjpyhnIeeeKw+Vlm8tO1yUYy5xLuCmysQ3bm43Dy/9 qZP1+fHm08ZOXUrxJ7Yc27csQomlOCPRUIu5qDgRAPdDPVp3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrDt5l0Kawd2/zBZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AVR8ctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnksfvGByWPrLzuPxqnX2Dw+b5IL4I/isklJ zcksSy3St0vgyvh3ei9jwSqVik8d01gaGF/KdDFyckgImEj0nepkBrHZBNQlbtz4CWaLCJhJ HGz9ww5iMwvcZZI40M8GYgsLZEt8WfQdqIaDg0VAVeJ/HwtImFfAVOL5xaOsECPlJVZvOAA2 hhNoTP/7Q4wg5UJANRf+hkxg5FrAyLCKUSQzryw3MTPHVK84O6MyL7NCLzk/dxMjMIiX1f6Z uIPxy2X3Q4wCHIxKPLwnDsinCbEmlhVX5h5ilOBgVhLhbRGWTRPiTUmsrEotyo8vKs1JLT7E KM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgPB46x3Lj1tbjzflrl97XqFt6eOG9ZCe+ pb+2irWY3Dq1ZPqNhdf1tkdJrFJ0y3bR/rqxOUjsrPFcxsLS3gvzbvBt2nMxq/1HV9LMULF6 772ff37tKeZd/KNj+rXtG7OYmY3nBhhLLItxeie+4WlXmZTr2+ZXN9l/8k7N6XfMjtN6d/ER 45pJ2UosxRmJhlrMRcWJAHomsZpeAgAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5560D16000D X-Stat-Signature: kmisu9gnay984gs8psm6sop7d3kaxwrp X-Rspam-User: X-HE-Tag: 1713420957-265662 X-HE-Meta: U2FsdGVkX19uxgR+DSbcgzaN8oNH4claf8ZyFYHo/2ZmF2Wog8PD5UB2v8yaZnJ0Cr8SGymMKZ4A9pMt6UbfTcbWv2bR01GAgu7lwGNfRUoJqQvmWKR0K8By33RtTIQ4E6kuUCQdueQUH+tCMqtJ+ovEFXws4CV/k2M3rTDaNR0dkaf1V/jI0CAXj841CxHkjMctFle4gCnoAgDL3DJMbm+0b9bnv02kSOnZOP41mCv7vc4FKH5xZYb+zB8ZArO1E7xLkpptVEW2Qu1HeGfpv1bBcXhEV6LViW5l5M22w3SPE5jVeVAjDX0r2rKIOA/2xKayWgbO8AHrx7Cxw6OvY0yjMB28nECqT3oFKSAT6m2ial60gndzWOwqPaj8vmEy1FElFUGyZ23tyrHckdeYO/vPYxhfESzBmorE2tDBiad4ulwcQrv3177ei0WqnYyWHtGuS7E4U0c9i5N8UY2kJo2cJqfDswu1sP4sFIiL1xkF2Ls8oXhvZejWkeEZqF5oP5dFqVKJl50S8RrPWEoDmZ90Wbx6+kmk2JMRJca/Cp3gZtEZSbhOMjBKfe//0+4XORD3oOc2dV+JlRkbL6YVhjzZM7358N8+X0LwXcYDylegJnvBqTCActM+rvgqK9UPlJ3ON11g2YRtun6suLwUNXF/oIK+iRhLUHxMWF/12xRS7y8B3wyWX501xesG1Kzyfhqdhf/HWO7Nvf5k9Wz5G9ozxxJpp9NLLBh41I7v7zpFM6jLyzVg0OczVo0ugr6ctwhp02T0LLIPv9IjVECdowk5l5BzQ4FHVAoybR2xrGWDXdI0eO+AEuHpB1WE+norB038bkk+kjI00plFliVhvMRIraUe7aIv4qN6JNleCineCtBtXdBncXu9djhk6EQM8RtY8XdtQ7D8J22MjpDfFhfWyQdwv0oisIPLfrLskzTE+UPbJGvsfwylwD0YZVGnO3S7HpSeYseFjSLUuGE RVw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that tracks need of tlb flush for each page residing in buddy, using a generation number in struct page. Fortunately, since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short int. So splitted it into two smaller ones, order and mgen, so that the both can be used in buddy at the same time. Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 39 ++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 13 ++++++++----- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index db0adf5721cc..47fd3780bd19 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -108,13 +108,24 @@ struct page { pgoff_t index; /* Our offset within mapping. */ unsigned long share; /* share count for fsdax */ }; - /** - * @private: Mapping-private opaque data. - * Usually used for buffer_heads if PagePrivate. - * Used for swp_entry_t if PageSwapCache. - * Indicates order in the buddy system if PageBuddy. - */ - unsigned long private; + union { + /** + * @private: Mapping-private opaque data. + * Usually used for buffer_heads if PagePrivate. + * Used for swp_entry_t if PageSwapCache. + */ + unsigned long private; + struct { + /* + * Indicates order in the buddy system if PageBuddy. + */ + unsigned short int order; + /* + * Tracks need of tlb flush used by migrc + */ + unsigned short int mgen; + }; + }; }; struct { /* page_pool used by netstack */ /** @@ -521,6 +532,20 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +#define page_buddy_order(page) ((page)->order) + +static inline void set_page_buddy_order(struct page *page, unsigned int order) +{ + page->order = (unsigned short int)order; +} + +#define page_buddy_mgen(page) ((page)->mgen) + +static inline void set_page_buddy_mgen(struct page *page, unsigned short int mgen) +{ + page->mgen = mgen; +} + static inline void *folio_get_private(struct folio *folio) { return folio->private; diff --git a/mm/internal.h b/mm/internal.h index b34d9e627132..0336375c6e8b 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -453,7 +453,7 @@ struct alloc_context { static inline unsigned int buddy_order(struct page *page) { /* PageBuddy() must be checked by the caller */ - return page_private(page); + return page_buddy_order(page); } /* @@ -467,7 +467,7 @@ static inline unsigned int buddy_order(struct page *page) * times, potentially observing different values in the tests and the actual * use of the result. */ -#define buddy_order_unsafe(page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe(page) READ_ONCE(page_buddy_order(page)) /* * This function checks whether a page is free && is the buddy diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 33d4a1be927b..cbde22c4c189 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,9 +565,12 @@ void prep_compound_page(struct page *page, unsigned int order) prep_compound_head(page, order); } -static inline void set_buddy_order(struct page *page, unsigned int order) +static inline void set_buddy_order_mgen(struct page *page, + unsigned int order, + unsigned short int mgen) { - set_page_private(page, order); + set_page_buddy_order(page, order); + set_page_buddy_mgen(page, order); __SetPageBuddy(page); } @@ -834,7 +837,7 @@ static inline void __free_one_page(struct page *page, } done_merging: - set_buddy_order(page, order); + set_buddy_order_mgen(page, order, 0); if (fpi_flags & FPI_TO_TAIL) to_tail = true; @@ -1344,7 +1347,7 @@ static inline void expand(struct zone *zone, struct page *page, continue; __add_to_free_list(&page[size], zone, high, migratetype, false); - set_buddy_order(&page[size], high); + set_buddy_order_mgen(&page[size], high, 0); nr_added += size; } account_freepages(zone, nr_added, migratetype); @@ -6802,7 +6805,7 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, continue; add_to_free_list(current_buddy, zone, high, migratetype, false); - set_buddy_order(current_buddy, high); + set_buddy_order_mgen(current_buddy, high, 0); } }