From patchwork Wed Apr 17 07:18:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13632934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72957C04FF9 for ; Wed, 17 Apr 2024 07:19:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B16B06B009E; Wed, 17 Apr 2024 03:19:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A01BF6B009D; Wed, 17 Apr 2024 03:19:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7406E6B009B; Wed, 17 Apr 2024 03:19:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5435D6B0093 for ; Wed, 17 Apr 2024 03:19:12 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E0C7E1A0E94 for ; Wed, 17 Apr 2024 07:19:11 +0000 (UTC) X-FDA: 82018172502.12.F36B798 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf12.hostedemail.com (Postfix) with ESMTP id 2FA984000B for ; Wed, 17 Apr 2024 07:19:08 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713338350; a=rsa-sha256; cv=none; b=tK2yRc++El9Idz5z3FMoAuVo1yc6zuB8P+DvorcaXZKHNmdFaxOE7JKAwk10dz3Eiz09E/ 0i4JBIPwoZrizWhfvwsiGPMAPUxys+loTzgohPzZ6hze6YqJ7LqXDZHb5a1dvNe0sDTr38 aAohT2ImVzRbU8x7tAvDuWL4ziQH6uM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713338350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=oVNWWG+0CQkcyA+DIk7yFlw00zmRrOMdSG4j4afHB8k=; b=cExpY9+2M26ffKGmmwx9hERCN+7KyboU1o+6xr7YCUqEheTy/ZH+2oo+Eji9M3WhcfBgCD k9wH1ZLxyftarBh97QxtQyfGtmW3W2LWiQYOqW4v/cqQinn2Jc0LJI0ot9ZcFt3w/qFxcA 9+I6WCSGR7cx0DXk8GLOph/AiLuxRZY= X-AuditID: a67dfc5b-d6dff70000001748-ac-661f77e2f414 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 6/8] mm: buddy: make room for a new variable, mgen, in struct page Date: Wed, 17 Apr 2024 16:18:45 +0900 Message-Id: <20240417071847.29584-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnke6jcvk0g5dflSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG v1kX2AquK1c8fuzZwLhIpouRk0NCwESi+fFKdhj7VuMaRhCbTUBd4saNn8wgtoiAmcTB1j9g NcwCd5kkDvSzgdjCAqESsz7NZwWxWQRUJV49XgtUz8HBK2AqMaOxBmKkvMTqDQfAxnACjdnc vAasVQio5PuJj0A2F1DNZzaJV+fes0I0SEocXHGDZQIj7wJGhlWMQpl5ZbmJmTkmehmVeZkV esn5uZsYgWG/rPZP9A7GTxeCDzEKcDAq8fAaRMmlCbEmlhVX5h5ilOBgVhLhbRGWTRPiTUms rEotyo8vKs1JLT7EKM3BoiTOa/StPEVIID2xJDU7NbUgtQgmy8TBKdXAuHhWdVnnz+yK0n0y p5g0kk3sdm1K3yxft1qYbZf4rdi6Jp49kyeK9a3IlZ5yb21n4ZOGiUcS2qemPnsRsvbMva4r 87fE/LdRZ55Yzcq7z+XiVPU1x5/LRr6+Jm84wW+5gv6ts/v2FLRL/ZgaJMej5fN/6bzd/d9i g2Osv2ar7g8qPeDwx6jeQomlOCPRUIu5qDgRACm0Dwh3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7N4NwBSYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZfybdYGt4LpyxePHng2Mi2S6GDk5JARMJG41rmEEsdkE1CVu3PjJDGKLCJhJ HGz9ww5iMwvcZZI40M8GYgsLhErM+jSfFcRmEVCVePV4LVA9BwevgKnEjMYaiJHyEqs3HAAb wwk0ZnPzGrBWIaCS7yc+sk1g5FrAyLCKUSQzryw3MTPHVK84O6MyL7NCLzk/dxMjMIiX1f6Z uIPxy2X3Q4wCHIxKPLwGUXJpQqyJZcWVuYcYJTiYlUR4W4Rl04R4UxIrq1KL8uOLSnNSiw8x SnOwKInzeoWnJggJpCeWpGanphakFsFkmTg4pRoYjeqZG5+ViM9xYxTSX7HtQYF43aNDl72/ 1FZe2v/XbMmD3wwtdve2i8ns/LcwpWFT4PN/4ls81lddvP7UJnBrx98oLZvcSxeEr23b9lBk +4sXMZdSf0lfT9VhZuxb5nFocd+k+yb8ev8z655f3LZ1HZ/fpaZ57zmeT52s/TXV5ardnVXt oct7lyuxFGckGmoxFxUnAgC2pfBtXgIAAA== X-CFilter-Loop: Reflected X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2FA984000B X-Stat-Signature: aoyeu1byqi9d5n8bjsso8rcijwxe3emf X-Rspam-User: X-HE-Tag: 1713338348-407447 X-HE-Meta: U2FsdGVkX1+AbfX4R/YldjhsqVcuNu9JTKHE8P4X/ox06A6L85okeZw15u+uwRQQYo+obeONGFWDope4HD08skDjteCoB6EmI7yxS29zwwZ7cs1GkxG3CCEOyFDmWQXlsjnh6jYj5yOhhcZCdWJVghv18R/tYbs2L7zIE1QSBTH4O6KFOOc6ig0gPJJZMOvjbWUVB08joAF2uhlmmY6Rqc1W6Y8TOhOxkYLlwQzp6Gs0xvePW+MpnR+3GaAzW1kQb0Wrz+Ql7j+FO4aZTtD2W6Doc1RTrEENNh/ZztNHJ5FzipG4L/yMMsJsX+t3+j/vg0wLSpBtRXCY33SBURPj7CL9hVLq3jUup9OBjS49ftHLGutU7bZR6dCmi5LngQ7+BFIEp4DrNILs+ci+itLtny7VKIMJME7zUD93Is70FNm6V/bBwlK1rcMHZYqZZDepBU4WnTkOPLmOvXgOhvew5Jp6xTbgsDP5wBQ2v0fOHhT5svZWj99Xw2sf9LQ0qWjkP3+vCfG1vBly9QenwjaLGo9SOfZKKiLfome27qen/BFm1v1SETJK0MY84jYV0v/cm1UZVQIyQZVnf0pKF01j7zCwTAs+wyfJvv1BczBrEh1wv+lSmZz97fZPDIHNh8Jfa5RNGK3sLl5VUnn8EErdo9D9xj/AHc8fvXKRYucg0qOETsEltokvdFcDoj1hoIunwoch59Wo0VWSHLrb0Ez33ligLYcXDm4ZhjGr2OCZM69TgnsUj0bjB5hR2lYSnVNK1vC3tqb/wo/+1cWqOYUsCDL0krYE90YYxiCpbJMAT8Ofuca4cuLjQqyDANXF4ZucYkLYaSK9rpna2hffkcDZvzobC8idb0usj60RrkY/3yMU3Bo1THs7o32DdD+y0yNNimjqzGWYztelpZpMxRr/w4lNd0rECyJSstgode1U2djvuctMUcpetldrcoA7OtW0cKeo7W7492AML8RTFur 9/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that tracks need of tlb flush for each page residing in buddy, using a generation number in struct page. Fortunately, since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short int. So splitted it into two smaller ones, order and mgen, so that the both can be used in buddy at the same time. Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 39 ++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 13 ++++++++----- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5240bd7bca33..4130d22977d6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -108,13 +108,24 @@ struct page { pgoff_t index; /* Our offset within mapping. */ unsigned long share; /* share count for fsdax */ }; - /** - * @private: Mapping-private opaque data. - * Usually used for buffer_heads if PagePrivate. - * Used for swp_entry_t if PageSwapCache. - * Indicates order in the buddy system if PageBuddy. - */ - unsigned long private; + union { + /** + * @private: Mapping-private opaque data. + * Usually used for buffer_heads if PagePrivate. + * Used for swp_entry_t if PageSwapCache. + */ + unsigned long private; + struct { + /* + * Indicates order in the buddy system if PageBuddy. + */ + unsigned short int order; + /* + * Tracks need of tlb flush used by migrc + */ + unsigned short int mgen; + }; + }; }; struct { /* page_pool used by netstack */ /** @@ -520,6 +531,20 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +#define page_buddy_order(page) ((page)->order) + +static inline void set_page_buddy_order(struct page *page, unsigned int order) +{ + page->order = (unsigned short int)order; +} + +#define page_buddy_mgen(page) ((page)->mgen) + +static inline void set_page_buddy_mgen(struct page *page, unsigned short int mgen) +{ + page->mgen = mgen; +} + static inline void *folio_get_private(struct folio *folio) { return folio->private; diff --git a/mm/internal.h b/mm/internal.h index bbaf1fd97b1c..e3a8b77b58ca 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -364,7 +364,7 @@ struct alloc_context { static inline unsigned int buddy_order(struct page *page) { /* PageBuddy() must be checked by the caller */ - return page_private(page); + return page_buddy_order(page); } /* @@ -378,7 +378,7 @@ static inline unsigned int buddy_order(struct page *page) * times, potentially observing different values in the tests and the actual * use of the result. */ -#define buddy_order_unsafe(page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe(page) READ_ONCE(page_buddy_order(page)) /* * This function checks whether a page is free && is the buddy diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 14d39f34d336..555f6d2e33ea 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -604,9 +604,12 @@ void destroy_large_folio(struct folio *folio) free_the_page(&folio->page, folio_order(folio)); } -static inline void set_buddy_order(struct page *page, unsigned int order) +static inline void set_buddy_order_mgen(struct page *page, + unsigned int order, + unsigned short int mgen) { - set_page_private(page, order); + set_page_buddy_order(page, order); + set_page_buddy_mgen(page, order); __SetPageBuddy(page); } @@ -824,7 +827,7 @@ static inline void __free_one_page(struct page *page, } done_merging: - set_buddy_order(page, order); + set_buddy_order_mgen(page, order, 0); if (fpi_flags & FPI_TO_TAIL) to_tail = true; @@ -1404,7 +1407,7 @@ static inline void expand(struct zone *zone, struct page *page, continue; add_to_free_list(&page[size], zone, high, migratetype); - set_buddy_order(&page[size], high); + set_buddy_order_mgen(&page[size], high, 0); } } @@ -6710,7 +6713,7 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, continue; add_to_free_list(current_buddy, zone, high, migratetype); - set_buddy_order(current_buddy, high); + set_buddy_order_mgen(current_buddy, high, 0); } }