From patchwork Mon Nov 4 21:05:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13862010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2295DD1BDF4 for ; Mon, 4 Nov 2024 21:06:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B4996B00A2; Mon, 4 Nov 2024 16:06:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53E946B00A3; Mon, 4 Nov 2024 16:06:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F5636B00A4; Mon, 4 Nov 2024 16:06:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 078596B00A2 for ; Mon, 4 Nov 2024 16:06:12 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B1A8340910 for ; Mon, 4 Nov 2024 21:06:11 +0000 (UTC) X-FDA: 82749645048.28.06B3A02 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 57AB5120021 for ; Mon, 4 Nov 2024 21:05:26 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eKNcimCI; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730754205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gXYh3cqHcJHGeUBg4M162tqHpDf+n9+pJfHXppJVvX8=; b=Y4NcxZPRXFCUc/E4+YM5g69d76o6lI3c7ghtu17NTqPqoKTxBJVtZ+2U1c/OGAuiVHPPIf 6flKJpHzw216yGzHjL0orGK+6XyPENkc//rFAUgNiLTEEM0ws9TU3/Bh56JSs6nkIwKaG4 EA5Ztmlah2LYOahFHZ+p6gLI/irD404= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eKNcimCI; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730754205; a=rsa-sha256; cv=none; b=X1zFaJ3R1Fi43i2nzJbTi14GiB5k/rMO/+ZyfbjK5V+DGwM18gqOvAXBUAkKiLExLTnj3w xGN8V7Ejy5oJEIN5wbsAMR5q/hTTNsTbjXMTGKcdCVU3LDZkdKgWtaN+HFLIIi23HpJF6H NZuluJFJokiJNWduMSw11eKhF25C/OU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gXYh3cqHcJHGeUBg4M162tqHpDf+n9+pJfHXppJVvX8=; b=eKNcimCIWz8r8AhbydXdSOF4Bm vjAZ+AnNuSAHBM79pVcg8E3yOWxdhKZdrshxl64iYJae5w7OXaJpvjCZ+GDp/NXiTk/lfgPg+uLLy 9VoOx4HYikCSE8Cr4D39wxYFPf5onfG78FVxRtWXfeIUehak7qAO/SyfLDNDI2tbwqeJpgKpPRq6K BUps+r8AUI3S4B+BlS14v+IHkT/esIZ7hRxJxR77qqLtVp18bSo+jggQWqdmxHdWzgL1MHgPz0GkT QV5uDpMtMh4J+AYwWU5s32vATaIYTsVam9KW63kldO0sFODSr1Z24roYSwLalSZVoY0WwCKjhjMl/ r3a52hdQ==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1t84Gy-00000001ZYK-0hmI; Mon, 04 Nov 2024 21:06:04 +0000 From: "Matthew Wilcox (Oracle)" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: "Matthew Wilcox (Oracle)" , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/3] mm: Opencode split_page_memcg() in __split_huge_page() Date: Mon, 4 Nov 2024 21:05:58 +0000 Message-ID: <20241104210602.374975-2-willy@infradead.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241104210602.374975-1-willy@infradead.org> References: <20241104210602.374975-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 57AB5120021 X-Stat-Signature: a7bfyc4smk8qj13hqtaywbmgokee4i6p X-Rspam-User: X-HE-Tag: 1730754326-639724 X-HE-Meta: U2FsdGVkX1+HI4MSs8nDuI56Vlu5ChT4kILwpG4lnBntdn0EBWA0dpOjl90EzdsytqRMwj0SFE0xVfS8Yldjzlwudtrcq53cThthahutkjGyiYuOHe315Q0E+hy/PJgzTd3F8UsulGeEBTm3mwnlkk9Q2KK8FE2u3m1riHWjkIAX08PVE334evDsTgTLedAUdvSLh6KuI5eok29c6H2nWVE4wD2gPuyDgahvSEnwayJKPtn+hs1+ObniY15nHkD3upH1vQLNmva1/HvJ5ss7SQOp6IB8wf+cSSOZiWph6vg2erWjrMQ/+moa8TL7cEHI4Nb98y1HD7PqdtltlsEcw6odJo0J9RLVzzvdWkExavgMpIqpHmuAdWG+DpgCBcW6n+/LD3KX29lq4sEyCsBpxtu6SP7RK2kYgCuEL1j/LWuoEdkeNZsRe+fBFbPII1f8Dkw1/Fn0ZCMUm6S6VV0ATDledDJbJTdBSgglPWSIJ9d/8IObmmyGb7kbU/5wdqwk2PqxSFr2qC8uJr/Z3aiU046/WnIShVERs56H60MFwNlYPxw8e5STVHRT6V42pDoNgdw6QdmpQhe4KwP5se+TRLD+/uoJNVyfib3wwKvWyriNUsOvrcWNU3jSIYX+nJW0PL7KvqR5PLTnvRrXupxehglIZnDuBMsQYQec+gAJaugIgTapiAUiprB/5k81s3JBUaUKbXtNCdRI+cTzvv3FGk70MCQ/qUMBZ3gAhS4+NvuAf6dANZaKQnCTVUqlT9Yba5ufxef1/zpGala3LJzxW81oN8sUByyJLvy4F+6zbG+S+ZqEuxYScMUvne3F6nbI/ciVvcRGxZcLnd71PcXcmwo/zrzKu7/Omv9fC1pkKUZJ9zOgKp2aQyNjcXDzW43JHCdfnUUs1bufxwjX1/2Crj21xUMSr2XlASFjBLIPBkI6hxftVhPqlEy7CZJttp/Wpgsai+oVmJ0ImYxZyBC YxtvT9ui xCOFt5ClhXaMxlawK6yJX/QQUCvQnEo4yIVGAlxktDjBeMYUI5AoTJhqeDQXDipPnM8cBd5FbmYQt0Kq0bA2+huwRvYkkDXg4TWF8BFwwZOmPTN21xscKAcCHW6u+qLI9tEnAEZ2QCb/1AB+N+ANLSJ31TW3Qa0uViSA21tnPBYUo6/V9FbQCDRWqWXFL9B6Az0jWpjxGzVjOwAs8jJLXdbUhIfkIbKI8zg/ENuigNKSxjPPEvmx6FheL+sLdCPvFofci6rJXJLi+DJDlIk5CLxpKVPq3tV6sqZhH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is in preparation for only handling kmem pages in __split_huge_page(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f92068864469..44d25a74b611 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3234,6 +3234,10 @@ static void __split_huge_page_tail(struct folio *folio, int tail, folio_set_large_rmappable(new_folio); } +#ifdef CONFIG_MEMCG + new_folio->memcg_data = folio->memcg_data; +#endif + /* Finally unfreeze refcount. Additional reference from page cache. */ page_ref_unfreeze(page_tail, 1 + ((!folio_test_anon(folio) || folio_test_swapcache(folio)) ? @@ -3267,8 +3271,11 @@ static void __split_huge_page(struct page *page, struct list_head *list, int order = folio_order(folio); unsigned int nr = 1 << order; - /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order, new_order); +#ifdef CONFIG_MEMCG + if (folio_memcg_charged(folio)) + css_get_many(&folio_memcg(folio)->css, + (1 << (order - new_order)) - 1); +#endif if (folio_test_anon(folio) && folio_test_swapcache(folio)) { offset = swap_cache_index(folio->swap); From patchwork Mon Nov 4 21:05:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13862012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54D85D1BDF3 for ; Mon, 4 Nov 2024 21:06:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7DE56B00A0; Mon, 4 Nov 2024 16:06:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D078A6B00A4; Mon, 4 Nov 2024 16:06:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B80FB6B00A5; Mon, 4 Nov 2024 16:06:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9987F6B00A0 for ; Mon, 4 Nov 2024 16:06:17 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 59F71140ACE for ; Mon, 4 Nov 2024 21:06:17 +0000 (UTC) X-FDA: 82749643368.20.57BB9E8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 4C28C20007 for ; Mon, 4 Nov 2024 21:05:42 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LBGDMQwO; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730754292; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n4HJqHIFL6KoDyu6o3HDh3SUN9SIibGS4LEHqczrBb0=; b=bxAD9Tmz/kBKrCZiXzDVECEpZ0p9Eu2jxd6h+oz05fid/ijDkj1qL4ru1e+hQ/zBTRS+MH M//8a03Fv4kU9FwH236o1vkRHf4IGfLEFtfjto444td8SleSn41ZnPhzYZRmbLJIF4Dgpj +TpwS9wLqYwUwtfpwYLWPwLG9D2FjvU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LBGDMQwO; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730754292; a=rsa-sha256; cv=none; b=kZqzQAd9lD0NQMFaD9oWtbrcsVuw94HK6m4CcO3voHErz6IpMpu/k5nRlLfkXCYESKKc1w NDzpz1B7jbsCSnlz8WlXxdsfEfNLgfEOnLVQtksKeVNiJTKiwchPZmaOfi3kOmqVcOG/Nl OiWjEef91wJHzwG+FB5lwRXPo+OnxZc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=n4HJqHIFL6KoDyu6o3HDh3SUN9SIibGS4LEHqczrBb0=; b=LBGDMQwOUt8tta8VHNMF+zE9ED SVBWmfKJzxz+yptFcLR0vrOQCmdJ9MPFhP9i/N/OTwaUsy7KfnOY2V/QeD9deQuKfWrxakLdBbpRM ddImJROE0Ug4pFwod2ux+4/ao6ATz71tF7RTeSGcQhoUz142XbT4FH5V750HdeGxhk4DBEqA5Y2KR rGDqgBflCT1dpbTbiXe0X9lrMIgEMrouVA7GYw66o7sbQ7H94EFQsIuBho3ivKvCkGntqyzBPKnE9 nNS4KpMWfWmiHR16S9pzfqeBz4s3zNtRbkVKpE3rdm0PcQYUUSpKGhDdp0dMVkxzzezuuQ1jm+SdP AIkfZaIQ==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1t84Gy-00000001ZYN-15RY; Mon, 04 Nov 2024 21:06:04 +0000 From: "Matthew Wilcox (Oracle)" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: "Matthew Wilcox (Oracle)" , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/3] mm: Simplify split_page_memcg() Date: Mon, 4 Nov 2024 21:05:59 +0000 Message-ID: <20241104210602.374975-3-willy@infradead.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241104210602.374975-1-willy@infradead.org> References: <20241104210602.374975-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4C28C20007 X-Stat-Signature: 4m45wy7i5mdiqx5boawczpzirg89nb7r X-HE-Tag: 1730754342-439810 X-HE-Meta: U2FsdGVkX18089v/AMaI4faAXdRDGBIG2TKdaj5aCmbAFTA4VQ5mBzImcgwKxLsGJx+v/8jo5DdtJ09Qne9053v6Go5G/C6KQeki8zHztVUfhgAv/6PIakbTBduiBRN5FzIzdfr7fTwhcJARNtcasXs/BURqvLEUPnc+/g0S1IfMbFrbki/Sr0tHSf+evkP0z9ypdNQWiS+0vXPWWEvCvVhrl6TfobX13PUiQdcf8fbIfqUlUU5v7fyT1PYXE2WQFTTU/zj5/uriMXBDJv/t171yaRuXCU+Wpw45dZv0L6FtzXrarng47x72+V4MT5NsQ5c7gdYb2Lm8hqqiF8tPTzY7KhttspsDVn2LCdeq84vELcxe2XHxqJF8dZmxdK3GaNLctAoO0hFwmN6IQ+pTH4LgGb0nhZ6H4zk32z7RJMtirjhbbZeatrJ+SYzC9iMj9e4XrcTA0YYEa5NG4wXTRzRClNhKHGkwhPEEiP80dai3x3PtQ/qam3qign7C8ur5Li/S2KTesGyhp9xeysKaSEztqv6JjaRZMrz1OywO5i72gKvq0eY2SGv963khmo/QrJdBmwd5ehfw62hUoPUzcjCVXfNWTS/iDSKjDI4CtXZe6BOK7JheDPud2Hi7Av/BAKWNKT3A2rruWYfwhCVoPJeG+rRQhjlZgA6UCx8Z2VYdHQhu4/BOchZDhG079xwX7rz+5ni9cxRjWfBjRw8Hgaxmu5p9NzDIWf87eVjiyun2MfVSv8xO6ktQcXfW9y34I5bDccidXyl9OjFSnNKZ4O79eMThqwJnhl0IFWxRN1QBfh7MBR1rzMyDXLfypGalqoblykeah0guLcUVVeHg0p5hEVtW8E7d7r1aRz4gxmUoU7ICPu+GO9Iuz8pDkob1QhD8XbjCpK4iF7kceE7vK+bR9PhDGkQfKpqBb2N3HLkQSUdH4tabX6yq4HYWxYWqEEwpWtUswfyygeYsFsO WNTQjj8o NPlyRCO6MvPrUDsHUoLh+rpSPCb8XUJgYYR6VvFzkAoL4T/X1tPlT6K6+BkaD9Tx2bmQjeEtmSiH3U60jcSCGXdU6m50MOuciiqu5JuwYVX+LDpDTjxxaoimGPOaiwxnLpBjXPagDpdmjAQBGVB+O8vYbuIpSV5si2aZDIx8HQhVRdUV6ic/TLPr8Eqe+z+hx1pJSk7hqRfit4T2Yp4qdeyZvKbwEwt13RjHZ4OqThihIHLlUS21swLlzmmNfSD44eHMj93vl3guZr7CAfkempbYXfkQutLYEeBDv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The last argument to split_page_memcg() is now always 0, so remove it, effectively reverting commit b8791381d7ed. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 26 ++++++++++++++------------ mm/page_alloc.c | 4 ++-- 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5502aa8e138e..a787080f814f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1044,7 +1044,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, rcu_read_unlock(); } -void split_page_memcg(struct page *head, int old_order, int new_order); +void split_page_memcg(struct page *first, int order); #else /* CONFIG_MEMCG */ @@ -1463,7 +1463,7 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) { } -static inline void split_page_memcg(struct page *head, int old_order, int new_order) +static inline void split_page_memcg(struct page *first, int order) { } #endif /* CONFIG_MEMCG */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5e44d6e7591e..506439a5dcfe 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3034,25 +3034,27 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, } /* - * Because folio_memcg(head) is not set on tails, set it now. + * The memcg data is only set on the first page, now transfer it to all the + * other pages. */ -void split_page_memcg(struct page *head, int old_order, int new_order) +void split_page_memcg(struct page *first, int order) { - struct folio *folio = page_folio(head); + unsigned long memcg_data = first->memcg_data; + struct obj_cgroup *objcg; int i; - unsigned int old_nr = 1 << old_order; - unsigned int new_nr = 1 << new_order; + unsigned int nr = 1 << order; - if (mem_cgroup_disabled() || !folio_memcg_charged(folio)) + if (!memcg_data) return; - for (i = new_nr; i < old_nr; i += new_nr) - folio_page(folio, i)->memcg_data = folio->memcg_data; + VM_BUG_ON_PAGE((memcg_data & OBJEXTS_FLAGS_MASK) != MEMCG_DATA_KMEM, + first); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); - if (folio_memcg_kmem(folio)) - obj_cgroup_get_many(__folio_objcg(folio), old_nr / new_nr - 1); - else - css_get_many(&folio_memcg(folio)->css, old_nr / new_nr - 1); + for (i = 1; i < nr; i++) + first[i].memcg_data = memcg_data; + + obj_cgroup_get_many(objcg, nr - 1); } unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 47048b39b8ca..5523654c9759 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2823,7 +2823,7 @@ void split_page(struct page *page, unsigned int order) set_page_refcounted(page + i); split_page_owner(page, order, 0); pgalloc_tag_split(page_folio(page), order, 0); - split_page_memcg(page, order, 0); + split_page_memcg(page, order); } EXPORT_SYMBOL_GPL(split_page); @@ -5020,7 +5020,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, split_page_owner(page, order, 0); pgalloc_tag_split(page_folio(page), order, 0); - split_page_memcg(page, order, 0); + split_page_memcg(page, order); while (page < --last) set_page_refcounted(last); From patchwork Mon Nov 4 21:06:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13862009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39A93D1BDF3 for ; Mon, 4 Nov 2024 21:06:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C0FE6B00A1; Mon, 4 Nov 2024 16:06:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 84B026B00A2; Mon, 4 Nov 2024 16:06:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EBB46B00A3; Mon, 4 Nov 2024 16:06:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 512126B00A1 for ; Mon, 4 Nov 2024 16:06:11 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0AD44C0B1D for ; Mon, 4 Nov 2024 21:06:11 +0000 (UTC) X-FDA: 82749644376.22.3491935 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 95F33A0011 for ; Mon, 4 Nov 2024 21:05:45 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Qhp+i0Z8; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730754311; a=rsa-sha256; cv=none; b=h2yoHldtYI/dUvkMgxe9Jc+33AQ/7T0qdkSk9Uv2pI/py4Uvg42U3zmAcaUjBLdXpaP+4P fosp0IeGmanpcSqoUXwlEnfkr1tsMBGGstiNTJ9l66C36kko6oIAbBwYkrdyEfKNvh+5uh B+7q+X6rw3AWSO9DVw0Y3gAVOYdwkBo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Qhp+i0Z8; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730754311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MlKXMVkwoR8OxX3WpEQg0gTq0YNyoaDmxbJW107WDCQ=; b=tXZ2KxORyyEgRUkH3MJLifrkVEQC8fJ/1XUW8YJWM1xvcbDl/awF53bNwpgZecKAF13mvr LlMskRD4mZQarkhQw43PTSWVX43Rs1K7aKReGoMNUJsT5M8HUs35oeFZYuRuYhBmTiuk7Y P9NMR74n5fK4RasNLIKvtEfWfKm8igU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=MlKXMVkwoR8OxX3WpEQg0gTq0YNyoaDmxbJW107WDCQ=; b=Qhp+i0Z8Wvi3p0vj2GbUlbmevq 6Huxn+l0vgzCe1MsbhAUVi0/+gJixni6PkK/HqegEN8hSaLTldkKEKv0Fy6MCl0JGj9LFZMkfqqi8 fMj4Sg2/hQ70lLVDYy+bGSaZaj9J7VYZV7g8hIgJvuIE0bPUzN4gB/zDp8/g8K/wnT+qyzis7kOHo SuWNK+mci/XiNBdhfauPOmPBN3MZlTFtDC6sKnLkCoGTU63Ytpd8k8ibY9lmI7vJZI7HHz4aZxwJu MAjwEpqgggawL5LTKk/JxCusvGFPoB3QGRA5F8aD0jDEngnEUJNLOAVjE7UCiDHguUw8fdhpRfYPc zE9FKzcA==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1t84Gy-00000001ZYP-1XXt; Mon, 04 Nov 2024 21:06:04 +0000 From: "Matthew Wilcox (Oracle)" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: "Matthew Wilcox (Oracle)" , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/3] mm: Introduce acctmem Date: Mon, 4 Nov 2024 21:06:00 +0000 Message-ID: <20241104210602.374975-4-willy@infradead.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241104210602.374975-1-willy@infradead.org> References: <20241104210602.374975-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 95F33A0011 X-Rspamd-Server: rspam11 X-Stat-Signature: ps4qtjgr8ntfetksbyysmoffq7gc1wdi X-HE-Tag: 1730754345-794454 X-HE-Meta: U2FsdGVkX1/joBKeL1szwrU53sQhZwD+US22vk0VfbgrJF4qiLMDlk08ftg3pGHF7p+HrYd+F44XFiX5z99aiwjH0QZyZ1tq4E2DxGCPV/mU33o4dzp8NqmWBogwciuesdYLDoJYlzzB6FYZKYY3WkLKxhXUZ/hEN4HR3fVZfqa26nS2Y3ZxzZ5g6W6gEnyWIpXSyDXFffhEAD98NoB8ENF8ZVU5fAbAKCKi8oCxkqOQnqvxFZSydDSDarDxqlfJhj+NG6hAUOvhIsEjxmfq6RBuMf/bJHZgm3bnzrOXXBvmvGNfxRMIu5gz/dLHNTHC1nBmRk0YrjYrzapeWqBzfTk7lF2mu+NHK9/g1U0zNpH0mKckq/sIJBEkf8O8K8fRn9VZJUmQlcHLdtGux8HBkkIGn5N83XAv1JDryVP1NOq00ZFxharbnkL1/fuGo2y94PBmqotnFm9maKKImW2GQKl1bFYWGMoU/scut4n11rlI/fBq0O6ssUBTaWJfLaxyIeOHIm8axyxBnscp3WHWDAJumRNPPZeiOaPeS4RQjF2urIZBhlDZ2EZjIh+W/BnYe94zKUxHCzpZ8VNJNgCmtL+ovskr+/xWATPJMbxcPsRq/Ln5cjAukoQbMv7EV//OaFJwCKcZ2sz/5SMSxwIaZ6oMHMOsvZQ4p+A1WN/p8L4UKjr+I76C7Mk4syh0V6QqTSMoZFfxlWWEx4t8+ckFWXH122Al1Zpo9IMAdFkjpOBFwxNjIts1vpFS6NEJe7pC5R7Gu1Tb32QHUgr6XJ95iMdJ8ZHGh6+2B3KMZ1siz3qfHubYy1iEjkXIHMt4MJDROKRnKrb4VMo9VT7vx1CGh07hmwBE5ctACGz/nyh/cPxadcG/2xLtFSfav9UHyrzlMh9fqUo9DUHjL0OE4XzwnYVVVCrg+m3oVCukmskbq3K1iEPNCxrGxdW5AZ5cSRqObqU2mtTexf7IgZyeXwC iUDCg0Bu qWx2tXMJ+RJEqDOq2Y9l6ZIXGyh6dTCFVfKajTPLQJIZhLWmcE1AoO3ijeJc67Ab2qQfZmKRshVamTWz5fyh7LkReyyC6ruxj24v7PHkexabEogr4xrYqnp9grzWkyvXepD/Sfp0ajUKOUjR6eLfNXelpJVC/YN1/IF8XQma5SMAGR30paD+yktfcaUIkMLCzCLpDSFQlEK4jpY/yoWXcKiu9GvEQG3kxFH2SovJC3ZhFu5o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: struct acctmem is used for MEMCG_DATA_KMEM allocations. We're still a bit loose with our casting to folios instead of acctmem, but that's a problem to solve later. The build asserts ensure that this carelessness doesn't cause any new bugs today. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 24 ++++++++++++++++++++++++ include/linux/mm_types.h | 6 +++--- mm/memcontrol.c | 7 ++++--- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 2 +- mm/slab.h | 2 +- 6 files changed, 35 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a787080f814f..19ee98abea0f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -30,6 +30,30 @@ struct page; struct mm_struct; struct kmem_cache; +/* + * For now, this data structure overlays struct page. Eventually it + * will be separately allocated and become a memdesc type of its own + * like slab and ptdesc. memcg_data is only valid on the first page + * of an allocation, but that allocation might not be compound! + */ +struct acctmem { + unsigned long __page_flags; + unsigned long __padding[5]; + unsigned int ___padding[2]; + unsigned long memcg_data; +}; +#ifdef CONFIG_MEMCG +static_assert(offsetof(struct page, __acct_memcg_data) == + offsetof(struct acctmem, memcg_data)); +static_assert(offsetof(struct folio, memcg_data) == + offsetof(struct acctmem, memcg_data)); +static_assert(sizeof(struct acctmem) <= sizeof(struct page)); +#endif + +#define page_acctmem(_page) (_Generic((_page), \ + const struct page *: (const struct acctmem *)(_page), \ + struct page *: (struct acctmem *)(_page))) + /* Cgroup-specific page state, on top of universal node page state */ enum memcg_stat_item { MEMCG_SWAP = NR_VM_NODE_STAT_ITEMS, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2b694f9a4518..274b125df0df 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -181,7 +181,7 @@ struct page { atomic_t _refcount; #ifdef CONFIG_MEMCG - unsigned long memcg_data; + unsigned long __acct_memcg_data; #elif defined(CONFIG_SLAB_OBJ_EXT) unsigned long _unused_slab_obj_exts; #endif @@ -410,7 +410,7 @@ FOLIO_MATCH(private, private); FOLIO_MATCH(_mapcount, _mapcount); FOLIO_MATCH(_refcount, _refcount); #ifdef CONFIG_MEMCG -FOLIO_MATCH(memcg_data, memcg_data); +FOLIO_MATCH(__acct_memcg_data, memcg_data); #endif #if defined(WANT_PAGE_VIRTUAL) FOLIO_MATCH(virtual, virtual); @@ -499,7 +499,7 @@ TABLE_MATCH(rcu_head, pt_rcu_head); TABLE_MATCH(page_type, __page_type); TABLE_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG -TABLE_MATCH(memcg_data, pt_memcg_data); +TABLE_MATCH(__acct_memcg_data, pt_memcg_data); #endif #undef TABLE_MATCH static_assert(sizeof(struct ptdesc) <= sizeof(struct page)); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 506439a5dcfe..89c9d206c209 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2661,6 +2661,7 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, */ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { + struct acctmem *acctmem = page_acctmem(page); struct obj_cgroup *objcg; int ret = 0; @@ -2669,7 +2670,7 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { obj_cgroup_get(objcg); - page->memcg_data = (unsigned long)objcg | + acctmem->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } @@ -3039,7 +3040,7 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, */ void split_page_memcg(struct page *first, int order) { - unsigned long memcg_data = first->memcg_data; + unsigned long memcg_data = page_acctmem(first)->memcg_data; struct obj_cgroup *objcg; int i; unsigned int nr = 1 << order; @@ -3052,7 +3053,7 @@ void split_page_memcg(struct page *first, int order) objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); for (i = 1; i < nr; i++) - first[i].memcg_data = memcg_data; + page_acctmem(first + i)->memcg_data = memcg_data; obj_cgroup_get_many(objcg, nr - 1); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5523654c9759..07d9302882b2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -870,7 +870,7 @@ static inline bool page_expected_state(struct page *page, if (unlikely((unsigned long)page->mapping | page_ref_count(page) | #ifdef CONFIG_MEMCG - page->memcg_data | + page_acctmem(page)->memcg_data | #endif #ifdef CONFIG_PAGE_POOL ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) | @@ -898,7 +898,7 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) bad_reason = "PAGE_FLAGS_CHECK_AT_FREE flag(s) set"; } #ifdef CONFIG_MEMCG - if (unlikely(page->memcg_data)) + if (unlikely(page_acctmem(page)->memcg_data)) bad_reason = "page still charged to cgroup"; #endif #ifdef CONFIG_PAGE_POOL diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb..71e183f8988b 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -506,7 +506,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, char name[80]; rcu_read_lock(); - memcg_data = READ_ONCE(page->memcg_data); + memcg_data = READ_ONCE(page_acctmem(page)->memcg_data); if (!memcg_data) goto out_unlock; diff --git a/mm/slab.h b/mm/slab.h index 632fedd71fea..ee9ab84f7c4d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -103,7 +103,7 @@ SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, obj_exts); +SLAB_MATCH(__acct_memcg_data, obj_exts); #elif defined(CONFIG_SLAB_OBJ_EXT) SLAB_MATCH(_unused_slab_obj_exts, obj_exts); #endif