From patchwork Mon Nov 4 21:06:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13862009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39A93D1BDF3 for ; Mon, 4 Nov 2024 21:06:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C0FE6B00A1; Mon, 4 Nov 2024 16:06:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 84B026B00A2; Mon, 4 Nov 2024 16:06:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EBB46B00A3; Mon, 4 Nov 2024 16:06:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 512126B00A1 for ; Mon, 4 Nov 2024 16:06:11 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0AD44C0B1D for ; Mon, 4 Nov 2024 21:06:11 +0000 (UTC) X-FDA: 82749644376.22.3491935 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 95F33A0011 for ; Mon, 4 Nov 2024 21:05:45 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Qhp+i0Z8; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730754311; a=rsa-sha256; cv=none; b=h2yoHldtYI/dUvkMgxe9Jc+33AQ/7T0qdkSk9Uv2pI/py4Uvg42U3zmAcaUjBLdXpaP+4P fosp0IeGmanpcSqoUXwlEnfkr1tsMBGGstiNTJ9l66C36kko6oIAbBwYkrdyEfKNvh+5uh B+7q+X6rw3AWSO9DVw0Y3gAVOYdwkBo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Qhp+i0Z8; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730754311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MlKXMVkwoR8OxX3WpEQg0gTq0YNyoaDmxbJW107WDCQ=; b=tXZ2KxORyyEgRUkH3MJLifrkVEQC8fJ/1XUW8YJWM1xvcbDl/awF53bNwpgZecKAF13mvr LlMskRD4mZQarkhQw43PTSWVX43Rs1K7aKReGoMNUJsT5M8HUs35oeFZYuRuYhBmTiuk7Y P9NMR74n5fK4RasNLIKvtEfWfKm8igU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=MlKXMVkwoR8OxX3WpEQg0gTq0YNyoaDmxbJW107WDCQ=; b=Qhp+i0Z8Wvi3p0vj2GbUlbmevq 6Huxn+l0vgzCe1MsbhAUVi0/+gJixni6PkK/HqegEN8hSaLTldkKEKv0Fy6MCl0JGj9LFZMkfqqi8 fMj4Sg2/hQ70lLVDYy+bGSaZaj9J7VYZV7g8hIgJvuIE0bPUzN4gB/zDp8/g8K/wnT+qyzis7kOHo SuWNK+mci/XiNBdhfauPOmPBN3MZlTFtDC6sKnLkCoGTU63Ytpd8k8ibY9lmI7vJZI7HHz4aZxwJu MAjwEpqgggawL5LTKk/JxCusvGFPoB3QGRA5F8aD0jDEngnEUJNLOAVjE7UCiDHguUw8fdhpRfYPc zE9FKzcA==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1t84Gy-00000001ZYP-1XXt; Mon, 04 Nov 2024 21:06:04 +0000 From: "Matthew Wilcox (Oracle)" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: "Matthew Wilcox (Oracle)" , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/3] mm: Introduce acctmem Date: Mon, 4 Nov 2024 21:06:00 +0000 Message-ID: <20241104210602.374975-4-willy@infradead.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241104210602.374975-1-willy@infradead.org> References: <20241104210602.374975-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 95F33A0011 X-Rspamd-Server: rspam11 X-Stat-Signature: ps4qtjgr8ntfetksbyysmoffq7gc1wdi X-HE-Tag: 1730754345-794454 X-HE-Meta: U2FsdGVkX1/joBKeL1szwrU53sQhZwD+US22vk0VfbgrJF4qiLMDlk08ftg3pGHF7p+HrYd+F44XFiX5z99aiwjH0QZyZ1tq4E2DxGCPV/mU33o4dzp8NqmWBogwciuesdYLDoJYlzzB6FYZKYY3WkLKxhXUZ/hEN4HR3fVZfqa26nS2Y3ZxzZ5g6W6gEnyWIpXSyDXFffhEAD98NoB8ENF8ZVU5fAbAKCKi8oCxkqOQnqvxFZSydDSDarDxqlfJhj+NG6hAUOvhIsEjxmfq6RBuMf/bJHZgm3bnzrOXXBvmvGNfxRMIu5gz/dLHNTHC1nBmRk0YrjYrzapeWqBzfTk7lF2mu+NHK9/g1U0zNpH0mKckq/sIJBEkf8O8K8fRn9VZJUmQlcHLdtGux8HBkkIGn5N83XAv1JDryVP1NOq00ZFxharbnkL1/fuGo2y94PBmqotnFm9maKKImW2GQKl1bFYWGMoU/scut4n11rlI/fBq0O6ssUBTaWJfLaxyIeOHIm8axyxBnscp3WHWDAJumRNPPZeiOaPeS4RQjF2urIZBhlDZ2EZjIh+W/BnYe94zKUxHCzpZ8VNJNgCmtL+ovskr+/xWATPJMbxcPsRq/Ln5cjAukoQbMv7EV//OaFJwCKcZ2sz/5SMSxwIaZ6oMHMOsvZQ4p+A1WN/p8L4UKjr+I76C7Mk4syh0V6QqTSMoZFfxlWWEx4t8+ckFWXH122Al1Zpo9IMAdFkjpOBFwxNjIts1vpFS6NEJe7pC5R7Gu1Tb32QHUgr6XJ95iMdJ8ZHGh6+2B3KMZ1siz3qfHubYy1iEjkXIHMt4MJDROKRnKrb4VMo9VT7vx1CGh07hmwBE5ctACGz/nyh/cPxadcG/2xLtFSfav9UHyrzlMh9fqUo9DUHjL0OE4XzwnYVVVCrg+m3oVCukmskbq3K1iEPNCxrGxdW5AZ5cSRqObqU2mtTexf7IgZyeXwC iUDCg0Bu qWx2tXMJ+RJEqDOq2Y9l6ZIXGyh6dTCFVfKajTPLQJIZhLWmcE1AoO3ijeJc67Ab2qQfZmKRshVamTWz5fyh7LkReyyC6ruxj24v7PHkexabEogr4xrYqnp9grzWkyvXepD/Sfp0ajUKOUjR6eLfNXelpJVC/YN1/IF8XQma5SMAGR30paD+yktfcaUIkMLCzCLpDSFQlEK4jpY/yoWXcKiu9GvEQG3kxFH2SovJC3ZhFu5o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: struct acctmem is used for MEMCG_DATA_KMEM allocations. We're still a bit loose with our casting to folios instead of acctmem, but that's a problem to solve later. The build asserts ensure that this carelessness doesn't cause any new bugs today. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 24 ++++++++++++++++++++++++ include/linux/mm_types.h | 6 +++--- mm/memcontrol.c | 7 ++++--- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 2 +- mm/slab.h | 2 +- 6 files changed, 35 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a787080f814f..19ee98abea0f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -30,6 +30,30 @@ struct page; struct mm_struct; struct kmem_cache; +/* + * For now, this data structure overlays struct page. Eventually it + * will be separately allocated and become a memdesc type of its own + * like slab and ptdesc. memcg_data is only valid on the first page + * of an allocation, but that allocation might not be compound! + */ +struct acctmem { + unsigned long __page_flags; + unsigned long __padding[5]; + unsigned int ___padding[2]; + unsigned long memcg_data; +}; +#ifdef CONFIG_MEMCG +static_assert(offsetof(struct page, __acct_memcg_data) == + offsetof(struct acctmem, memcg_data)); +static_assert(offsetof(struct folio, memcg_data) == + offsetof(struct acctmem, memcg_data)); +static_assert(sizeof(struct acctmem) <= sizeof(struct page)); +#endif + +#define page_acctmem(_page) (_Generic((_page), \ + const struct page *: (const struct acctmem *)(_page), \ + struct page *: (struct acctmem *)(_page))) + /* Cgroup-specific page state, on top of universal node page state */ enum memcg_stat_item { MEMCG_SWAP = NR_VM_NODE_STAT_ITEMS, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2b694f9a4518..274b125df0df 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -181,7 +181,7 @@ struct page { atomic_t _refcount; #ifdef CONFIG_MEMCG - unsigned long memcg_data; + unsigned long __acct_memcg_data; #elif defined(CONFIG_SLAB_OBJ_EXT) unsigned long _unused_slab_obj_exts; #endif @@ -410,7 +410,7 @@ FOLIO_MATCH(private, private); FOLIO_MATCH(_mapcount, _mapcount); FOLIO_MATCH(_refcount, _refcount); #ifdef CONFIG_MEMCG -FOLIO_MATCH(memcg_data, memcg_data); +FOLIO_MATCH(__acct_memcg_data, memcg_data); #endif #if defined(WANT_PAGE_VIRTUAL) FOLIO_MATCH(virtual, virtual); @@ -499,7 +499,7 @@ TABLE_MATCH(rcu_head, pt_rcu_head); TABLE_MATCH(page_type, __page_type); TABLE_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG -TABLE_MATCH(memcg_data, pt_memcg_data); +TABLE_MATCH(__acct_memcg_data, pt_memcg_data); #endif #undef TABLE_MATCH static_assert(sizeof(struct ptdesc) <= sizeof(struct page)); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 506439a5dcfe..89c9d206c209 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2661,6 +2661,7 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, */ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { + struct acctmem *acctmem = page_acctmem(page); struct obj_cgroup *objcg; int ret = 0; @@ -2669,7 +2670,7 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { obj_cgroup_get(objcg); - page->memcg_data = (unsigned long)objcg | + acctmem->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } @@ -3039,7 +3040,7 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, */ void split_page_memcg(struct page *first, int order) { - unsigned long memcg_data = first->memcg_data; + unsigned long memcg_data = page_acctmem(first)->memcg_data; struct obj_cgroup *objcg; int i; unsigned int nr = 1 << order; @@ -3052,7 +3053,7 @@ void split_page_memcg(struct page *first, int order) objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); for (i = 1; i < nr; i++) - first[i].memcg_data = memcg_data; + page_acctmem(first + i)->memcg_data = memcg_data; obj_cgroup_get_many(objcg, nr - 1); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5523654c9759..07d9302882b2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -870,7 +870,7 @@ static inline bool page_expected_state(struct page *page, if (unlikely((unsigned long)page->mapping | page_ref_count(page) | #ifdef CONFIG_MEMCG - page->memcg_data | + page_acctmem(page)->memcg_data | #endif #ifdef CONFIG_PAGE_POOL ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) | @@ -898,7 +898,7 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) bad_reason = "PAGE_FLAGS_CHECK_AT_FREE flag(s) set"; } #ifdef CONFIG_MEMCG - if (unlikely(page->memcg_data)) + if (unlikely(page_acctmem(page)->memcg_data)) bad_reason = "page still charged to cgroup"; #endif #ifdef CONFIG_PAGE_POOL diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb..71e183f8988b 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -506,7 +506,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, char name[80]; rcu_read_lock(); - memcg_data = READ_ONCE(page->memcg_data); + memcg_data = READ_ONCE(page_acctmem(page)->memcg_data); if (!memcg_data) goto out_unlock; diff --git a/mm/slab.h b/mm/slab.h index 632fedd71fea..ee9ab84f7c4d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -103,7 +103,7 @@ SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, obj_exts); +SLAB_MATCH(__acct_memcg_data, obj_exts); #elif defined(CONFIG_SLAB_OBJ_EXT) SLAB_MATCH(_unused_slab_obj_exts, obj_exts); #endif