From patchwork Mon Mar 25 08:20:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13601618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 903C6C54E58 for ; Mon, 25 Mar 2024 08:21:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A34A86B0083; Mon, 25 Mar 2024 04:21:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 928216B0088; Mon, 25 Mar 2024 04:21:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 661566B0085; Mon, 25 Mar 2024 04:21:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 457506B0082 for ; Mon, 25 Mar 2024 04:21:10 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 435D0807DB for ; Mon, 25 Mar 2024 08:21:09 +0000 (UTC) X-FDA: 81934866258.25.9F7D416 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf28.hostedemail.com (Postfix) with ESMTP id 0FEF8C0007 for ; Mon, 25 Mar 2024 08:21:06 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711354867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qBRgqt4jbuW8UvbrviqbACBT8Ecy7jzcUyGe+8D7Vco=; b=cSsJviuLjINAo6Lv0eORYl966Q2XuoV8gT8JJeltjoNrK662uoG9pNJYEc17qsFg7AxYDI yGkBCjZ+x6LvSxYq+xM7AXcaTK3rZl8whqv6lTmOAzCd05ShNeVzcdXHTRQhaG8TTFnbYd J223JCv58zKiCTURR123LesUDKf5PLE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711354867; a=rsa-sha256; cv=none; b=it/Q1z0xrVyj1auNV1qw5hfaEzq7km35d+DEFg/qMSzabav3nXpsl+GoRpYaOnLMr0BLpA vT5+xeOiG6ZPsPeuRF+Hf8/ZWaBcdWIVwwW1kyMffBrM30WF4WBRPF4C49gYXLp182rR0a F+ANcclUmC5G0kIc8gV3RvNrdm1NE88= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 8DF8134F0B; Mon, 25 Mar 2024 08:21:05 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 660A1137CA; Mon, 25 Mar 2024 08:21:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id iMOgGPEzAWZdHgAAD6G6ig (envelope-from ); Mon, 25 Mar 2024 08:21:05 +0000 From: Vlastimil Babka Date: Mon, 25 Mar 2024 09:20:32 +0100 Subject: [PATCH v2 1/2] mm, slab: move memcg charging to post-alloc hook MIME-Version: 1.0 Message-Id: <20240325-slab-memcg-v2-1-900a458233a6@suse.cz> References: <20240325-slab-memcg-v2-0-900a458233a6@suse.cz> In-Reply-To: <20240325-slab-memcg-v2-0-900a458233a6@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara , Shakeel Butt Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka , Chengming Zhou X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10712; i=vbabka@suse.cz; h=from:subject:message-id; bh=euMCa5d5VUUeF8W9UqgWWtNYyJ+ULe5ZncwkAMI4uMY=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmATPqJwFS6fIZ1mjjEmvOkVxte6JGZrD9lA0Wi QuJqKBoztKJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZgEz6gAKCRC74LB10kWI mv37B/0dztlTlEmEPLpyCpWq8pMACuW6aDrVkXxyxg7plY0ei1MuxVmcGaK/n5MrLcQKsH/1OoE bFovHHksBYmeYTRuVdVtgi0OdEI9RM3o9MS2eT6MHhoQaN0z36zlFcP3BLFjqmpLvXNo9diCoRo +x7ddobnFbNO2TnC7QxG8bK91zXAiHz+9xk9t7wTOoPuw82FRAKZbhh7ejhLbV9IBYCUf8VUuuW PZN6PNJqArcj0gF9Bgd4PU1UWsY7WpyFqbgkEJ2qkOZabV3O82y1TFIaKpgGzPfbBl5ThexMhgV bfLnO/bty35Dms0e0oz8+NVk2cSsumOVeNV8o8gR75Upru1f X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Queue-Id: 0FEF8C0007 X-Rspam-User: X-Stat-Signature: sg4mqbt1esj7rsnjfuqc6ize8hfjpu78 X-Rspamd-Server: rspam03 X-HE-Tag: 1711354866-189344 X-HE-Meta: U2FsdGVkX18BD0zK7+B8YX7QMtXbwa/ruHluwAZrdBVJA69UhdDYEx9b9KgmO2NqgpJFyRH5ACueWd5Ciu+aAn92aZSJW1oEY2NF3X+AF8je/z29jAj5UKHuJO4kIBFar3vL203l/zOsODzl//rsnqMP3QBOCTOq8FZuHD0oMVpZ5gRariU7iq8puIaxKO7qQzXtTnCGDBMbbKLWn2f4XQYkbubodaSocnsUDWxKLsz1ChBZQNzykXjsW4xf5Ncds2zsyX5vgz48VLqlOg3w8ynNFdvyj259IROqJHzezpO6O0O0DWskv2Xz7wCiJQBvmUxd/8aYyKlcXOeCkdg51yS8I7gWU0ligeyTNvRPA5EaWxv/kh8Q0Q8aRHE3BGNsP3U/4D1A/D8hq67FbQsJ0qQqDHDHVt4rVihbCOP+Dp1CH7wOAqx7zn7/TcCOc2uSAqrkzHGCpIw5SfjJ7jWRbDOqh2Mx+xnlat8mlRNMpGgmbge2oX/RWJAwEq91mKlz0JloBQ6wwp9aOs//HDRv4VGnX4Py3feT/7b8abhGgVRCYnmmGVhfBg3bfh7MnGPtEMqAzsDS08SX9LSqqmVuT2c85fyi2cyPcchAkoB3NuXAWU0d+uuDeCxUzH4n66WDmOmVXgt0vcsXy7kEnuE5JBfTAD2gyiR5ejWE+G6k/3ztt5gZPHzAfp4JMeaCSVegQq3FeVgwlWgfslTL58WHS9TMyL6MPrdSQU03wRTqE+Y/9nimz3zfi3cD8y4NqKRcXa3s94iWl/RhNMf8X8Y6CIzRJ+4F2kH7UJiexvSdJNfdqbx6aYGs1k3sdvZquyvAwLXWljuZs0cX9Lh3eY0c6pSKf7eYQg08AJX/jnOdUdI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The MEMCG_KMEM integration with slab currently relies on two hooks during allocation. memcg_slab_pre_alloc_hook() determines the objcg and charges it, and memcg_slab_post_alloc_hook() assigns the objcg pointer to the allocated object(s). As Linus pointed out, this is unnecessarily complex. Failing to charge due to memcg limits should be rare, so we can optimistically allocate the object(s) and do the charging together with assigning the objcg pointer in a single post_alloc hook. In the rare case the charging fails, we can free the object(s) back. This simplifies the code (no need to pass around the objcg pointer) and potentially allows to separate charging from allocation in cases where it's common that the allocation would be immediately freed, and the memcg handling overhead could be saved. Suggested-by: Linus Torvalds Link: https://lore.kernel.org/all/CAHk-=whYOOdM7jWy5jdrAm8LxcgCMFyk2bt8fYYvZzM4U-zAQA@mail.gmail.com/ Reviewed-by: Roman Gushchin Reviewed-by: Chengming Zhou Signed-off-by: Vlastimil Babka Reported-by: Aishwarya TCV Signed-off-by: Vlastimil Babka Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt --- mm/slub.c | 180 +++++++++++++++++++++++++++----------------------------------- 1 file changed, 77 insertions(+), 103 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1bb2a93cf7b6..2440984503c8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1887,23 +1887,36 @@ static inline size_t obj_full_size(struct kmem_cache *s) return s->size + sizeof(struct obj_cgroup *); } -/* - * Returns false if the allocation should fail. - */ -static bool __memcg_slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t objects, gfp_t flags) +static bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, + gfp_t flags, size_t size, + void **p) { + struct obj_cgroup *objcg; + struct slab *slab; + unsigned long off; + size_t i; + /* * The obtained objcg pointer is safe to use within the current scope, * defined by current task or set_active_memcg() pair. * obj_cgroup_get() is used to get a permanent reference. */ - struct obj_cgroup *objcg = current_obj_cgroup(); + objcg = current_obj_cgroup(); if (!objcg) return true; + /* + * slab_alloc_node() avoids the NULL check, so we might be called with a + * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill + * the whole requested size. + * return success as there's nothing to free back + */ + if (unlikely(*p == NULL)) + return true; + + flags &= gfp_allowed_mask; + if (lru) { int ret; struct mem_cgroup *memcg; @@ -1916,71 +1929,51 @@ static bool __memcg_slab_pre_alloc_hook(struct kmem_cache *s, return false; } - if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) + if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) return false; - *objcgp = objcg; + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + + if (!slab_objcgs(slab) && + memcg_alloc_slab_cgroups(slab, s, flags, false)) { + obj_cgroup_uncharge(objcg, obj_full_size(s)); + continue; + } + + off = obj_to_index(s, slab, p[i]); + obj_cgroup_get(objcg); + slab_objcgs(slab)[off] = objcg; + mod_objcg_state(objcg, slab_pgdat(slab), + cache_vmstat_idx(s), obj_full_size(s)); + } + return true; } -/* - * Returns false if the allocation should fail. - */ +static void memcg_alloc_abort_single(struct kmem_cache *s, void *object); + static __fastpath_inline -bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, - struct obj_cgroup **objcgp, size_t objects, - gfp_t flags) +bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p) { - if (!memcg_kmem_online()) + if (likely(!memcg_kmem_online())) return true; if (likely(!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT))) return true; - return likely(__memcg_slab_pre_alloc_hook(s, lru, objcgp, objects, - flags)); -} - -static void __memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct obj_cgroup *objcg, - gfp_t flags, size_t size, - void **p) -{ - struct slab *slab; - unsigned long off; - size_t i; - - flags &= gfp_allowed_mask; - - for (i = 0; i < size; i++) { - if (likely(p[i])) { - slab = virt_to_slab(p[i]); - - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - continue; - } + if (likely(__memcg_slab_post_alloc_hook(s, lru, flags, size, p))) + return true; - off = obj_to_index(s, slab, p[i]); - obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; - mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); - } else { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - } + if (likely(size == 1)) { + memcg_alloc_abort_single(s, p); + *p = NULL; + } else { + kmem_cache_free_bulk(s, size, p); } -} - -static __fastpath_inline -void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, - gfp_t flags, size_t size, void **p) -{ - if (likely(!memcg_kmem_online() || !objcg)) - return; - return __memcg_slab_post_alloc_hook(s, objcg, flags, size, p); + return false; } static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, @@ -2019,44 +2012,23 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, __memcg_slab_free_hook(s, slab, p, objects, objcgs); } - -static inline -void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, - struct obj_cgroup *objcg) -{ - if (objcg) - obj_cgroup_uncharge(objcg, objects * obj_full_size(s)); -} #else /* CONFIG_MEMCG_KMEM */ static inline void memcg_free_slab_cgroups(struct slab *slab) { } -static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t objects, gfp_t flags) -{ - return true; -} - -static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct obj_cgroup *objcg, +static inline bool memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, gfp_t flags, size_t size, void **p) { + return true; } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { } - -static inline -void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, - struct obj_cgroup *objcg) -{ -} #endif /* CONFIG_MEMCG_KMEM */ /* @@ -3736,10 +3708,7 @@ noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) ALLOW_ERROR_INJECTION(should_failslab, ERRNO); static __fastpath_inline -struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t size, gfp_t flags) +struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) { flags &= gfp_allowed_mask; @@ -3748,14 +3717,11 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, if (unlikely(should_failslab(s, flags))) return NULL; - if (unlikely(!memcg_slab_pre_alloc_hook(s, lru, objcgp, size, flags))) - return NULL; - return s; } static __fastpath_inline -void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, +bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, gfp_t flags, size_t size, void **p, bool init, unsigned int orig_size) { @@ -3804,7 +3770,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, kmsan_slab_alloc(s, p[i], init_flags); } - memcg_slab_post_alloc_hook(s, objcg, flags, size, p); + return memcg_slab_post_alloc_hook(s, lru, flags, size, p); } /* @@ -3821,10 +3787,9 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) { void *object; - struct obj_cgroup *objcg = NULL; bool init = false; - s = slab_pre_alloc_hook(s, lru, &objcg, 1, gfpflags); + s = slab_pre_alloc_hook(s, gfpflags); if (unlikely(!s)) return NULL; @@ -3841,8 +3806,10 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list /* * When init equals 'true', like for kzalloc() family, only * @orig_size bytes might be zeroed instead of s->object_size + * In case this fails due to memcg_slab_post_alloc_hook(), + * object is set to NULL */ - slab_post_alloc_hook(s, objcg, gfpflags, 1, &object, init, orig_size); + slab_post_alloc_hook(s, lru, gfpflags, 1, &object, init, orig_size); return object; } @@ -4281,6 +4248,16 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, do_slab_free(s, slab, object, object, 1, addr); } +#ifdef CONFIG_MEMCG_KMEM +/* Do not inline the rare memcg charging failed path into the allocation path */ +static noinline +void memcg_alloc_abort_single(struct kmem_cache *s, void *object) +{ + if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) + do_slab_free(s, virt_to_slab(object), object, object, 1, _RET_IP_); +} +#endif + static __fastpath_inline void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) @@ -4616,29 +4593,26 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) { int i; - struct obj_cgroup *objcg = NULL; if (!size) return 0; - /* memcg and kmem_cache debug support */ - s = slab_pre_alloc_hook(s, NULL, &objcg, size, flags); + s = slab_pre_alloc_hook(s, flags); if (unlikely(!s)) return 0; i = __kmem_cache_alloc_bulk(s, flags, size, p); + if (unlikely(i == 0)) + return 0; /* * memcg and kmem_cache debug support and memory initialization. * Done outside of the IRQ disabled fastpath loop. */ - if (likely(i != 0)) { - slab_post_alloc_hook(s, objcg, flags, size, p, - slab_want_init_on_alloc(flags, s), s->object_size); - } else { - memcg_slab_alloc_error_hook(s, size, objcg); + if (unlikely(!slab_post_alloc_hook(s, NULL, flags, size, p, + slab_want_init_on_alloc(flags, s), s->object_size))) { + return 0; } - return i; } EXPORT_SYMBOL(kmem_cache_alloc_bulk); From patchwork Mon Mar 25 08:20:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13601619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A314CD11DF for ; Mon, 25 Mar 2024 08:21:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10CB56B0087; Mon, 25 Mar 2024 04:21:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 096FD6B0088; Mon, 25 Mar 2024 04:21:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDC5F6B0089; Mon, 25 Mar 2024 04:21:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CB5656B0087 for ; Mon, 25 Mar 2024 04:21:14 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7877C1C00BB for ; Mon, 25 Mar 2024 08:21:14 +0000 (UTC) X-FDA: 81934866468.25.CA7EA70 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf26.hostedemail.com (Postfix) with ESMTP id 48BAC14000D for ; Mon, 25 Mar 2024 08:21:11 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=knfY3406; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=B7ZjA+xY; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=hKp9U3Q7; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=1hSbzsvT; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711354872; a=rsa-sha256; cv=none; b=ziiRrXI8KvSfVacRrIlXv7+e1neprOuITE6/QX8zr/QNqiqbqiFL3c0ffwqSI2HAG5ZcBM l7xos3mOCYiz/M24mzdYfnhC2bos1+ntge8lYwFxwqyCGrP4jqYKSkxCA1DrmQk32Dd5pt Xu8XMGiRfVgQAg4jvC8o6hwADnpfv+w= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=knfY3406; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=B7ZjA+xY; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=hKp9U3Q7; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=1hSbzsvT; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711354872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=aBb1CKXkxVXb/giQraisqbV7TXEiaFGNrUtlJ2vHqR43t+V9x7IQ/tIE8nBq72uZrcV9GU kmtBQ73ltWjjMiBgUb5bgtccD38YWFW3Zg4K7nLFaHP4kq0niz1Ned3/4p1aR99k/gntZ1 1LhnspLuhkSIrQbymsRFo6/Dz3aLnuA= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id AC4D534F0D; Mon, 25 Mar 2024 08:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1711354867; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=knfY3406+qtqcyYqNGEeleYSScdJKPTl+tYGIkaYRnxsX9ObPJTUR+por0ziqXd0joWWGE P5it5IDBnHaAfUtlN9YPJsg4byjCfmGAuVd0cO9cJFu+Xg/D9p0MazBx/JQyr0OeyGmgJW 1f6/Ad0DP+bseTiTIj+lnDW1cESeiAg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1711354867; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=B7ZjA+xYbbMJNB2Q3+hxPYkHZY9dpu8bWhMq24q7D6wYSqD+NJlOBPFjS0YTr/aQMlnqqX q90VLofJMJsFbcAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1711354865; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=hKp9U3Q7NN+Eo+d57QIoPzID+K0Vacn4hOBbK1fRyjzXIqfEjAzQQygjEz7HeHs9MX/UW4 +DLp3Umjlr8Zw7+updM4oS9Hj28AU5X7Lrwu1B9yBLvEjpo1siC1RAc0LYG1WB3oJCYhzG chrafWrIvw/u1Iid5kVskLehNM7SOAc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1711354865; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=1hSbzsvTE7u8hb5Y6OxbRRAQkz5qIXUAUGMRilWI/nOyYEFYGA7J2DSZG3TtU60ld+rwXR 4tzmC+OC5g70LUAA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8877F137EB; Mon, 25 Mar 2024 08:21:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id gEAKIfEzAWZdHgAAD6G6ig (envelope-from ); Mon, 25 Mar 2024 08:21:05 +0000 From: Vlastimil Babka Date: Mon, 25 Mar 2024 09:20:33 +0100 Subject: [PATCH v2 2/2] mm, slab: move slab_memcg hooks to mm/memcontrol.c MIME-Version: 1.0 Message-Id: <20240325-slab-memcg-v2-2-900a458233a6@suse.cz> References: <20240325-slab-memcg-v2-0-900a458233a6@suse.cz> In-Reply-To: <20240325-slab-memcg-v2-0-900a458233a6@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara , Shakeel Butt Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7960; i=vbabka@suse.cz; h=from:subject:message-id; bh=ZXVKQAcpsecEUJrLf6SMbGT4m0x0VupUvlNRbOOhWz8=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmATPtqqfjDNILmUB7ToGV79nSfofuBNG/8B0z3 V/UTeKLnsmJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZgEz7QAKCRC74LB10kWI mqxyB/9AqeFQWjXz1ARTA0lp91IjDyuq3WsUEJJaVjfyJOX42Uuqle+axIyEYvNVG3MvhKbpIxQ D9deAf2gsFjJaGw3Ec8C3p42QB0e4mgHNqIss7ZWN42CKeEgIQc4ognHl5WC/Uj6hlEN4OgyhmS mxHCNXMotOhu61sRiwPtv0XfcSSL9cNMTm13UjruoA1fEf3pVtH+wS5u0tkFUp5UrhrARkB0yit JNLCbfW5hQxzM0J4QMfjcjKoV33zFcpAY7s5MkP7TM49hxr38/gFt16NRbiSl+yY0vUP59uQJqP NORledQ++brCmzAm7XFCvYPYw3ttV5gsYfC9BoRqwjMmvDPf X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 48BAC14000D X-Stat-Signature: ei813q46mhbfmyg19cbqk5dnatbyxbu7 X-HE-Tag: 1711354871-943390 X-HE-Meta: U2FsdGVkX1/ABcYaRzVd/oPw9fH0D16ptaGLL6T3n7dCVZ7wI9TFJNGNb0CDB56JD6MyIQDB3bAVUQWVPUpZZPHMIzym2ksZM+L1byaLmncSuQhmU7MLeYBhzhDWUiM27w4FljlIK+pCN8zD1DptcVD+tB1iEBhIlLFDQ1JsSzOf3h2bnOnJGswLMuEY7ENRuW2LAZUbvr+6YBUc2n4iS555ZaOSUHFK0Be+nhzQUVU7CYD2mMJnwqV1fORueCgXHul2mWi4TTy24dnetAJGou0PAIIRS9kSqFI3VfTFXpwzpfPUWb+6torVOHTEQU2/NNTneK7p7E2phMyI4e6egi3Kwav4293HFqaECa3m2D9ABk0qlMbKuzkrFB1zzGjjyOqy8U7/tLQxk07fXR2gzpM20Al+z1aG8/eq2Ae+Wzg6GE89Z0qeyZCQ3QUxkPNjOsXDO2t9C7+ggeGyeEC/2K4cPGMWhlN8K2Njb37Q+T7WX+j7RiXKkLJjqiYBbmysyt1K4BhMjCWJDAU7xJazBQq1Ayo2QiZi5TJI8MX7RT1u9TN1CMvSbt13n8tyCfQyA3JD+gSq/e9MljU9vhdojPBs8XyPNMLD1kSk8LrAGTor+mvdFudQdWLK2+VDo0KimY7gYdWIxntuUQ3ZsDQ846KU0zlO/updd5LMvpmrRnz6O5X5b5VOCkJ4nvRHr62VmDX45YgVlID2U6boYihPIV6WNEwR749l2SMu7GWy0lRDwvaOSRGA2U5SyUvDxMFtv2lasM+wMZG7BfUnlyjgkBUwZ3KfVyZhc0YnupT/r2pfq17jbxElBC9r7vBUF2c6jWnXWxfSVVRhKfnB7yCDvtjBE05k1RQXbDEybdjH8JiYlcKr3YzDefKdmqaXh6+2Bl/R1YBvIednHuTz1SccAcfjFlU46c8M4R15JCXNBzY/8J1RNAGjSmaHM06i9vt3Gi1LeRIXxPhPmt2EEyv 8Xt3mqCf 1HZbs6sQEIycofVsmX3MtTPuJ3CH2BtaxXZmTAgqAg4633oI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The hooks make multiple calls to functions in mm/memcontrol.c, including to th current_obj_cgroup() marked __always_inline. It might be faster to make a single call to the hook in mm/memcontrol.c instead. The hooks also don't use almost anything from mm/slub.c. obj_full_size() can move with the hooks and cache_vmstat_idx() to the internal mm/slab.h Reviewed-by: Roman Gushchin Signed-off-by: Vlastimil Babka Acked-by: Shakeel Butt --- mm/memcontrol.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 10 ++++++ mm/slub.c | 100 -------------------------------------------------------- 3 files changed, 100 insertions(+), 100 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index fabce2b50c69..fb101ff1f37c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3602,6 +3602,96 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) refill_obj_stock(objcg, size, true); } +static inline size_t obj_full_size(struct kmem_cache *s) +{ + /* + * For each accounted object there is an extra space which is used + * to store obj_cgroup membership. Charge it too. + */ + return s->size + sizeof(struct obj_cgroup *); +} + +bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p) +{ + struct obj_cgroup *objcg; + struct slab *slab; + unsigned long off; + size_t i; + + /* + * The obtained objcg pointer is safe to use within the current scope, + * defined by current task or set_active_memcg() pair. + * obj_cgroup_get() is used to get a permanent reference. + */ + objcg = current_obj_cgroup(); + if (!objcg) + return true; + + /* + * slab_alloc_node() avoids the NULL check, so we might be called with a + * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill + * the whole requested size. + * return success as there's nothing to free back + */ + if (unlikely(*p == NULL)) + return true; + + flags &= gfp_allowed_mask; + + if (lru) { + int ret; + struct mem_cgroup *memcg; + + memcg = get_mem_cgroup_from_objcg(objcg); + ret = memcg_list_lru_alloc(memcg, lru, flags); + css_put(&memcg->css); + + if (ret) + return false; + } + + if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) + return false; + + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + + if (!slab_objcgs(slab) && + memcg_alloc_slab_cgroups(slab, s, flags, false)) { + obj_cgroup_uncharge(objcg, obj_full_size(s)); + continue; + } + + off = obj_to_index(s, slab, p[i]); + obj_cgroup_get(objcg); + slab_objcgs(slab)[off] = objcg; + mod_objcg_state(objcg, slab_pgdat(slab), + cache_vmstat_idx(s), obj_full_size(s)); + } + + return true; +} + +void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects, struct obj_cgroup **objcgs) +{ + for (int i = 0; i < objects; i++) { + struct obj_cgroup *objcg; + unsigned int off; + + off = obj_to_index(s, slab, p[i]); + objcg = objcgs[off]; + if (!objcg) + continue; + + objcgs[off] = NULL; + obj_cgroup_uncharge(objcg, obj_full_size(s)); + mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), + -obj_full_size(s)); + obj_cgroup_put(objcg); + } +} #endif /* CONFIG_MEMCG_KMEM */ /* diff --git a/mm/slab.h b/mm/slab.h index d2bc9b191222..d8052dda78d7 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -536,6 +536,12 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } +static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; +} + #ifdef CONFIG_MEMCG_KMEM /* * slab_objcgs - get the object cgroups vector associated with a slab @@ -559,6 +565,10 @@ int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); +bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p); +void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects, struct obj_cgroup **objcgs); #else /* CONFIG_MEMCG_KMEM */ static inline struct obj_cgroup **slab_objcgs(struct slab *slab) { diff --git a/mm/slub.c b/mm/slub.c index 2440984503c8..87fa76c1105e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1865,12 +1865,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, #endif #endif /* CONFIG_SLUB_DEBUG */ -static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) -{ - return (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; -} - #ifdef CONFIG_MEMCG_KMEM static inline void memcg_free_slab_cgroups(struct slab *slab) { @@ -1878,79 +1872,6 @@ static inline void memcg_free_slab_cgroups(struct slab *slab) slab->memcg_data = 0; } -static inline size_t obj_full_size(struct kmem_cache *s) -{ - /* - * For each accounted object there is an extra space which is used - * to store obj_cgroup membership. Charge it too. - */ - return s->size + sizeof(struct obj_cgroup *); -} - -static bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - gfp_t flags, size_t size, - void **p) -{ - struct obj_cgroup *objcg; - struct slab *slab; - unsigned long off; - size_t i; - - /* - * The obtained objcg pointer is safe to use within the current scope, - * defined by current task or set_active_memcg() pair. - * obj_cgroup_get() is used to get a permanent reference. - */ - objcg = current_obj_cgroup(); - if (!objcg) - return true; - - /* - * slab_alloc_node() avoids the NULL check, so we might be called with a - * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill - * the whole requested size. - * return success as there's nothing to free back - */ - if (unlikely(*p == NULL)) - return true; - - flags &= gfp_allowed_mask; - - if (lru) { - int ret; - struct mem_cgroup *memcg; - - memcg = get_mem_cgroup_from_objcg(objcg); - ret = memcg_list_lru_alloc(memcg, lru, flags); - css_put(&memcg->css); - - if (ret) - return false; - } - - if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) - return false; - - for (i = 0; i < size; i++) { - slab = virt_to_slab(p[i]); - - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - continue; - } - - off = obj_to_index(s, slab, p[i]); - obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; - mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); - } - - return true; -} - static void memcg_alloc_abort_single(struct kmem_cache *s, void *object); static __fastpath_inline @@ -1976,27 +1897,6 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, return false; } -static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, - void **p, int objects, - struct obj_cgroup **objcgs) -{ - for (int i = 0; i < objects; i++) { - struct obj_cgroup *objcg; - unsigned int off; - - off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; - if (!objcg) - continue; - - objcgs[off] = NULL; - obj_cgroup_uncharge(objcg, obj_full_size(s)); - mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), - -obj_full_size(s)); - obj_cgroup_put(objcg); - } -} - static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects)