From patchwork Mon Mar 25 08:20:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13601619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A314CD11DF for ; Mon, 25 Mar 2024 08:21:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10CB56B0087; Mon, 25 Mar 2024 04:21:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 096FD6B0088; Mon, 25 Mar 2024 04:21:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDC5F6B0089; Mon, 25 Mar 2024 04:21:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CB5656B0087 for ; Mon, 25 Mar 2024 04:21:14 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7877C1C00BB for ; Mon, 25 Mar 2024 08:21:14 +0000 (UTC) X-FDA: 81934866468.25.CA7EA70 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf26.hostedemail.com (Postfix) with ESMTP id 48BAC14000D for ; Mon, 25 Mar 2024 08:21:11 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=knfY3406; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=B7ZjA+xY; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=hKp9U3Q7; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=1hSbzsvT; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711354872; a=rsa-sha256; cv=none; b=ziiRrXI8KvSfVacRrIlXv7+e1neprOuITE6/QX8zr/QNqiqbqiFL3c0ffwqSI2HAG5ZcBM l7xos3mOCYiz/M24mzdYfnhC2bos1+ntge8lYwFxwqyCGrP4jqYKSkxCA1DrmQk32Dd5pt Xu8XMGiRfVgQAg4jvC8o6hwADnpfv+w= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=knfY3406; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=B7ZjA+xY; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=hKp9U3Q7; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=1hSbzsvT; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711354872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=aBb1CKXkxVXb/giQraisqbV7TXEiaFGNrUtlJ2vHqR43t+V9x7IQ/tIE8nBq72uZrcV9GU kmtBQ73ltWjjMiBgUb5bgtccD38YWFW3Zg4K7nLFaHP4kq0niz1Ned3/4p1aR99k/gntZ1 1LhnspLuhkSIrQbymsRFo6/Dz3aLnuA= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id AC4D534F0D; Mon, 25 Mar 2024 08:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1711354867; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=knfY3406+qtqcyYqNGEeleYSScdJKPTl+tYGIkaYRnxsX9ObPJTUR+por0ziqXd0joWWGE P5it5IDBnHaAfUtlN9YPJsg4byjCfmGAuVd0cO9cJFu+Xg/D9p0MazBx/JQyr0OeyGmgJW 1f6/Ad0DP+bseTiTIj+lnDW1cESeiAg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1711354867; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=B7ZjA+xYbbMJNB2Q3+hxPYkHZY9dpu8bWhMq24q7D6wYSqD+NJlOBPFjS0YTr/aQMlnqqX q90VLofJMJsFbcAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1711354865; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=hKp9U3Q7NN+Eo+d57QIoPzID+K0Vacn4hOBbK1fRyjzXIqfEjAzQQygjEz7HeHs9MX/UW4 +DLp3Umjlr8Zw7+updM4oS9Hj28AU5X7Lrwu1B9yBLvEjpo1siC1RAc0LYG1WB3oJCYhzG chrafWrIvw/u1Iid5kVskLehNM7SOAc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1711354865; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZ5FujWhE4ePSZABvJJY1Hjqo6a8La3zymd24T6fptk=; b=1hSbzsvTE7u8hb5Y6OxbRRAQkz5qIXUAUGMRilWI/nOyYEFYGA7J2DSZG3TtU60ld+rwXR 4tzmC+OC5g70LUAA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8877F137EB; Mon, 25 Mar 2024 08:21:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id gEAKIfEzAWZdHgAAD6G6ig (envelope-from ); Mon, 25 Mar 2024 08:21:05 +0000 From: Vlastimil Babka Date: Mon, 25 Mar 2024 09:20:33 +0100 Subject: [PATCH v2 2/2] mm, slab: move slab_memcg hooks to mm/memcontrol.c MIME-Version: 1.0 Message-Id: <20240325-slab-memcg-v2-2-900a458233a6@suse.cz> References: <20240325-slab-memcg-v2-0-900a458233a6@suse.cz> In-Reply-To: <20240325-slab-memcg-v2-0-900a458233a6@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara , Shakeel Butt Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7960; i=vbabka@suse.cz; h=from:subject:message-id; bh=ZXVKQAcpsecEUJrLf6SMbGT4m0x0VupUvlNRbOOhWz8=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmATPtqqfjDNILmUB7ToGV79nSfofuBNG/8B0z3 V/UTeKLnsmJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZgEz7QAKCRC74LB10kWI mqxyB/9AqeFQWjXz1ARTA0lp91IjDyuq3WsUEJJaVjfyJOX42Uuqle+axIyEYvNVG3MvhKbpIxQ D9deAf2gsFjJaGw3Ec8C3p42QB0e4mgHNqIss7ZWN42CKeEgIQc4ognHl5WC/Uj6hlEN4OgyhmS mxHCNXMotOhu61sRiwPtv0XfcSSL9cNMTm13UjruoA1fEf3pVtH+wS5u0tkFUp5UrhrARkB0yit JNLCbfW5hQxzM0J4QMfjcjKoV33zFcpAY7s5MkP7TM49hxr38/gFt16NRbiSl+yY0vUP59uQJqP NORledQ++brCmzAm7XFCvYPYw3ttV5gsYfC9BoRqwjMmvDPf X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 48BAC14000D X-Stat-Signature: ei813q46mhbfmyg19cbqk5dnatbyxbu7 X-HE-Tag: 1711354871-943390 X-HE-Meta: U2FsdGVkX1/ABcYaRzVd/oPw9fH0D16ptaGLL6T3n7dCVZ7wI9TFJNGNb0CDB56JD6MyIQDB3bAVUQWVPUpZZPHMIzym2ksZM+L1byaLmncSuQhmU7MLeYBhzhDWUiM27w4FljlIK+pCN8zD1DptcVD+tB1iEBhIlLFDQ1JsSzOf3h2bnOnJGswLMuEY7ENRuW2LAZUbvr+6YBUc2n4iS555ZaOSUHFK0Be+nhzQUVU7CYD2mMJnwqV1fORueCgXHul2mWi4TTy24dnetAJGou0PAIIRS9kSqFI3VfTFXpwzpfPUWb+6torVOHTEQU2/NNTneK7p7E2phMyI4e6egi3Kwav4293HFqaECa3m2D9ABk0qlMbKuzkrFB1zzGjjyOqy8U7/tLQxk07fXR2gzpM20Al+z1aG8/eq2Ae+Wzg6GE89Z0qeyZCQ3QUxkPNjOsXDO2t9C7+ggeGyeEC/2K4cPGMWhlN8K2Njb37Q+T7WX+j7RiXKkLJjqiYBbmysyt1K4BhMjCWJDAU7xJazBQq1Ayo2QiZi5TJI8MX7RT1u9TN1CMvSbt13n8tyCfQyA3JD+gSq/e9MljU9vhdojPBs8XyPNMLD1kSk8LrAGTor+mvdFudQdWLK2+VDo0KimY7gYdWIxntuUQ3ZsDQ846KU0zlO/updd5LMvpmrRnz6O5X5b5VOCkJ4nvRHr62VmDX45YgVlID2U6boYihPIV6WNEwR749l2SMu7GWy0lRDwvaOSRGA2U5SyUvDxMFtv2lasM+wMZG7BfUnlyjgkBUwZ3KfVyZhc0YnupT/r2pfq17jbxElBC9r7vBUF2c6jWnXWxfSVVRhKfnB7yCDvtjBE05k1RQXbDEybdjH8JiYlcKr3YzDefKdmqaXh6+2Bl/R1YBvIednHuTz1SccAcfjFlU46c8M4R15JCXNBzY/8J1RNAGjSmaHM06i9vt3Gi1LeRIXxPhPmt2EEyv 8Xt3mqCf 1HZbs6sQEIycofVsmX3MtTPuJ3CH2BtaxXZmTAgqAg4633oI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The hooks make multiple calls to functions in mm/memcontrol.c, including to th current_obj_cgroup() marked __always_inline. It might be faster to make a single call to the hook in mm/memcontrol.c instead. The hooks also don't use almost anything from mm/slub.c. obj_full_size() can move with the hooks and cache_vmstat_idx() to the internal mm/slab.h Reviewed-by: Roman Gushchin Signed-off-by: Vlastimil Babka Acked-by: Shakeel Butt --- mm/memcontrol.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 10 ++++++ mm/slub.c | 100 -------------------------------------------------------- 3 files changed, 100 insertions(+), 100 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index fabce2b50c69..fb101ff1f37c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3602,6 +3602,96 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) refill_obj_stock(objcg, size, true); } +static inline size_t obj_full_size(struct kmem_cache *s) +{ + /* + * For each accounted object there is an extra space which is used + * to store obj_cgroup membership. Charge it too. + */ + return s->size + sizeof(struct obj_cgroup *); +} + +bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p) +{ + struct obj_cgroup *objcg; + struct slab *slab; + unsigned long off; + size_t i; + + /* + * The obtained objcg pointer is safe to use within the current scope, + * defined by current task or set_active_memcg() pair. + * obj_cgroup_get() is used to get a permanent reference. + */ + objcg = current_obj_cgroup(); + if (!objcg) + return true; + + /* + * slab_alloc_node() avoids the NULL check, so we might be called with a + * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill + * the whole requested size. + * return success as there's nothing to free back + */ + if (unlikely(*p == NULL)) + return true; + + flags &= gfp_allowed_mask; + + if (lru) { + int ret; + struct mem_cgroup *memcg; + + memcg = get_mem_cgroup_from_objcg(objcg); + ret = memcg_list_lru_alloc(memcg, lru, flags); + css_put(&memcg->css); + + if (ret) + return false; + } + + if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) + return false; + + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + + if (!slab_objcgs(slab) && + memcg_alloc_slab_cgroups(slab, s, flags, false)) { + obj_cgroup_uncharge(objcg, obj_full_size(s)); + continue; + } + + off = obj_to_index(s, slab, p[i]); + obj_cgroup_get(objcg); + slab_objcgs(slab)[off] = objcg; + mod_objcg_state(objcg, slab_pgdat(slab), + cache_vmstat_idx(s), obj_full_size(s)); + } + + return true; +} + +void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects, struct obj_cgroup **objcgs) +{ + for (int i = 0; i < objects; i++) { + struct obj_cgroup *objcg; + unsigned int off; + + off = obj_to_index(s, slab, p[i]); + objcg = objcgs[off]; + if (!objcg) + continue; + + objcgs[off] = NULL; + obj_cgroup_uncharge(objcg, obj_full_size(s)); + mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), + -obj_full_size(s)); + obj_cgroup_put(objcg); + } +} #endif /* CONFIG_MEMCG_KMEM */ /* diff --git a/mm/slab.h b/mm/slab.h index d2bc9b191222..d8052dda78d7 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -536,6 +536,12 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } +static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; +} + #ifdef CONFIG_MEMCG_KMEM /* * slab_objcgs - get the object cgroups vector associated with a slab @@ -559,6 +565,10 @@ int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); +bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p); +void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects, struct obj_cgroup **objcgs); #else /* CONFIG_MEMCG_KMEM */ static inline struct obj_cgroup **slab_objcgs(struct slab *slab) { diff --git a/mm/slub.c b/mm/slub.c index 2440984503c8..87fa76c1105e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1865,12 +1865,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, #endif #endif /* CONFIG_SLUB_DEBUG */ -static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) -{ - return (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; -} - #ifdef CONFIG_MEMCG_KMEM static inline void memcg_free_slab_cgroups(struct slab *slab) { @@ -1878,79 +1872,6 @@ static inline void memcg_free_slab_cgroups(struct slab *slab) slab->memcg_data = 0; } -static inline size_t obj_full_size(struct kmem_cache *s) -{ - /* - * For each accounted object there is an extra space which is used - * to store obj_cgroup membership. Charge it too. - */ - return s->size + sizeof(struct obj_cgroup *); -} - -static bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - gfp_t flags, size_t size, - void **p) -{ - struct obj_cgroup *objcg; - struct slab *slab; - unsigned long off; - size_t i; - - /* - * The obtained objcg pointer is safe to use within the current scope, - * defined by current task or set_active_memcg() pair. - * obj_cgroup_get() is used to get a permanent reference. - */ - objcg = current_obj_cgroup(); - if (!objcg) - return true; - - /* - * slab_alloc_node() avoids the NULL check, so we might be called with a - * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill - * the whole requested size. - * return success as there's nothing to free back - */ - if (unlikely(*p == NULL)) - return true; - - flags &= gfp_allowed_mask; - - if (lru) { - int ret; - struct mem_cgroup *memcg; - - memcg = get_mem_cgroup_from_objcg(objcg); - ret = memcg_list_lru_alloc(memcg, lru, flags); - css_put(&memcg->css); - - if (ret) - return false; - } - - if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) - return false; - - for (i = 0; i < size; i++) { - slab = virt_to_slab(p[i]); - - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - continue; - } - - off = obj_to_index(s, slab, p[i]); - obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; - mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); - } - - return true; -} - static void memcg_alloc_abort_single(struct kmem_cache *s, void *object); static __fastpath_inline @@ -1976,27 +1897,6 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, return false; } -static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, - void **p, int objects, - struct obj_cgroup **objcgs) -{ - for (int i = 0; i < objects; i++) { - struct obj_cgroup *objcg; - unsigned int off; - - off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; - if (!objcg) - continue; - - objcgs[off] = NULL; - obj_cgroup_uncharge(objcg, obj_full_size(s)); - mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), - -obj_full_size(s)); - obj_cgroup_put(objcg); - } -} - static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects)