From patchwork Wed Sep 27 15:08:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13400987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 551FAE82CA6 for ; Wed, 27 Sep 2023 15:08:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C742E6B0163; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C24D16B0165; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC4DB6B0167; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9D5F76B0163 for ; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 781421C9588 for ; Wed, 27 Sep 2023 15:08:58 +0000 (UTC) X-FDA: 81282709956.04.490408E Received: from out-195.mta1.migadu.com (out-195.mta1.migadu.com [95.215.58.195]) by imf27.hostedemail.com (Postfix) with ESMTP id 8179640031 for ; Wed, 27 Sep 2023 15:08:56 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Q8hMpLXl; spf=pass (imf27.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.195 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827336; a=rsa-sha256; cv=none; b=xA+M06NeQtA+M6JjBKZ7nKvFclrwP5snO2fk3zkiw+sFDilDpRPBhNhya6mjPpRph4bwXT MGDD2b0DAnzCq3jgDnbndBbF2Sm1UCeNfevnpUKCb7M1+pMV0koVQjPXJQTLf6jazI6iQT v2OJDX/4yoiw1IufY84YRTKwle1btxo= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Q8hMpLXl; spf=pass (imf27.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.195 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827336; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=csA9lqKqSWug/CUi1CRNnTt3apPTiWEZn90EWKQHAwp5lsNX3kA6sAsuW4beNhxB8n+D2Q 86aCLuf+Qzd14Y4sssscRbNrv9w9PCy/ePF2PVGsZ7QX7dTC4638IyeH+MbmtPWDZ6Xs6I mxK1umW4NW2ka77j7scL9lyP9w7bbas= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=Q8hMpLXlwyHYuhYTY7QmN3ucJ4BymSYDpw0OZAW82OI1HjO9QhUqV7hIgxRE3/+F1Zdf1X 9pwvp7A1SXynG+K4IEOnW40uYr4B8pTu6BDJ26Y+WSYnLIfaJXQFLXbhSCwQWYyEHZxai0 mR7jQfl6X8WWs0zHRyt6d4SZPcg5cVY= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Wed, 27 Sep 2023 08:08:28 -0700 Message-ID: <20230927150832.335132-2-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8179640031 X-Stat-Signature: 8y9wo3gh1qkgoma89j76fsgwbco88fko X-Rspam-User: X-HE-Tag: 1695827336-418902 X-HE-Meta: U2FsdGVkX18TW1YQE9NePlw2rra9WjPYA6k8QQOti65JcK1pvh2lyvotJ2hT4MKFOIEDmi4LXSm8vCdmDyfIQlvFzJRdtekzpFjAmCnIz5IfLdFnJvRylHlOiNNBOHeIIaSY0V0DJjTxbiplEr8PuwR84NcIFp/rHt6QRv9u/b3kU6FdXj7oQ3VsQPDffL6XR6RYlEmnTh0ClUr5tT/tbq1iIzhXeq+RAj4rcB7U/Hr8uPT0+NjAnMLX0GgGW1aU2ot3cbMGBtJlaKtJoWHyWghZ92VQcslio9BP0cq4U/5Yd7RzQhJdcR6rHjTzeZFQ0NY6f49yv7HdvxSK1egLNzVavm6apTJj0ldjcC7DnJlieUCMhPWOQPaCZDEvNiP5dgK/ZmaPL5zLAWcHPte48Onyr6Qy0pwSLzTNbVju0KGHcawEZyixmkIrH/XhWrx7QJP35eLbswW5wRhR/DdQPVT1/zsfTFEEcqL5SPR/cCqTdbDTOvkIL8Sqze2L9GcMIbF09CGG/B2sEQQc+twF8767yc/IvI6+42ro8msUJa1uJ4mwCRhD1cho/RHLZehNsj5UuRpcnDXScngtGk6U159M/8mXKBLFKv9OG19A2/4O6BMWfztLVSKZtC4dbYnJAjbQWlNt0uVZ3X32UeuDhYyptSUI1G3gsqQbc7dlPhgH0K4qRkPFNqUEyifBxHzReuLiPoIwBhV76iFKDTxllXJiKNDB85/pDAF9c5I/jg6Zw6y/XY4vvqDosRHTjBTTWHFPEmci77dvIU/ZBoLEKOFZMwSQQd2GbymcWPuS8L5bHGGpzmkKpXa7E0YJugFRAVioPcIlJ5FGM9h5+GhcLjmKO52tSJXsQ8A9yGlI4fj00Iony9V9aixBOcAVM3UsjyS1tBHOrCs95ReN3ZnB4b/I6sb2KSNlh/2nkleqBmcvewnlaVFAsa6+W4uC3iQ6nfMlO3Xb+M7WqLy0cIq JkmtLOdL yi0KJJFm8x/Y4BVY4Uyt5a92Esfo7cY3KBtWJ5OGTQIVrmRgXM5moP5AfHyl3P1nTcagfpRBjwxlCl44EOM45ul0Vrji8UHINN6pgoQ8QLaJ09ZV7xJCmhAMF7kF7bZYxt7ErAnNMIR/eTA9gKKDm+j3KDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); From patchwork Wed Sep 27 15:08:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13400988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82315E82CA5 for ; Wed, 27 Sep 2023 15:09:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4EB06B0167; Wed, 27 Sep 2023 11:09:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD71F6B0168; Wed, 27 Sep 2023 11:09:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4FD46B0169; Wed, 27 Sep 2023 11:09:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 924036B0167 for ; Wed, 27 Sep 2023 11:09:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6716A1A0A7B for ; Wed, 27 Sep 2023 15:09:00 +0000 (UTC) X-FDA: 81282710040.03.ED8C9DD Received: from out-202.mta1.migadu.com (out-202.mta1.migadu.com [95.215.58.202]) by imf05.hostedemail.com (Postfix) with ESMTP id B336B100017 for ; Wed, 27 Sep 2023 15:08:58 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=UsUhkDML; spf=pass (imf05.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.202 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827338; a=rsa-sha256; cv=none; b=IOKvYRs0Jxp57RPUH0jrBMKaGB8gzkSBUh2L+bIbru1IZAIGM1GplwPKOHEWoHxMdCUvEU 51i2AWXo5RJxSYlMd6P7WffBOqEJxHSywfz8yLZo3uCxXJ5HxbwsecfLe4DM5pL/kbu2tQ nBRq+iaBV490OYKRUl4dlLoHVTY/cwM= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=UsUhkDML; spf=pass (imf05.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.202 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rcjc8iZxDm0DIYnvNFEg9AlvdD1o0gQiWksXrzSEaP4=; b=LKc/dtnTQ98iYHqKD3D3tIulHhSlGUi0UQv44ERFGbTCn3SzKNkpPhBTkqp3pC1efo9m65 GdGrrgSyklmkWRSTA4XKzMlIDscw4Wcge3yZMg4w3xsGfJ1SF5BUTmZ9EO9xjeXCDy8TLI Rrobx/89oUbH3iKse/aaCa4yRq2wpcs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rcjc8iZxDm0DIYnvNFEg9AlvdD1o0gQiWksXrzSEaP4=; b=UsUhkDMLWoqWyU+5nbHGAFCVQZ5r2vg+XsHsShhnKPKLyiziXVyVxR9lcV7QyDWCZsJ8KL 7K/IZURrLDyBSSujwFVKE2aqWxPUYANqDoYmQ7nFA92KgzQTpifzj6n13nvCNmo4/FEFGw NyMkSUQBXz32JTg1sqlghlRIHfkR4f8= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 2/5] mm: kmem: add direct objcg pointer to task_struct Date: Wed, 27 Sep 2023 08:08:29 -0700 Message-ID: <20230927150832.335132-3-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B336B100017 X-Stat-Signature: obendnfw1unh7kytie59d4sr15p9xybq X-Rspam-User: X-HE-Tag: 1695827338-938302 X-HE-Meta: U2FsdGVkX1+yerSukwiJYkQfW4zAF/mb2X/jfyWm8nuqBCgKUUit8uLUviIybKFc7GKpYA9EhZkmy9XMsYoU2DGY2L2/CKRh/3xu6HHC8CZYhFDZOKehpMR+NmyAlYmkx2ttfAaVzRkchv3BrNZS8kuEJ5Qo+DEhq8p1lHfWZpPBemO5aeg/h5H1Yfaa8ZpMGJBQmbM5dk/PBIg1P3l0BzTg17GsDUbQvfBWHHL2r0ouSLoSm7Fe1em1oADv3mAifCuwIgdzVuszaD9fdEC/UiCLkKIs5f3dgnuIxTyLnXvpaXXyxCkuBjg2IS8n0orCo+k5oYXZhgNh88mRrFF79W91v/w22W4rgcGDJsb7pf1BkMfLShuKlogFAskSvSpye5UmosLeudCv4DUaBS02Djb9LPw5RwJvjXQtP3NMs6i9XUFX4JDo4lXsQe4QjJM6YHuCR6rSdeM8EzgLwlbUPhLnOpCGO1c/GlHPp/4xXaaER94qZV2BkQ73huP9zOHug85ClxqmY5B4nxaUXe9dwUvpfAgYUc0WLJMXv7INEq3mZfOhEUCEKgAaCZ3TpCr75uwASmqi8hP7tSeP5Lm0bfXpubEbdImL0H08N9hLRVv80iIPkNBO+HQ56DpwweRzW8WFkjWgtgRn2ivnymVdKKn+6tlr8cfIx8Uk9+QdEguxzeSqduivdr/Blxeeo0B/gOVvo25qNwe4q5YE+Cnxkje2k2CGtdpOFfViQdtTjA0G1ebeLV5VWaPBOyvUuF53/z49QG8xmW55PScARvdo1HGGhDFsTcxmbyV0lgQA43ZLXXUaB6kiI7/6tAUT/qjBfg2dFbBq50mdop/Z67JkL4947rqpEihjmSaPESbxssbYda6TiWdxxMREkUPitYfNHTzbwcMYJqflibwZrCkVr3prQyBAnk+nzgc2jk1kS7Vw3ewMHk5btthT02aghnzNYa+YCgx05Am31+zZqKm FrYKkbJp VCX2QqCnUIkk9ullSMgShWhXju1y6zhdtb/rOAAJo+Y+6REynOwUjNKRLZz6ztV/qtsYu8jrfgl7ag3pe5BTQBGNv6nr4aKIdoYKiIHlyF1g7djNZPXICHb5fMrvJQ9PgPukj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg(). Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder. To avoid any costly synchronization the following rules are applied: 1) A task sets it's objcg pointer itself. 2) If a task is being migrated to another cgroup, the least significant bit of the objcg pointer is set. 3) On the allocation path the objcg pointer is obtained locklessly using the READ_ONCE() macro and the least significant bit is checked. If it set, the task updates it's objcg before proceeding with an allocation. 4) Operations 1) and 4) are synchronized via a new spinlock, so that if a task is moved twice, the update bit can't be lost. This allows to keep the hot path fully lockless. Because the task is keeping a reference to the objcg, it can't go away while the task is alive. This commit doesn't change the way the remote memcg charging works. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 10 ++++ include/linux/sched.h | 4 ++ mm/memcontrol.c | 107 +++++++++++++++++++++++++++++++++---- 3 files changed, 112 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..84425bfe4124 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -553,6 +553,16 @@ static inline bool folio_memcg_kmem(struct folio *folio) return folio->memcg_data & MEMCG_DATA_KMEM; } +static inline bool current_objcg_needs_update(struct obj_cgroup *objcg) +{ + return (struct obj_cgroup *)((unsigned long)objcg & 0x1); +} + +static inline struct obj_cgroup * +current_objcg_clear_update_flag(struct obj_cgroup *objcg) +{ + return (struct obj_cgroup *)((unsigned long)objcg & ~0x1); +} #else static inline bool folio_memcg_kmem(struct folio *folio) diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac385f7..60de42715b56 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1443,6 +1443,10 @@ struct task_struct { struct mem_cgroup *active_memcg; #endif +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *objcg; +#endif + #ifdef CONFIG_BLK_CGROUP struct gendisk *throttle_disk; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 16ac2a5838fb..7f33a503d600 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3001,6 +3001,47 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return objcg; } +static DEFINE_SPINLOCK(current_objcg_lock); + +static struct obj_cgroup *current_objcg_update(struct obj_cgroup *old) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + unsigned long flags; + + old = current_objcg_clear_update_flag(old); + if (old) + obj_cgroup_put(old); + + spin_lock_irqsave(¤t_objcg_lock, flags); + rcu_read_lock(); + memcg = mem_cgroup_from_task(current); + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { + objcg = rcu_dereference(memcg->objcg); + if (objcg && obj_cgroup_tryget(objcg)) + break; + objcg = NULL; + } + rcu_read_unlock(); + + WRITE_ONCE(current->objcg, objcg); + spin_unlock_irqrestore(¤t_objcg_lock, flags); + + return objcg; +} + +static inline void current_objcg_set_needs_update(struct task_struct *task) +{ + struct obj_cgroup *objcg; + unsigned long flags; + + spin_lock_irqsave(¤t_objcg_lock, flags); + objcg = READ_ONCE(task->objcg); + objcg = (struct obj_cgroup *)((unsigned long)objcg | 0x1); + WRITE_ONCE(task->objcg, objcg); + spin_unlock_irqrestore(¤t_objcg_lock, flags); +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct mem_cgroup *memcg; @@ -3008,19 +3049,26 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) if (in_task()) { memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; - /* Memcg to charge can't be determined. */ - if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) - return NULL; + objcg = READ_ONCE(current->objcg); + if (unlikely(current_objcg_needs_update(objcg))) + objcg = current_objcg_update(objcg); + + if (objcg) { + obj_cgroup_get(objcg); + return objcg; + } } else { memcg = this_cpu_read(int_active_memcg); - if (likely(!memcg)) - return NULL; + if (unlikely(memcg)) + goto from_memcg; } + return NULL; +from_memcg: rcu_read_lock(); - if (!memcg) - memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); return objcg; @@ -6345,6 +6393,22 @@ static void mem_cgroup_move_task(void) mem_cgroup_clear_mc(); } } + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_fork(struct task_struct *task) +{ + task->objcg = (struct obj_cgroup *)0x1; +} + +static void mem_cgroup_exit(struct task_struct *task) +{ + struct obj_cgroup *objcg = current_objcg_clear_update_flag(task->objcg); + + if (objcg) + obj_cgroup_put(objcg); +} +#endif + #else /* !CONFIG_MMU */ static int mem_cgroup_can_attach(struct cgroup_taskset *tset) { @@ -6359,7 +6423,7 @@ static void mem_cgroup_move_task(void) #endif #ifdef CONFIG_LRU_GEN -static void mem_cgroup_attach(struct cgroup_taskset *tset) +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) { struct task_struct *task; struct cgroup_subsys_state *css; @@ -6377,10 +6441,29 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) task_unlock(task); } #else +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_LRU_GEN */ + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) + current_objcg_set_needs_update(task); +} +#else +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_MEMCG_KMEM */ + +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) static void mem_cgroup_attach(struct cgroup_taskset *tset) { + mem_cgroup_lru_gen_attach(tset); + mem_cgroup_kmem_attach(tset); } -#endif /* CONFIG_LRU_GEN */ +#endif static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) { @@ -6824,9 +6907,15 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) .attach = mem_cgroup_attach, +#endif .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, +#ifdef CONFIG_MEMCG_KMEM + .fork = mem_cgroup_fork, + .exit = mem_cgroup_exit, +#endif .dfl_cftypes = memory_files, .legacy_cftypes = mem_cgroup_legacy_files, .early_init = 0, From patchwork Wed Sep 27 15:08:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13400989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2C8FE82CA7 for ; Wed, 27 Sep 2023 15:09:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 302896B0169; Wed, 27 Sep 2023 11:09:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AE9A6B016B; Wed, 27 Sep 2023 11:09:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 176776B016C; Wed, 27 Sep 2023 11:09:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 08C4E6B0169 for ; Wed, 27 Sep 2023 11:09:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D7DA6120A12 for ; Wed, 27 Sep 2023 15:09:02 +0000 (UTC) X-FDA: 81282710124.03.911C1C5 Received: from out-202.mta1.migadu.com (out-202.mta1.migadu.com [95.215.58.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 2248F1C0040 for ; Wed, 27 Sep 2023 15:09:00 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=pNocy6o9; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf20.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.202 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827341; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NJPTphOT9Ah3iL7MZ9KbERD0ROIs5HQQzvQjuLNJnns=; b=Q8bsOH/LzBFdIhYlDaGnkobvze2vROtTq1sO3TNTHeKwZNYo3aFLlM4oJoHKT+xii6TKOP t4+r8qpZTrSWyh92BsMrNoJBo/7yx4JsQoKPRDL8diaXyiVMi0NieDQcEpo+AkBh0xRRfm UA1ppKhd9eYLNsOdyOs4xbwRZqe8UTs= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=pNocy6o9; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf20.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.202 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827341; a=rsa-sha256; cv=none; b=S0fxoSSQkLiFdqexX6/ZikmW9t9qA5yGdd//Uhzs2noh7rxu76GnZ1KgWspSmK254Ciuwo +NJVfIbNP8K4u20M4vgDz1tKZJTFOVI59wNYA3FUsICZm7gfbZ6rp9YFmSWYIOJihi1Yzs rz7bs5EvFKeNFuYq1Aj1XM1wvv3bVL8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827339; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NJPTphOT9Ah3iL7MZ9KbERD0ROIs5HQQzvQjuLNJnns=; b=pNocy6o9sigu+gpBp2nZU0nrp9UmQjhqPdMh8hND8MX7ixqhRzLWJmdV9/s/k8Rs1A5Qa5 6IMESlh4qTyehhfmr0rDc9in0xsrFhOJIDRwOsOUFW2sCFbjw/rB+qADTL7a3cw1BDr2Zq iq5OmF6L+8t6+ySa0+DaftZUm1opJI4= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 3/5] mm: kmem: make memcg keep a reference to the original objcg Date: Wed, 27 Sep 2023 08:08:30 -0700 Message-ID: <20230927150832.335132-4-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 2248F1C0040 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: o65qc1xuiwhupigpya3e4jk7w6t15ufu X-HE-Tag: 1695827340-109954 X-HE-Meta: U2FsdGVkX1+insPqRi9AQuhP/wXUZY9wzuF2oJyTn8wCT1F0erbBXeBWnGlwkL8hsG/sa0nChLR5AznNaYQhcmRg7lK7htzZVzoOTpwjJww01cQHTBuiZQ3ej66YHayGuT5ngRcRGR7D5AclXvXrbZYCX5RkWxu3WsapPfWsF6Hv117VzbzYdd0oaWptTMJTlRh/X824mInDD9QW9ksagWmLeUs/yHBfPeIiXiONGO4YmvmvIErvP17jeFWICJ0GPC90/02R7NClowT7FFSVjLR/rjigd18N+hm6FtK+7EUz6DJ/3nMMPjYVAjgCyLjR8vc8/2wyDIcoWmhhx1mklkVZXKnf6JgTbL7T88SyjFn3i0k/aTuaUA1oGq2XbqzLZA6YCYrNgeMbXqKohxmA6zXizF46y/LEHDQZS1pOXyPJgl/sXHr5yO1+yyQ5Wf3uBjZDd6+uNswvkz4BfkLELCr/NtwzgIJrPVK9F4Rwo9xdSC5zLpjYlDszzMHLMKGnX+tEWQOt7ZGAkFveiHkIzuTFrkPGBDkzckB9XkQdtUhUTM4x8a1nGIxsqqn+nGPa+2idAf/yMjemYZtNZp1dg6ivsI2g+eFbS1i0necbA+VBBxkn59k8i4xDk2BFgJJOxYoAmZZaetyFU073GNlMzq8r35KTUw48dkGJYP+2pt8i2tIlyOwIrC8IGka0wHR8eF/dFgg5npXfPxqi2wCluZybOYZJC4f4spKNUCmPbchDU5CdcvTx9BjrTCjCH3ky3m8BT8F6lbWyyy7zIPcHd2sp5xWXDWLJSOSQSLK3hw8bTH1/UXFUh9IGazHSu+ED4CtER+PA05cvTPoN6viTg7H5Sv2l+anTSN6tmBYBmgzqkeQ53ED4kzRpJ1BqeMhfE/KPmw5N5FY1dSfeSe6Gs+iUnKZnZMSMAc2+46ZSBJKomQmF2LLlsRx0XZzA8fcQlaq8h6wpezMQ0k18BHD Yt/1NAvJ GFaZO5vD6T4tBaC5f7INq+W/G+LHwgMm9EZJkoxmKnCkyK2Ymb1B3cilbTe7zrwTgzdoxdbMnBJUSW00= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Keep a reference to the original objcg object for the entire life of a memcg structure. This allows to simplify the synchronization on the kernel memory allocation paths: pinning a (live) memcg will also pin the corresponding objcg. The memory overhead of this change is minimal because object cgroups usually outlive their corresponding memory cgroups even without this change, so it's only an additional pointer per memcg. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 8 +++++++- mm/memcontrol.c | 5 +++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 84425bfe4124..412ff0e8694d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -299,7 +299,13 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - struct obj_cgroup __rcu *objcg; + /* + * memcg->objcg is wiped out as a part of the objcg repaprenting + * process. memcg->orig_objcg preserves a pointer (and a reference) + * to the original objcg until the end of live of memcg. + */ + struct obj_cgroup __rcu *objcg; + struct obj_cgroup *orig_objcg; /* list of inherited objcgs, protected by objcg_lock */ struct list_head objcg_list; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7f33a503d600..4815f897758c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3803,6 +3803,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); + obj_cgroup_get(objcg); + memcg->orig_objcg = objcg; static_branch_enable(&memcg_kmem_online_key); @@ -5297,6 +5299,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) { int node; + if (memcg->orig_objcg) + obj_cgroup_put(memcg->orig_objcg); + for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); kfree(memcg->vmstats); From patchwork Wed Sep 27 15:08:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13400990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 711BFE82CA6 for ; Wed, 27 Sep 2023 15:09:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93FA86B016B; Wed, 27 Sep 2023 11:09:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F0DD6B016E; Wed, 27 Sep 2023 11:09:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CCE76B0170; Wed, 27 Sep 2023 11:09:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5B7A46B016B for ; Wed, 27 Sep 2023 11:09:05 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 267DC1A09AD for ; Wed, 27 Sep 2023 15:09:05 +0000 (UTC) X-FDA: 81282710250.27.6E7CC13 Received: from out-196.mta1.migadu.com (out-196.mta1.migadu.com [95.215.58.196]) by imf03.hostedemail.com (Postfix) with ESMTP id 5918720022 for ; Wed, 27 Sep 2023 15:09:03 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=c6ZuccPL; spf=pass (imf03.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.196 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jeQJ/eIJXU0AYA9yk+s4aO7OKAgQk2qdJmGkoMOLMDk=; b=8leeZ9Z1jf/UACgRZBCvWa2SCBZ6QPj8l0df4Y27xVuItk+VTCAcsOld5XOfb89Sqw912W txDAm/V82zc4xNkkSkp6ZiVliCasWMZSYKOkCmsZGJB2U9tAo08aBNUIGZoHA5SyCoLNQy PVOtQaQZI++7OZrf87UwzFfcWM4entE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827343; a=rsa-sha256; cv=none; b=cWWq0IyCOniXDjpA7Kjz14OnUwaULwCO4+jtq3b4KMmv7mU6JXT49PEnEuUmINbU7qE2O0 UBkRuHGeJ9+Gv6pHiqQnrZwKJvSs1d2gBPN7/nVyNomfT+HppJ7rxZh2jT3boYIzZzxqEF X0eT02LzDpCy6sbp8u4FsRj9mYcsPJo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=c6ZuccPL; spf=pass (imf03.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.196 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827341; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jeQJ/eIJXU0AYA9yk+s4aO7OKAgQk2qdJmGkoMOLMDk=; b=c6ZuccPLxLC+Ce4wWZ9aCvoGVbip0UFScbqDK/VCUHTzImAhP4V7Ea1dW8CIa6yy8rjLZJ 0Sp2j/SlVWnSus6gz0MlC0eiKozxqUtiaYSQ8bVhbIgA/srrcTqQNsoM1ecUX491WcPVhw WEyQ3zRkmTvotbuHgAt2bGqQFn9TnMI= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 4/5] mm: kmem: scoped objcg protection Date: Wed, 27 Sep 2023 08:08:31 -0700 Message-ID: <20230927150832.335132-5-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 5918720022 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ysc9oo5tgchq18e91ug87a3mpdi5tc4a X-HE-Tag: 1695827343-783950 X-HE-Meta: U2FsdGVkX19Q1nBm4PLw/D+sB6ocJzfxRiADUEC9Et5EHWdKm8Vrzcc+6lZqdXMQHagW5cYhz4E3OeHAWi0gkj4IKQD0ajb/qe5kgQi475MiEU1kTqMdNQLNE7I9Duqo+Bt3EveZsXNqiMmSwAigbxtSf37+yAk60/uK+pSTbo49jlZrXefwzvaeHuGb/mEV81fPZgOlWDlc6UZH/AKzQAux1CkJWfES17wKhSa82qnx1RvqELB62QUgwgoNSfYP2j3st4YG8K+V4w6voZqx3GtWephzeNJ9jcddSv8ZbPkK+OTyBrOk/XZ6XJElkxXcvYBQk+P6PUuI0CXjoeHC0HifC9BhPj3qdI8QhIxZzwmlA7v4k5oMaZU4+oq+WPzNa1COINcumAT0X+CMBOIEAzzQq7JfxaW1mjYCGETUpTSmd006cfU/qw5ylAsfLoagAOdmbWf85v/OTTkajExP/nWILq6SmpjYBLzGaSMZs0k2QhSvuQZojsYqDktjpM82E29O0oQtxoQ/CspCI9Shs7clMcYDDG6MDHBybPf9mLiVHktA9zHQRswj1zTLEBlRSx0lYIRM7keIGfTuUQ6lIGFiZwxu95Vp3FPn1am6/hu58wXZ0Ka8p9l3WRFZJgOlFO60s0Od0Ut72mfx2SI4KmqGMn9dyNrdvYk/Z9SD7NIJb+iSIH3vH+bFw99Dn5eFIxNPUP1ELGIDGj0JpyQuCAoKyvikqg7yMIlQFyoU8PxGlujlPt2xQFZQTPYNIxcrP/S0lWJvXyIqfkt1fSyXaZz5Lk8nRT4uq3I90MkAtcuGhwpQUJb7POb2KlOoAoZRudqZY3/btdbEQ9VFFVOE0jyH2p7ECTXwp4U43GZqBPpsGUfGN4L7Z6RdeLKUV5JYYlLHwr/aKAOTGgdf3CautGiZItl39YjP1W+U/1+SuVHR8BFoBOXFFKH6BolxQDg0p7TKpz3kqcQP5pJdeYu gAduND7Z ySCSpmLYCTV5fAYhBuMNvn+8OoAX8RkSU9nLgEB0UQIG9tuIF6ZKG1Nn9BvpZFwIDZZDvxqAucZAN4sI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Switch to a scope-based protection of the objcg pointer on slab/kmem allocation paths. Instead of using the get_() semantics in the pre-allocation hook and put the reference afterwards, let's rely on the fact that objcg is pinned by the scope. It's possible because: 1) if the objcg is received from the current task struct, the task is keeping a reference to the objcg. 2) if the objcg is received from an active memcg (remote charging), the memcg is pinned by the scope and has a reference to the corresponding objcg. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 6 +++++ mm/memcontrol.c | 46 ++++++++++++++++++++++++++++++++++++-- mm/slab.h | 10 +++------ 3 files changed, 53 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 412ff0e8694d..9a5212d3b9d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1779,6 +1779,12 @@ bool mem_cgroup_kmem_disabled(void); int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +/* + * The returned objcg pointer is safe to use without additional + * protection within a scope, refer to the implementation for the + * additional details. + */ +struct obj_cgroup *current_obj_cgroup(void); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4815f897758c..76557370f212 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3074,6 +3074,48 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return objcg; } +__always_inline struct obj_cgroup *current_obj_cgroup(void) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + + if (in_task()) { + memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; + + objcg = READ_ONCE(current->objcg); + if (unlikely(current_objcg_needs_update(objcg))) + objcg = current_objcg_update(objcg); + /* + * Objcg reference is kept by the task, so it's safe + * to use the objcg by the current task. + */ + return objcg; + } else { + memcg = this_cpu_read(int_active_memcg); + if (unlikely(memcg)) + goto from_memcg; + } + return NULL; + +from_memcg: + for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { + /* + * Memcg pointer is protected by scope (see set_active_memcg()) + * and is pinning the corresponding objcg, so objcg can't go + * away and can be used within the scope without any additional + * protection. + */ + objcg = rcu_dereference_check(memcg->objcg, 1); + if (likely(objcg)) + break; + objcg = NULL; + } + + return objcg; +} + struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) { struct obj_cgroup *objcg; @@ -3168,15 +3210,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) struct obj_cgroup *objcg; int ret = 0; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (objcg) { ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { + obj_cgroup_get(objcg); page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - obj_cgroup_put(objcg); } return ret; } diff --git a/mm/slab.h b/mm/slab.h index 799a315695c6..8cd3294fedf5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -484,7 +484,7 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, if (!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; @@ -497,17 +497,14 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, css_put(&memcg->css); if (ret) - goto out; + return false; } if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) - goto out; + return false; *objcgp = objcg; return true; -out: - obj_cgroup_put(objcg); - return false; } static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, @@ -542,7 +539,6 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, obj_cgroup_uncharge(objcg, obj_full_size(s)); } } - obj_cgroup_put(objcg); } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, From patchwork Wed Sep 27 15:08:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13400991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE5ABE82CA5 for ; Wed, 27 Sep 2023 15:09:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEE956B0170; Wed, 27 Sep 2023 11:09:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9E3B6B0171; Wed, 27 Sep 2023 11:09:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A23D6B0173; Wed, 27 Sep 2023 11:09:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 792766B0170 for ; Wed, 27 Sep 2023 11:09:07 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 589EA1CA295 for ; Wed, 27 Sep 2023 15:09:07 +0000 (UTC) X-FDA: 81282710334.30.DF7DDD2 Received: from out-193.mta1.migadu.com (out-193.mta1.migadu.com [95.215.58.193]) by imf20.hostedemail.com (Postfix) with ESMTP id 8E7C01C0037 for ; Wed, 27 Sep 2023 15:09:05 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="V//ihQR3"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf20.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.193 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; b=u3h/o7QMIzE34e2/fc8FZtY4cwSdIrXuMT4WUD6NJ44kEiGnteAXkDwJ89OB0UzdnhfkNV Ns9m3RWpSKKzfGjmIWUwEa8r6aWl+E+OY3FSsx6Mfj8NCUxWVPcB+OlirWIQjNfqLYkR76 XP3pFGHRRqyrMMmeXxJNs+CInDHOS3U= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="V//ihQR3"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf20.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.193 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827345; a=rsa-sha256; cv=none; b=n99dq8lyPdfUbuJR3Ooe5eRbT0JsK6V+e5HaY5pqc7Bh4qWOT3ibTtfax5OQWDHwxbM+D6 WBhbmpOrL26jR5CCtz/P1I2Bl3br6sGHv718CtvV/F7M7tqIh+1M5li6Ad32TECXs0j6qN SxN3jNAW2uL1T7hT4dR5ahLoJ0zHhLA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; b=V//ihQR3BNX8mCaIN2RHbsj24ybf5tsZR80ZurjIq/P4zdNE6/CNiqXGaUHELfnQzFCvFu ald/Rulhzhr/xKdfWTbsCyE4U6wbBWcnVys7yuOgrgVNmXQqA6dKbwvtqUzJLSfKLVSbkz VEv/GP4pB24vCqh16SVt3PIzQgO5+VI= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 5/5] percpu: scoped objcg protection Date: Wed, 27 Sep 2023 08:08:32 -0700 Message-ID: <20230927150832.335132-6-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8E7C01C0037 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: k7ncptu5tsh1mi98zkpuuwu36b1g4zzm X-HE-Tag: 1695827345-936086 X-HE-Meta: U2FsdGVkX1/YylXs6eJ5ZxZwozrCjlhcXuu/5k7R0jnFU59s+IwDQRuiXNUjfyO4FU1ylBxO9zy6GEKS+vknB2biaqfzk1UsulksaLCpXFLEaiI6F8HbX2vKK7Q4vgW2hw976AvyvUYR/bW11QX/1C4pKQK6QFURiWeZDEN+aPkCmbwCwXqbhQsiZ8D77yPlP8fx8Nv1nWFvm0ZSIT4DYwJChWl35lBRhnuZIJ40MYT2D/Q3ZF6edZuF6DKss0g3fCgD7KBrAygrXl17POaubGy6BnUgW24qQoVemFuckDp99sMRZ8Kifp0zM+Lk17etoIRUH5I0cyNijiVumsGOInIiyWsbAj6DKQ3HJxHs0rV+TVX3sw7GJQLTVloawKJuX7O4ITLYfWUj8m2rMHp6vi6SNEUih/jjr0Mrc2DeI9IzBN6LMqO23oNcxkTY9BK9gewGZSwntla8M7sNz28MJKKpul8c5CnQtODDuxTvfhjWpzACq22elgx7OtgIWGDw7sHyVY3h7akmMkwU8d6A11dgdt/cvdz3f9e1dClOx3WLfPaINNYDErLUmeQiFUU8EC9BaGXjsAJd4XdPzqEJheQ0dNHpyw/Agd6AVbNmx8ia+9S8jhJ4Xy+AujM0lVt1ggORlvyuBaFAJFnoZ8mRpSdJ3m6haPJSdjoSlzB2GCntmUhlwGvo+oQqNZQBI1+450yF/xYAmxCDIhJZ17VedNpSLZXMu6L9lbm675GuBl6HVYiqRcgUK33O2tuVbQdx9kCSQJGaQb14nhIbX9kVFl3ovgbq6ChufxFbwpFrNvqYH5D9ydMO1ho+rFSwEOFwicYXC0ql3p4S2IHMQ8JTmhHTkSODhSVETm2SamwCFVnArE+gsV0IIx6Fg1MegLhkMmGpjMDH856yv3lJOWtsxVdi87QvKoxiyS1A0I5CADZsgs1TjJB2XNU7CHY6I0Ygzv+u8DpShdXRe8QlnOk 8IEW0f5b 6kz+KSTYJkzyzFLZpkIFlmvyckBEmnYLC/Z+CfgF7qKFl42yqz87gg53ZMSWWNJlQ+7W3QRh5s39WycM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to slab and kmem, switch to a scope-based protection of the objcg pointer to avoid. Signed-off-by: Roman Gushchin (Cruise) --- mm/percpu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..f53ba692d67a 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1628,14 +1628,12 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, if (!memcg_kmem_online() || !(gfp & __GFP_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; - if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) { - obj_cgroup_put(objcg); + if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) return false; - } *objcgp = objcg; return true; @@ -1649,6 +1647,7 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, return; if (likely(chunk && chunk->obj_cgroups)) { + obj_cgroup_get(objcg); chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; rcu_read_lock(); @@ -1657,7 +1656,6 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, rcu_read_unlock(); } else { obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); - obj_cgroup_put(objcg); } }