From patchwork Fri Sep 29 18:00:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13404691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46761E728D7 for ; Fri, 29 Sep 2023 18:01:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBD128D00F9; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C46CA8D002B; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABF1D8D00F9; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 95DF38D002B for ; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4242AC0ECA for ; Fri, 29 Sep 2023 18:01:27 +0000 (UTC) X-FDA: 81290402214.24.83D66CC Received: from out-199.mta0.migadu.com (out-199.mta0.migadu.com [91.218.175.199]) by imf12.hostedemail.com (Postfix) with ESMTP id 1542B40038 for ; Fri, 29 Sep 2023 18:01:24 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=iy2KaghV; spf=pass (imf12.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.199 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696010485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=KAYqpSv/3MAP6iirul2pq9NvXF2s3pPyl5tswfaYlMFbgen0jV4aOkUj9x2k45XdBjKHRC i/4cPbalTENfnmlCPW3+XcZMhSp5CoXkyQhqpt20MQARA7KFU0F+arfFE0NNZAz9wkoe0z 0h+DQTeH11QmCB04cdOdcn3iylkELKE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696010485; a=rsa-sha256; cv=none; b=bv7G4zrTFf0P45wVNbQrheNpQeq4P0dDxdsipyMsK60+6FDOzD2bVyuJZAwlKW1rdjxBbF ctB64rMGCP6GjYTC9MDv0KsISjdPd1nAOoMV+bpD75qfnwOnWQaLk2f8UoNIL7muDrbBAe ZDIpK/0FdZRltqvRNY9SBUAEvvLlHAQ= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=iy2KaghV; spf=pass (imf12.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.199 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=iy2KaghVwmRlwfpu6b7+AOhBWIOkmf5hD6kyAjLMK5XlcnnjDizfH24Jr/mHe5sgea67k7 4WiyJ8S09oEuEbj4kODKMMxIn9KtFUE1gjaYCUzSTGsq7ELsKoT6yLcMwvrxUsNOMZ4P39 APJaRZfIVaJdT9iKHDsVsLui/4J0bOY= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Fri, 29 Sep 2023 11:00:51 -0700 Message-ID: <20230929180056.1122002-2-roman.gushchin@linux.dev> In-Reply-To: <20230929180056.1122002-1-roman.gushchin@linux.dev> References: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 1542B40038 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8kbs4ot5xr7ed8y9u9ya9zifrsks7yfq X-HE-Tag: 1696010484-126644 X-HE-Meta: U2FsdGVkX19crwk5ZlDv/GD9n9HFbj7NOTjHFsBhKR6Fzkk9GL9fokAUYloUJLm41Z4ymSZ0nFN0WH0zwjMwozkqTWxyZaskDyUF7oHHmM9iy5sUNtbnNTvf9DLwjquQvy7NmjcYiApx9kzYXwYuxmKge4TTlwblbiPXwKeDuxCgnGkCfH2Z2BYcyqUrWaKlpZ+l+/Z8otxjeAcuChz2LgQD6R0HSe67rGo3pEx4hoHOSPVEQNfbFgdbTRmCR0/xs9VI48dQ//jWrS9eQP2YSuUW5pXk1D0NWEsBw9F4viUaeOufrfICTmhB8fU/IOz2Z27ViOdukiSvTYLc9sKX/Pw4zEqecIJSyWG3N4iwSPSVouiGbFBI1WwbdQeO2FftiWk+CmFVCCLTQR187ISBS+K+tbOY5fgPkBvlGWANVw+tIT6BQ6d2NSZyLnRok/24XCbTj8ROy4N5T1fC8/0GtDdVx5Nd9M4TN0YQuB/kc9ebL9LdVRyF0DJWZa2pZwonLEHNcwHcDgfcTECVrMfGD7frxpfxZclMU6N/OgYekUQse1nFGAEgov/1R8NFdVAdrJIdMgEhJnawg0JiwEy06WAZDvkwdPhqg0+x/JDLXW7eNDtfzimVN69oOPCsuhK4t4dSxl/Nf5GkvtrxS4VFboef9B4SvpcbdntT8vz+J+ZnqOlFN2xgTBCDbnh0Z8J54mCeNVLWggIeg7Vqr+lcK+EgFcxZspXYcufI9vYuA1+9GLNIYih5l7tdo2k4ljxO6Lh9V6jzvWwPZ5rA2pXdcvwPTIINiOi5eW45FtNiS13bszkoFwRF1KfnUTsSAK2HLz+xccQJWGqTynLyDxHZ5r3a7jATPuYbiOR8PJhGmJ0W3qKlbf+8Ly7q7p+lknSYptW0BtqgGwuUh7GVZnySe31UfXmwLHva14p9UCqTzB+5i8TO75STVkDzHu6W+4kJChGLJMCQkrfPUCs0Len H/A2IysN vfYABuOYdhYVwRgNFqYiaXpgoH8zpEA/OF141Zm+1DhpZnoZ4FADrxo0wf/V5VnVy9MpX8Lp2HyAwwRXeaVmFtp+ujnzZNOVwPCnbxoqrXkZfMuXZK3St6ttKki390TndbFnJeQ3+qOBICuPNxvqB8h1E6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); From patchwork Fri Sep 29 18:00:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13404690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4571E728D5 for ; Fri, 29 Sep 2023 18:01:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 239068D00F8; Fri, 29 Sep 2023 14:01:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BE988D002B; Fri, 29 Sep 2023 14:01:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 060478D00F8; Fri, 29 Sep 2023 14:01:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E54688D002B for ; Fri, 29 Sep 2023 14:01:25 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8DABB120FE5 for ; Fri, 29 Sep 2023 18:01:25 +0000 (UTC) X-FDA: 81290402130.28.09E5B43 Received: from out-198.mta0.migadu.com (out-198.mta0.migadu.com [91.218.175.198]) by imf25.hostedemail.com (Postfix) with ESMTP id A9E52A001A for ; Fri, 29 Sep 2023 18:01:23 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=hLwGG4pD; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.198 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696010483; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FeS4nT8XiugNKSp/kxx+xlW9pJcImHp1zwYguvQ6bzU=; b=JzEO/1jNoTgUu7ZvGRK+qrIM7yMiAUUqAEZZpjy4NxF+uIOMVlW62LX5xhUN6lTlRCcVfR dduK6Ekxzd8k8EbSTPmtsYnomYq9lO0DeXeiSHoafRXoOV1HxqKivkLp4v3Wn3hy2PPysN Hwxl3TSj/78G1oLmGQRFqr5PYZWJlhs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=hLwGG4pD; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.198 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696010483; a=rsa-sha256; cv=none; b=XHDMWZjDiH81DT6TJqOk18Y3uP2YeYQHQt68J0zpcdwQel3AGJIqt4eoO4bPXApykbmPNB +hzg7JXIRHuAnq8BMDs9LU50jlQ+Haw3o9etQ4EgU4QPIcPLLFMY9LRgSQxJqw85/mzSdd txyiA4yw7cJKrmPlZEq3NnGc4zLbftE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010480; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FeS4nT8XiugNKSp/kxx+xlW9pJcImHp1zwYguvQ6bzU=; b=hLwGG4pD4/FZU7k3OWWP3LrDWqtH2MNwyKtbA41xnyxv9s242QqSgCeir58a6G9PvefDHG y6w2Hm6M7iFiUfQ8kYoi1QiPxfiX3qlNW2SO0NAvibMHxwML/CrrIIFvkBW+mzzlsUxzRm +lkFSPuTtkySE8H6H/4jxh2aMPVfu3g= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 2/5] mm: kmem: add direct objcg pointer to task_struct Date: Fri, 29 Sep 2023 11:00:52 -0700 Message-ID: <20230929180056.1122002-3-roman.gushchin@linux.dev> In-Reply-To: <20230929180056.1122002-1-roman.gushchin@linux.dev> References: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Stat-Signature: qes37j5fdh5kk674bn5kqh1yi6o8mqcs X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A9E52A001A X-HE-Tag: 1696010483-622773 X-HE-Meta: U2FsdGVkX19KyxX2CY9YhTtFNP5Je6KyGfNPN1AiUWPhYlxuy48ViEZHerCDlLpj3TuZV5LJZTLdXYeBrE6Kb+UP9R+3x9mWB28ZKMTVGDjnbl4vGXxkq1XT9zjkFVnm5kN9/obqnLabU3Si3ZMpPE0jhNoT3+ogQ1zTYsBC/NxGvZF/jXl6EVIqwfztQUlyi/0Ta6qOai45pAFYgfIFYKvAb7rnVoTidqqRpilE/2yG4i0DB1nvTso+SVrt6kZ5+p3H09VLqf0WOv/BrPfOwnYCFVnauoWy/ei/ew10shpZmFh866zI5x8KV+3AR6DnYb5w/4VRi/S4vJ9+DbPKlzl2PCYunhrDfjDS0lzULAqS68NGFNnhxhO3tZUsxgO4wTVoEMPZYKtLA+ILguLtyRk25Kc7OEAlX+57BvO5m0n04spuBMIT0TXnnbALPqkYEoD3d/q5JQuxFpe2cjoBFKfkP7B+3E0noplthRZwWC/3W3g+UzHwYFhfWol+yL0CKCO7013GicZTwv6uUKOBWmGCoEfbDSq6i49kOf6NKJlap2/pRSjhwpuzzLehhpWzz/Vr2sGapDY1R7UWcqI6rGHwTD85PsJoMF0IfW6W5xqcFKGFWEeNb8+KWTgrA0CAJFW5kAYE6DHXDIaVvBGrv7ZKi18n2N5WxQPmN1encA0+oHJOIehAEut1fIzVu+6t+r51oPnWfU8IbIf8socdNI+b5lSmXsoUZEGtFs0O3Mem8+wt6YNwbYPW9gxv2ygfOGlTjipYVcMWE6G+rsDyT4zZj9RBJB+WY6TBSflJr9EsN6S46lkLihkRVX0pXQncqsdcrmvwfjGQsDSwqclEIEpEzQPErI2WLpj2nHiz06x//srC1C4NC3hfWjvRHRP39C5BJQAvbPXqnqZHK7s5Q1oRper28j6MVLfknG5EDUpjfCjWXd3pIRbZElFJM6/9ZlWcxDLj0LLyTeyvWEB wAxYO/gA kjtZxReW/5nFn5O4uYRamfhgU1bDhTjBnFtrJYqhu4ux6VeroLGIMaU1SUtEjwQ5pyM6Zmhrd7elXDpnNQyn/a6U1xcbymqXQBcDmep2u1L3XW1Woq4nn1/m+Bw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg(). Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder. To avoid any costly synchronization the following rules are applied: 1) A task sets it's objcg pointer itself. 2) If a task is being migrated to another cgroup, the least significant bit of the objcg pointer is set atomically. 3) On the allocation path the objcg pointer is obtained locklessly using the READ_ONCE() macro and the least significant bit is checked. If it's set, the following procedure is used to update it locklessly: - task->objcg is zeroed using cmpxcg - new objcg pointer is obtained - task->objcg is updated using try_cmpxchg - operation is repeated if try_cmpxcg fails It guarantees that no updates will be lost if task migration is racing against objcg pointer update. It also allows to keep both read and write paths fully lockless. Because the task is keeping a reference to the objcg, it can't go away while the task is alive. This commit doesn't change the way the remote memcg charging works. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 10 ++++ include/linux/sched.h | 4 ++ mm/memcontrol.c | 111 ++++++++++++++++++++++++++++++++++--- 3 files changed, 116 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..1c1ebb269ac1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -553,6 +553,16 @@ static inline bool folio_memcg_kmem(struct folio *folio) return folio->memcg_data & MEMCG_DATA_KMEM; } +static inline bool current_objcg_needs_update(struct obj_cgroup *objcg) +{ + return (struct obj_cgroup *)((unsigned long)objcg & 0x1); +} + +static inline struct obj_cgroup * +current_objcg_without_update_flag(struct obj_cgroup *objcg) +{ + return (struct obj_cgroup *)((unsigned long)objcg & ~0x1); +} #else static inline bool folio_memcg_kmem(struct folio *folio) diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac385f7..60de42715b56 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1443,6 +1443,10 @@ struct task_struct { struct mem_cgroup *active_memcg; #endif +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *objcg; +#endif + #ifdef CONFIG_BLK_CGROUP struct gendisk *throttle_disk; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 16ac2a5838fb..ec28f9cfc2f0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3001,6 +3001,47 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return objcg; } +static struct obj_cgroup *current_objcg_update(struct obj_cgroup *old) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg = NULL, *tmp = old; + + old = current_objcg_without_update_flag(old); + if (old) + obj_cgroup_put(old); + + rcu_read_lock(); + do { + /* Atomically drop the update bit, */ + WARN_ON_ONCE(cmpxchg(¤t->objcg, tmp, 0) != tmp); + + /* ...obtain the new objcg pointer */ + memcg = mem_cgroup_from_task(current); + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { + objcg = rcu_dereference(memcg->objcg); + if (objcg && obj_cgroup_tryget(objcg)) + break; + objcg = NULL; + } + + /* + * ...and try atomically set up a new objcg pointer. If it + * fails, it means the update flag was set concurrently, so + * the whole procedure should be repeated. + */ + tmp = 0; + } while (!try_cmpxchg(¤t->objcg, &tmp, objcg)); + rcu_read_unlock(); + + return objcg; +} + +static inline void current_objcg_set_needs_update(struct task_struct *task) +{ + /* atomically set the update bit */ + set_bit(0, (unsigned long *)¤t->objcg); +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct mem_cgroup *memcg; @@ -3008,19 +3049,26 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) if (in_task()) { memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; - /* Memcg to charge can't be determined. */ - if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) - return NULL; + objcg = READ_ONCE(current->objcg); + if (unlikely(current_objcg_needs_update(objcg))) + objcg = current_objcg_update(objcg); + + if (objcg) { + obj_cgroup_get(objcg); + return objcg; + } } else { memcg = this_cpu_read(int_active_memcg); - if (likely(!memcg)) - return NULL; + if (unlikely(memcg)) + goto from_memcg; } + return NULL; +from_memcg: rcu_read_lock(); - if (!memcg) - memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); return objcg; @@ -6345,6 +6393,7 @@ static void mem_cgroup_move_task(void) mem_cgroup_clear_mc(); } } + #else /* !CONFIG_MMU */ static int mem_cgroup_can_attach(struct cgroup_taskset *tset) { @@ -6358,8 +6407,27 @@ static void mem_cgroup_move_task(void) } #endif +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_fork(struct task_struct *task) +{ + /* + * Set the update flag to cause task->objcg to be initialized lazily + * on the first allocation. + */ + task->objcg = (struct obj_cgroup *)0x1; +} + +static void mem_cgroup_exit(struct task_struct *task) +{ + struct obj_cgroup *objcg = current_objcg_without_update_flag(task->objcg); + + if (objcg) + obj_cgroup_put(objcg); +} +#endif + #ifdef CONFIG_LRU_GEN -static void mem_cgroup_attach(struct cgroup_taskset *tset) +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) { struct task_struct *task; struct cgroup_subsys_state *css; @@ -6377,10 +6445,29 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) task_unlock(task); } #else +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_LRU_GEN */ + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) + current_objcg_set_needs_update(task); +} +#else +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_MEMCG_KMEM */ + +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) static void mem_cgroup_attach(struct cgroup_taskset *tset) { + mem_cgroup_lru_gen_attach(tset); + mem_cgroup_kmem_attach(tset); } -#endif /* CONFIG_LRU_GEN */ +#endif static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) { @@ -6824,9 +6911,15 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) .attach = mem_cgroup_attach, +#endif .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, +#ifdef CONFIG_MEMCG_KMEM + .fork = mem_cgroup_fork, + .exit = mem_cgroup_exit, +#endif .dfl_cftypes = memory_files, .legacy_cftypes = mem_cgroup_legacy_files, .early_init = 0, From patchwork Fri Sep 29 18:00:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13404692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7927CE728CF for ; Fri, 29 Sep 2023 18:01:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5903F8D00FA; Fri, 29 Sep 2023 14:01:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 540978D002B; Fri, 29 Sep 2023 14:01:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BAB18D00FA; Fri, 29 Sep 2023 14:01:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 291298D002B for ; Fri, 29 Sep 2023 14:01:28 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E9FFD4052F for ; Fri, 29 Sep 2023 18:01:27 +0000 (UTC) X-FDA: 81290402214.05.BDD49D4 Received: from out-203.mta0.migadu.com (out-203.mta0.migadu.com [91.218.175.203]) by imf26.hostedemail.com (Postfix) with ESMTP id 0838C14002B for ; Fri, 29 Sep 2023 18:01:25 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ImdhAu3A; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf26.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.203 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696010486; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QxcU5o8Zg0M6jb420o2zIxJRSsKU6LzkAHBX3GqSFms=; b=JVS5bABBE3aHtTu7dD8rHfVF0PitqjnEwVfVzvSWCMur77A2+MXkjUg0wC0BoUxiYrInWA vwXfJsxG36TRmkrV33bcnzAH8LLYDI2bmdAahHCCKhJ9ephKMDrPhlIXCkCExBnYdJ1dUH zPPJIv84gM6Q/GCc3RLUGDoINgc1WnE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ImdhAu3A; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf26.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.203 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696010486; a=rsa-sha256; cv=none; b=g2IAhmEk69mfx5dzlgqrbhi0NUa/l+oDGJo26BLAlUppRh+C3WsyBLvMbjG/gd2uwjPaIz DL0UOnK9InVN00Ktx67z0rVqcCRDAxW81Fx0oCj57xjI+2tvS26ifCY8p8Olr5HJCXMCEK gs7aEqBJQYMMWAeeSHgcyjUxpu4wpvk= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QxcU5o8Zg0M6jb420o2zIxJRSsKU6LzkAHBX3GqSFms=; b=ImdhAu3ALKSdHrgwpMCUhyBJ4W47XhCUnKHwtUs8l/RzPj6hc/NfPFnlbT6TZr/umtTkJh xhHHggDB8z5vJqFJ1wexSJW0oPmubHVLru/4OppYheK+YhIChrW1Atc7eG+tLhXmZpg8Xt CFlnH7b6YjSVLnA1ud3LguQ/xHfr6LM= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 3/5] mm: kmem: make memcg keep a reference to the original objcg Date: Fri, 29 Sep 2023 11:00:53 -0700 Message-ID: <20230929180056.1122002-4-roman.gushchin@linux.dev> In-Reply-To: <20230929180056.1122002-1-roman.gushchin@linux.dev> References: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 0838C14002B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qfu433n58sf9owzfn58m9uatduqdgedq X-HE-Tag: 1696010485-568737 X-HE-Meta: U2FsdGVkX19FyLhOWk9o2KpF7wjytTMCAoJemdJ5Sca1ICNY//WemcRo35C5obRHsm3kRQZODBZ73TohcENv/XWiiGcmJqcTJuFuJdA7X2ZubCG6DRl6K6+25ibXW4xOVgDRizXxvHc71Q2AyUapp9OmVnWzyvK4loP1/SyGh5OQLMChKyq2Eochg3xY3rEPMA+KbyW7lL1yt+TK3W1olpwD8mDbZmSun22JB/j7TF1a5bgTlQxGbRanw84iT6lUmRRv+9zB/fQy2jSu4XLj2YWmICDMs0C/3npciiSnzosCQTvdwFc7sQiB5FUdAegHZFDcnBoJNZ1A7WSwXlSobqknvhdFAwLtO2sbdkzmDend0HKMqYO01Hw6IgiEYEYoWCNrRd0TfvYLoX8/J/sBKrAvDCntw8J86Ht/RCYay1hvQupMmZBQWoXDiiWBIpOuVXCk0vn65rt524jg4MyDaze0aG7myT9/52jqF445EmtuFJhb4xu5jrc3fqTf7eeNqzz0oSWuhZCMG+i/9ROBe969MrFs5Wsk1n4MUNQN0XW8q/74Zqdxvwc5Xuwgg7jJc+OXjbeevjdr39X6dCUoscki7frzL3qbtpmLCK/kZqhzi0BUnIWabcpoLyiV6zAWRbQuz0WvIKDYyrB10DokmatjX8UZXFfYzKNjTsv3+r9GTzlZFsiuD5xx0i7bwkvdIjEW+uWm/W7+aOMm6XIXUAyJq7dn9Ys+tEV6L4GITHo3NBV0lGDaX6/77AGBAOawr8hEA/QLtJsbDcvscpqDkpufB/lG6+2yukFX9WSdDm4Yw7A29ec7VSLxQrtOUJiUSypoRvIMOxsvwZVpG4PqZAjMRQrs9NPs2Zc1vf1FHMJG1mta9az8AZSLAP5MkM/PULrp1XPhDPNzIkEpyiMY0MsDPClXpRZIvxW9BpiNXIRbhVrSb6nL/zXoKur0IruVkSDY86a2lsedsfJ8Zxg AQ5lsta2 wr642IhM0NHvCPX7YXQx+HIwNqR9DnBGfmv7tsdkON+xNvQS2bpY/Gr/sqmmyhYzMAOU37AMLdgx04OQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Keep a reference to the original objcg object for the entire life of a memcg structure. This allows to simplify the synchronization on the kernel memory allocation paths: pinning a (live) memcg will also pin the corresponding objcg. The memory overhead of this change is minimal because object cgroups usually outlive their corresponding memory cgroups even without this change, so it's only an additional pointer per memcg. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 8 +++++++- mm/memcontrol.c | 5 +++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1c1ebb269ac1..e59dea9d8666 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -299,7 +299,13 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - struct obj_cgroup __rcu *objcg; + /* + * memcg->objcg is wiped out as a part of the objcg repaprenting + * process. memcg->orig_objcg preserves a pointer (and a reference) + * to the original objcg until the end of live of memcg. + */ + struct obj_cgroup __rcu *objcg; + struct obj_cgroup *orig_objcg; /* list of inherited objcgs, protected by objcg_lock */ struct list_head objcg_list; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ec28f9cfc2f0..e9890f6e4da7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3803,6 +3803,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); + obj_cgroup_get(objcg); + memcg->orig_objcg = objcg; static_branch_enable(&memcg_kmem_online_key); @@ -5297,6 +5299,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) { int node; + if (memcg->orig_objcg) + obj_cgroup_put(memcg->orig_objcg); + for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); kfree(memcg->vmstats); From patchwork Fri Sep 29 18:00:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13404693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E615DE728D6 for ; Fri, 29 Sep 2023 18:01:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D2C18D00FB; Fri, 29 Sep 2023 14:01:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25BAE8D002B; Fri, 29 Sep 2023 14:01:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 030458D00FB; Fri, 29 Sep 2023 14:01:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E67138D002B for ; Fri, 29 Sep 2023 14:01:30 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B3FA9C0E9C for ; Fri, 29 Sep 2023 18:01:30 +0000 (UTC) X-FDA: 81290402340.17.8B72F94 Received: from out-201.mta0.migadu.com (out-201.mta0.migadu.com [91.218.175.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 89784120014 for ; Fri, 29 Sep 2023 18:01:28 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=reulyu8B; spf=pass (imf29.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.201 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696010488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O/HghY6af2rzSMfJ7xP4DH1sDr9DxTKbSixxOwxk1Wc=; b=QdyZfEDeWIIKQUH3noORAoJ0/1zPCx0AIhB3DCTK0VIPrIM4ZOf9yPVeU+J0snhYBiPV+1 YMzgUOlpJ7iQ5S11uRRVY/D5UMDa3ELGT17coxLwxJJbjZ3B9MsRat/MNjho1aCz1DLWJT cE8OLhtb6BPLXd6XiopACZ8Osl+vSmM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696010488; a=rsa-sha256; cv=none; b=VVwLcOXOQhrmRQZfkou6PziMCDZeIqMUvvYTmxhoFdy09FzB/XWNYPS/OUX/winsHN73Hv v8bY8QOUSvNRMJyhyB+tLTtQBn16670vfaeeCnVu2d6HFDyN1vMAVyNLfKTblvy1PmgKXV o6i1MgCwZtjUmFAy979l5gMZRb9OTAg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=reulyu8B; spf=pass (imf29.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.201 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O/HghY6af2rzSMfJ7xP4DH1sDr9DxTKbSixxOwxk1Wc=; b=reulyu8BPJ63WGorYfZOnZ+y3TQQyJvBgiYOWl05pCLFalHnzIPEPs/r+T2JEd8zTAmV+5 xq4tlzloY5TTOTNFcVw0n7ImC8bEtRN2wTux41sKvpuWQw+ZJwVV9YgZkFzFehUsdj59fc /cKiihLvc8ycwOyaCRkSf2htJ6JeWMI= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 4/5] mm: kmem: scoped objcg protection Date: Fri, 29 Sep 2023 11:00:54 -0700 Message-ID: <20230929180056.1122002-5-roman.gushchin@linux.dev> In-Reply-To: <20230929180056.1122002-1-roman.gushchin@linux.dev> References: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 89784120014 X-Rspam-User: X-Stat-Signature: hkdqstgfarzmccf69g7rg5edmzdwuxby X-Rspamd-Server: rspam03 X-HE-Tag: 1696010488-449391 X-HE-Meta: U2FsdGVkX1+aeVO93f2wyiM9jnj//9XU5g2KdHdA98dCQcR3K4JuDKnMOoWtLHus4HzdG1xb9nJrAkAB5oMLEzkaxfNvvTh0iTD5zQ7gMhRsfBML+PKoUnVIr4bSRPWnClpkgtPwcguxI2bYJN52Neab5+g4AnQdirht2jltt5Imz8/ePWltM4++AAuJBIloQ45uHWALmLF8/TZLXCII9ZesGibZ9APeV4TJeMkk94o61zqSMzfUjSnLln91REnGixhYmmPsUKlBBm+01KGAYmkxSXLm+K1FmitDEQrMPLTzWDYywmBBwvg0nrJcpwg7txsVphT2tengAJkkY32Hths00ajw5Jyeb9GoYqVaIJPkUj2lheAgIlS3yOFDATIx8OJgySWxbEs527z44WbVyRFor2vHroLN5KxhDPa1v5QsLoukg0TAWcOnqCgX9+ORTlc4EReA+EwNO4ZzRdBm4OCUeojubTSN7QGcBrzWH9EAM31zmTuezptvnKrYgQSuhLAdBT8h5EoAkFH8YWzRpy7BDvrSezz7bBEBNA8nVvgvdJUmkK3785jiKbecymsvZqitVJ7b1W+J2g6jQj5ebgk+pO8jCAJ1M1TG9QHWk4DoXM4t63b76vrVtzOy1UTeNnw7m/56i463peoacFw+ZCaXInueewOwEqBfWQRvJnT8zS4XI4DYFo1cEdBBgVsyRd311w6Tsi1LHyPqg8VHMvOrZdoUaJRngJ+EfHO3+lvCzy8G1JUNnxT9Kwa0kbEj0J+zqxUFC/dgXwGI/vEA/wbnczIjc6LlpM36kX+5Oj9x6TSwyLtdievfRzh5NHN9OXFl9FPHkpuJQsPEuDyg/cxkm4FL8na80DU9mm9SoZ0cOpO1eQ2EZPkbaLBrvsYs1fP6+8hl7ij4p4SPHdy2LpcZ6WeCfZ/ylCfHsmOa2lZwStrqPq1+rssmzDNYU5S1XL0MXE5xjV+tRZlG8mr FLKqcfkR qtcdRUYmWKWHOim7A+VSvA+iEYnrhDzPsqWGihfXt21Nkyl84CZ8M4tiovyvJFT/mZC3fe3xK1GtTFlBUbKPjO6KkSA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Switch to a scope-based protection of the objcg pointer on slab/kmem allocation paths. Instead of using the get_() semantics in the pre-allocation hook and put the reference afterwards, let's rely on the fact that objcg is pinned by the scope. It's possible because: 1) if the objcg is received from the current task struct, the task is keeping a reference to the objcg. 2) if the objcg is received from an active memcg (remote charging), the memcg is pinned by the scope and has a reference to the corresponding objcg. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 6 +++++ mm/memcontrol.c | 46 ++++++++++++++++++++++++++++++++++++-- mm/slab.h | 10 +++------ 3 files changed, 53 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e59dea9d8666..5a52327ab09a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1779,6 +1779,12 @@ bool mem_cgroup_kmem_disabled(void); int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +/* + * The returned objcg pointer is safe to use without additional + * protection within a scope, refer to the implementation for the + * additional details. + */ +struct obj_cgroup *current_obj_cgroup(void); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e9890f6e4da7..78ab36b5899f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3074,6 +3074,48 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return objcg; } +__always_inline struct obj_cgroup *current_obj_cgroup(void) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + + if (in_task()) { + memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; + + objcg = READ_ONCE(current->objcg); + if (unlikely(current_objcg_needs_update(objcg))) + objcg = current_objcg_update(objcg); + /* + * Objcg reference is kept by the task, so it's safe + * to use the objcg by the current task. + */ + return objcg; + } else { + memcg = this_cpu_read(int_active_memcg); + if (unlikely(memcg)) + goto from_memcg; + } + return NULL; + +from_memcg: + for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { + /* + * Memcg pointer is protected by scope (see set_active_memcg()) + * and is pinning the corresponding objcg, so objcg can't go + * away and can be used within the scope without any additional + * protection. + */ + objcg = rcu_dereference_check(memcg->objcg, 1); + if (likely(objcg)) + break; + objcg = NULL; + } + + return objcg; +} + struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) { struct obj_cgroup *objcg; @@ -3168,15 +3210,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) struct obj_cgroup *objcg; int ret = 0; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (objcg) { ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { + obj_cgroup_get(objcg); page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - obj_cgroup_put(objcg); } return ret; } diff --git a/mm/slab.h b/mm/slab.h index 799a315695c6..8cd3294fedf5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -484,7 +484,7 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, if (!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; @@ -497,17 +497,14 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, css_put(&memcg->css); if (ret) - goto out; + return false; } if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) - goto out; + return false; *objcgp = objcg; return true; -out: - obj_cgroup_put(objcg); - return false; } static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, @@ -542,7 +539,6 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, obj_cgroup_uncharge(objcg, obj_full_size(s)); } } - obj_cgroup_put(objcg); } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, From patchwork Fri Sep 29 18:00:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13404694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7973E728CF for ; Fri, 29 Sep 2023 18:01:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 377FF8D00FC; Fri, 29 Sep 2023 14:01:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F86E8D002B; Fri, 29 Sep 2023 14:01:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 197D18D00FC; Fri, 29 Sep 2023 14:01:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 04CD08D002B for ; Fri, 29 Sep 2023 14:01:35 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BA3DC1A0C6A for ; Fri, 29 Sep 2023 18:01:34 +0000 (UTC) X-FDA: 81290402508.19.372D015 Received: from out-209.mta0.migadu.com (out-209.mta0.migadu.com [91.218.175.209]) by imf07.hostedemail.com (Postfix) with ESMTP id 0A2D94002D for ; Fri, 29 Sep 2023 18:01:30 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=odkLZGFv; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf07.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.209 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696010491; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; b=bqAmBYeSm88vV592hWQme0xomIuCkHPF6mqJGonRv8FRCJZyZx9EGDEph2plCd5zJwAXvM 9dAJ2O1Yteimwml54Pp5CfOQ5xGSf6yNoz0ecUPiLGYVnK2D6N+T0vPEs05/Ns5OAmHqFC dx9XRqF4SDC2QzSLxGK/AV8+oUVcEBI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=odkLZGFv; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf07.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.209 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696010491; a=rsa-sha256; cv=none; b=QNjo9QZPQ0keJkdZAOCcfJmNA46cdmm7lSUIQn8WkjO+uyjx8ufaE9P6Ey+Ofd7KwFR8bu IbhpfnKzXxZ6fjXx5tJithuL0ZiQ7Bvj8HWnk1ftWDGtZIZh8QMrCR6ICAOvXQRZKzlm4P WMWglvet2pUR/EpwYT3+QK3pX3xwOyY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010488; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; b=odkLZGFvcc5TTuJvilLKehHXRk/B6mY4ABoJ1wkhqpX6hgrcQG2DXR/Xkyfy/HGnDasNt6 5JRdpKxIA1lbfyIu/AkdURXUJc8ZL7aiQApQLkXhhdoWIzk+PAht+vVeY3dVj+R1otYNom anh+3HHhyLLzhbFfFWBmw6+6M0/vXGg= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 5/5] percpu: scoped objcg protection Date: Fri, 29 Sep 2023 11:00:55 -0700 Message-ID: <20230929180056.1122002-6-roman.gushchin@linux.dev> In-Reply-To: <20230929180056.1122002-1-roman.gushchin@linux.dev> References: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 0A2D94002D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: xk1htk1sesu1cwtdj4p7kj9q3bbn3jhb X-HE-Tag: 1696010490-817409 X-HE-Meta: U2FsdGVkX18TFCmK0c1FOOPl1D7Ln+YzH+W1b3fL2vG5c3Diivkm8EsxYG0onJn3rl4BmClBqlSe/HTa+Mi4WL4WPtjKQ1GFww8ElYm7iAy/zpJ684/LecCBnKN2uUZm41OT1TYPswM5osIxblPIPfcH5wrloJCjWFic4O2wzCh9Kj7w/GXjaY86OUX6k9pOa2iwbyV8Hy1Pot8r3yrJ0Gfa2ZWkDYpiktAQqehDzPHqNPUHAL7O5KkWvUaClXHV0wKQrgrkpWcnWRiVIO0MwNjm448saQTiXNStJuLJpZTo1yByJMq23BqClzjDVL0MbmSKFjfASsVKk9ZEIupRZNWwmnoFrlGwAojpSss5Wvch2HU8LZqs+Sx7iU9J9BT3MR8RNgrA8/xnFbzKY6pZxBYRsodr2AxetDQ5axRgJKnfB+ZAZJbn+GpMbGxLmurvsCtqCLsOjFceYYYRUtxBr+7Xc79/mg9MS/1cNGaKV394hwzmmGvxj2PSOgkuDVUc9t5vx/qrHTAW0+Ye1wM2M9trxLCXMQHqk3BQOq76ptHPzblR/t+UVg501as94pWj/b4X3EJZxgjTDJlK2mB3iHjEpylDV/WoNRZTopsW24lSKMCJlZAgw7Go9+c0NMoVrhuWk1bN9WNA4G78tlk72IA6ovgmR56oRdT/mR3od5sGYjbZX+sbEMRWN1f9zt7BYBW7tO9qravhQrNd5IYZUlp+T1MOZJlb+xUICGNLEpcSJzVlq3Hp/+PDpsUzAIzeFFQE7WdxtSPM0thujJdabPfA0NzhIEK7ttnGkAIKREA9/W8duxhjwXdpEN4EPriDHCW0wVgN+ZjsuP5oynCU560hcq58LJN0zdbu1qYI/DDsQ9NjeirqGwlncD/rcN22oNnwGyeBwQ0U0nyi0XdPfPtDgOFtsPKkXtwX5BkueZq7X9vy5YTm9MGx1IzADc6DUWLQqWdR57MaQsQLUIm 0+58AiRO uOeuZiN9Tf4f0Apt3Y4J0ZVodIIV68KauPK4Gm1jZVybhuuqGJaczlaCEDbwr/JERDrJk++kM0UiQh5o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to slab and kmem, switch to a scope-based protection of the objcg pointer to avoid. Signed-off-by: Roman Gushchin (Cruise) --- mm/percpu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..f53ba692d67a 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1628,14 +1628,12 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, if (!memcg_kmem_online() || !(gfp & __GFP_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; - if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) { - obj_cgroup_put(objcg); + if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) return false; - } *objcgp = objcg; return true; @@ -1649,6 +1647,7 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, return; if (likely(chunk && chunk->obj_cgroups)) { + obj_cgroup_get(objcg); chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; rcu_read_lock(); @@ -1657,7 +1656,6 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, rcu_read_unlock(); } else { obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); - obj_cgroup_put(objcg); } }