From patchwork Tue Dec 20 18:27:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13078092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C4BC4167B for ; Tue, 20 Dec 2022 18:28:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFC8D8E0001; Tue, 20 Dec 2022 13:28:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B0FB28E0007; Tue, 20 Dec 2022 13:28:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 877FE8E0001; Tue, 20 Dec 2022 13:28:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 70A038E0003 for ; Tue, 20 Dec 2022 13:28:06 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 41EB7A072F for ; Tue, 20 Dec 2022 18:28:06 +0000 (UTC) X-FDA: 80263518972.12.38038FA Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by imf22.hostedemail.com (Postfix) with ESMTP id A0054C000E for ; Tue, 20 Dec 2022 18:28:04 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Wx7bQ5io; spf=pass (imf22.hostedemail.com: domain of roman.gushchin@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671560884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1c+efmi7i1PNMb6NmmQf0Ro/sE7SMscFhY/KBS3yOLk=; b=Y2xCfgYwJFGzCphUAHAuwbLZ/ebFP8fd0Cw9/3aDxG+BD6HK1VbUbuLx8TUi44+A2G+8uD vJ78BQZ332jMVBb8C+/1RIhuxQPZSbcodcSx7TVZHZBfPBAAOoPUVkIXXs7mJoHHt1DTSm rU7x7XtVXHCCppMbd5pOpN0Od+HTfD0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Wx7bQ5io; spf=pass (imf22.hostedemail.com: domain of roman.gushchin@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671560884; a=rsa-sha256; cv=none; b=6n4Nig5dKiV3ISM+OMHmepq+rgHx8tyAA/UZyGylwwSnzzBu/EQ1Yt6eG7IljvQdujh9uV cjA3KyW7kPtTbSttxw6rF75Y/3zbS9uTerQ1ttH16rnjTnw3jAEIzXNRlEqLGl63/Gvi7S zLWIlV3FsW9J1/8PFm+IGDM9FZ8VJiM= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1671560883; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1c+efmi7i1PNMb6NmmQf0Ro/sE7SMscFhY/KBS3yOLk=; b=Wx7bQ5ioJAD+g491R7nSJvJ8QTYMx+OPD/+3NQf2xDyjGmSAIeS5xvA6vaaypoUUowSvau T38EBM/MyoOs0D9shUHdyPMcOMxI88n+BDtB5+ZeMySUKoKja9i+0tgo8ESPYw+WmPupoG JdpUiHXhDx7tLpgzdmFRX6vTZ0dwRYM= From: Roman Gushchin To: linux-mm@kvack.org Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt , Johannes Weiner , Michal Hocko , Muchun Song , Andrew Morton , Roman Gushchin Subject: [PATCH RFC 1/2] mm: kmem: optimize get_obj_cgroup_from_current() Date: Tue, 20 Dec 2022 10:27:44 -0800 Message-Id: <20221220182745.1903540-2-roman.gushchin@linux.dev> In-Reply-To: <20221220182745.1903540-1-roman.gushchin@linux.dev> References: <20221220182745.1903540-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A0054C000E X-Rspam-User: X-Stat-Signature: 6gti94iztdeeucqknuucyhbsmjs16b3o X-HE-Tag: 1671560884-756701 X-HE-Meta: U2FsdGVkX1/AVoxGUiKDuW8jF8bcQHX/F2OC36rIw+woIEPAEwGavZaABI3cmUN6JLy0L3+qZHwNDc1XOwBOyziBgrukrwXbKwjbfGVph12AziWXLzL7kz78QZSWiHpv5spv9Z8wf8P7kiFMUyxDpJegvmPQxLrXLrIYSisu5bs/WQlvBw3KcBZkgAG1hELGsqw5xlDeTlIWp0+yAna7LfFLViPqvPtpd+wK4cY9cpY+a+/FaIdgs6yxXgN8brzCFoOgOH0mjqW6jsVSeGJ7mFjt7mpo+6MqmljQCWcZYnw1dr9mOJJMRq0mrkFlHSxwBeljDcNE4c/mph2af79scb64cNrCztX3cbvr5KczcvqpCHXoRws8AdJ9zHQeHu/3EWASPmLR+1P3pluX0jMjYBcXF9FW6tmbg6/oec3S1IQlJHdVlalAMK2lOTIdkCBqkxYqbjh5LIoHUztztVGopW3+j6gcXIc1C/NkRNljl2UT3eEn8LTVdcasanmQUZSrjvY4QTNsMjDEu2FTQ62bbA6+z0bTBzBDMGrMsCdlpi0BrP0Zah9ksD2wBOIjEfGYJWlfPtRNxb8LLlfK2kWKLXgw6LYGf5nP64rM4XJiH9J+WUowxWSjbIvePKnYZmrr/KPh+HT3jKMoVitCrFlPHDAEqD/h7zByOYhpz+qnY88wMjFkLolhCeOaswcBk2Ko5J4Tt44vDQShlog3yPPzCc52NnyPsE5Yz2zYHB3jiIvLn0s8LSnEecaonVnWIWF20Ojx9xYe4LvDWvKWyLTxbUd5eycu9vmFZ7xEIHvdngeEFkMWUMfWplHmcelnvjp0/I7IM3HaezyY8Ki2m7ePVkoEmxuA40IOLYcj/v/OIMIqVku4rVMR05b8NRXK5QxhI/l/4Bx8oXQ08Pc7jM76ZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin Acked-by: Shakeel Butt --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index bafd3cde4507..82828c51d2ea 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1047,19 +1047,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3004,7 +2991,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3013,16 +3000,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); From patchwork Tue Dec 20 18:27:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13078093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFEE6C3DA7A for ; Tue, 20 Dec 2022 18:28:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A1108E0007; Tue, 20 Dec 2022 13:28:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F7268E0003; Tue, 20 Dec 2022 13:28:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36FD88E0007; Tue, 20 Dec 2022 13:28:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 21CA48E0003 for ; Tue, 20 Dec 2022 13:28:09 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 01D96A042A for ; Tue, 20 Dec 2022 18:28:08 +0000 (UTC) X-FDA: 80263519098.22.BBA8D8A Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by imf07.hostedemail.com (Postfix) with ESMTP id 175B540020 for ; Tue, 20 Dec 2022 18:28:06 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="uj/7yvtK"; spf=pass (imf07.hostedemail.com: domain of roman.gushchin@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671560887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YesO5scRwOOin7uW7PZmPoS44XmivgS0ElHSpIPIVmE=; b=Hkav86StIEngLFcfPchrd/vs/GytpUsCP0HwAtGm9aUkWq0oc1tjICFYNmdG+zpKitTkjn ROojQ3QviNbFpFBwSUHmKzjXSyPs7eK/5R+4A5sEJlGksSWg4BiR3pLS4/xFucsVzh8lT0 pDQSFtz/LcVV3bLkW15Ppp3LFYKi/4c= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="uj/7yvtK"; spf=pass (imf07.hostedemail.com: domain of roman.gushchin@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671560887; a=rsa-sha256; cv=none; b=rmBgsJiHryTBo1wJ3IgpQYMBOQ8Yy6+Nei9+4NrGkmxojsl0V5yKxESyjFdBRfctu+0iEw +51T66HLblm5SA0x6gieVhaHfiayK28s4P817hCDCdEkhqBKYz28tk8RG+3VfToA+4gyGN 1+dEKr5D1CHHcdgxSaEIWx7z3eJxDZs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1671560885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YesO5scRwOOin7uW7PZmPoS44XmivgS0ElHSpIPIVmE=; b=uj/7yvtKjoxajolPjtroGBuMqcW9xd/ppyLXn1op6xg0jwxNhV/i7tWBgzXE7hMxU7ZOg2 5YnAteiyU3LPromj1Y2rl9NJzDsH3ZSZvNaCXoL5YyWtmpdEj5VNLkeKsdpOM0ZXg9BxQ1 nwtLRs6SLSSxrFSJHvJFeux6PLl778Y= From: Roman Gushchin To: linux-mm@kvack.org Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt , Johannes Weiner , Michal Hocko , Muchun Song , Andrew Morton , Roman Gushchin Subject: [PATCH RFC 2/2] mm: kmem: add direct objcg pointer to task_struct Date: Tue, 20 Dec 2022 10:27:45 -0800 Message-Id: <20221220182745.1903540-3-roman.gushchin@linux.dev> In-Reply-To: <20221220182745.1903540-1-roman.gushchin@linux.dev> References: <20221220182745.1903540-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 175B540020 X-Rspam-User: X-Stat-Signature: 4cgnoshp1ou4btzbg3m5nd8ctkcw7rku X-HE-Tag: 1671560886-238519 X-HE-Meta: U2FsdGVkX18HuypthgJWxl9oE1MczP7Mp2JxqHCEdU3my+5UVsT3ivyYAPcNpeKAtvTzP6GNriMsylLx9UNwCEUj3iga+y1cEyBrIcgkGTtNRi48ONZZJUFr/d4rEf+tf1Jt++B6ndPHQBlBkUSxj/fC5vd0KdazphaZv2btDmVUMV4MqOdILego5OdV0eCfv/WkrvY38muWIr6mCDsno2EZRxBoGTWVMluhg6H10miRX+wh2KD6XNQMCIPSAjWZyPRHloXcbArOmDWsPiIbIq51w8FsCSnJe9sCncSaoG1UFHzu8eQxcc0AFKfbTv+bypvCLF/t3II6qeyYJMaqY1cJlQf6H3wt14oYK/UwQOURK3Ifu/GclXNppRO668KHXf2Zy4r70JpCeYvI7GJDTtkI2Jhi+ponji1XLcgZSWMJxUTXo0gn+bZ7/VlasedUrC5lmij1ADHSvYH+cYUYraaq4t+MGYewYCQLSqQmdiImqPJefOev9vGrTKxrPCDX9FNBY5uiv8Ax41A7gPEZXd9oYi95mQSa+tn/kE15Rkh6JBZusxWo490gBlMdvUzWYYfpvAubvipeY1eMLwET+Ofr3d7RCNH0znOFhXrO+qliMA8DGsxkqQJrYxMBhCfOUrq/iTjV6lsPjx1Ugv4I6byfOwZ/+8iWcCU7sFP6W2LRxwXp8R/P3nX7dZT6Gc94XIFnAlAwdu9oEjXY+dhJnlynSKBUpsw0RWvPhY52z8z/A4ULXB0pMc5sw9nHXcE/qUoNdQQSKB1F7eFpM0YlLXDNmvvOx0OfGtVh979zpOvAz1f+9u0XO4VUr961RVfHmh2bcAUTbnwoHLHYgCY0wtyF2eebMytrSLmYZdvlse5Eh+Kq6XM5oW0lNn7RzSbb+Fh2QurocYT5Lebe6TtC/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg(). Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder. The old indirect way is still used for remote memcg charging. Signed-off-by: Roman Gushchin --- include/linux/sched.h | 4 +++ mm/memcontrol.c | 84 +++++++++++++++++++++++++++++++++++++------ 2 files changed, 77 insertions(+), 11 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 853d08f7562b..e17be609cbcb 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1435,6 +1435,10 @@ struct task_struct { struct mem_cgroup *active_memcg; #endif +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *objcg; +#endif + #ifdef CONFIG_BLK_CGROUP struct request_queue *throttle_queue; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 82828c51d2ea..e0547b224f40 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3001,23 +3001,29 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct mem_cgroup *memcg; - struct obj_cgroup *objcg; + struct obj_cgroup *objcg = NULL; if (in_task()) { memcg = current->active_memcg; - - /* Memcg to charge can't be determined. */ - if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) - return NULL; + if (unlikely(memcg)) + goto from_memcg; + + if (current->objcg) { + rcu_read_lock(); + do { + objcg = READ_ONCE(current->objcg); + } while (objcg && !obj_cgroup_tryget(objcg)); + rcu_read_unlock(); + } } else { memcg = this_cpu_read(int_active_memcg); - if (likely(!memcg)) - return NULL; + if (unlikely(memcg)) + goto from_memcg; } + return objcg; +from_memcg: rcu_read_lock(); - if (!memcg) - memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); return objcg; @@ -6303,6 +6309,28 @@ static void mem_cgroup_move_task(void) mem_cgroup_clear_mc(); } } + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_fork(struct task_struct *task) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); + memcg = mem_cgroup_from_task(task); + if (!memcg || mem_cgroup_is_root(memcg)) + task->objcg = NULL; + else + task->objcg = __get_obj_cgroup_from_memcg(memcg); + rcu_read_unlock(); +} + +static void mem_cgroup_exit(struct task_struct *task) +{ + if (task->objcg) + obj_cgroup_put(task->objcg); +} +#endif + #else /* !CONFIG_MMU */ static int mem_cgroup_can_attach(struct cgroup_taskset *tset) { @@ -6317,7 +6345,7 @@ static void mem_cgroup_move_task(void) #endif #ifdef CONFIG_LRU_GEN -static void mem_cgroup_attach(struct cgroup_taskset *tset) +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) { struct task_struct *task; struct cgroup_subsys_state *css; @@ -6335,10 +6363,38 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) task_unlock(task); } #else +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_LRU_GEN */ + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) { + struct mem_cgroup *memcg; + + if (task->objcg) + obj_cgroup_put(task->objcg); + + rcu_read_lock(); + memcg = container_of(css, struct mem_cgroup, css); + task->objcg = __get_obj_cgroup_from_memcg(memcg); + rcu_read_unlock(); + } +} +#else +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_MEMCG_KMEM */ + +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEMCG_KMEM) static void mem_cgroup_attach(struct cgroup_taskset *tset) { + mem_cgroup_lru_gen_attach(tset); + mem_cgroup_kmem_attach(tset); } -#endif /* CONFIG_LRU_GEN */ +#endif static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) { @@ -6816,9 +6872,15 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEMCG_KMEM) .attach = mem_cgroup_attach, +#endif .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, +#ifdef CONFIG_MEMCG_KMEM + .fork = mem_cgroup_fork, + .exit = mem_cgroup_exit, +#endif .dfl_cftypes = memory_files, .legacy_cftypes = mem_cgroup_legacy_files, .early_init = 0,