From patchwork Tue Dec 20 18:27:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13078092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C4BC4167B for ; Tue, 20 Dec 2022 18:28:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFC8D8E0001; Tue, 20 Dec 2022 13:28:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B0FB28E0007; Tue, 20 Dec 2022 13:28:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 877FE8E0001; Tue, 20 Dec 2022 13:28:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 70A038E0003 for ; Tue, 20 Dec 2022 13:28:06 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 41EB7A072F for ; Tue, 20 Dec 2022 18:28:06 +0000 (UTC) X-FDA: 80263518972.12.38038FA Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by imf22.hostedemail.com (Postfix) with ESMTP id A0054C000E for ; Tue, 20 Dec 2022 18:28:04 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Wx7bQ5io; spf=pass (imf22.hostedemail.com: domain of roman.gushchin@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671560884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1c+efmi7i1PNMb6NmmQf0Ro/sE7SMscFhY/KBS3yOLk=; b=Y2xCfgYwJFGzCphUAHAuwbLZ/ebFP8fd0Cw9/3aDxG+BD6HK1VbUbuLx8TUi44+A2G+8uD vJ78BQZ332jMVBb8C+/1RIhuxQPZSbcodcSx7TVZHZBfPBAAOoPUVkIXXs7mJoHHt1DTSm rU7x7XtVXHCCppMbd5pOpN0Od+HTfD0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Wx7bQ5io; spf=pass (imf22.hostedemail.com: domain of roman.gushchin@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671560884; a=rsa-sha256; cv=none; b=6n4Nig5dKiV3ISM+OMHmepq+rgHx8tyAA/UZyGylwwSnzzBu/EQ1Yt6eG7IljvQdujh9uV cjA3KyW7kPtTbSttxw6rF75Y/3zbS9uTerQ1ttH16rnjTnw3jAEIzXNRlEqLGl63/Gvi7S zLWIlV3FsW9J1/8PFm+IGDM9FZ8VJiM= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1671560883; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1c+efmi7i1PNMb6NmmQf0Ro/sE7SMscFhY/KBS3yOLk=; b=Wx7bQ5ioJAD+g491R7nSJvJ8QTYMx+OPD/+3NQf2xDyjGmSAIeS5xvA6vaaypoUUowSvau T38EBM/MyoOs0D9shUHdyPMcOMxI88n+BDtB5+ZeMySUKoKja9i+0tgo8ESPYw+WmPupoG JdpUiHXhDx7tLpgzdmFRX6vTZ0dwRYM= From: Roman Gushchin To: linux-mm@kvack.org Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Shakeel Butt , Johannes Weiner , Michal Hocko , Muchun Song , Andrew Morton , Roman Gushchin Subject: [PATCH RFC 1/2] mm: kmem: optimize get_obj_cgroup_from_current() Date: Tue, 20 Dec 2022 10:27:44 -0800 Message-Id: <20221220182745.1903540-2-roman.gushchin@linux.dev> In-Reply-To: <20221220182745.1903540-1-roman.gushchin@linux.dev> References: <20221220182745.1903540-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A0054C000E X-Rspam-User: X-Stat-Signature: 6gti94iztdeeucqknuucyhbsmjs16b3o X-HE-Tag: 1671560884-756701 X-HE-Meta: U2FsdGVkX1/AVoxGUiKDuW8jF8bcQHX/F2OC36rIw+woIEPAEwGavZaABI3cmUN6JLy0L3+qZHwNDc1XOwBOyziBgrukrwXbKwjbfGVph12AziWXLzL7kz78QZSWiHpv5spv9Z8wf8P7kiFMUyxDpJegvmPQxLrXLrIYSisu5bs/WQlvBw3KcBZkgAG1hELGsqw5xlDeTlIWp0+yAna7LfFLViPqvPtpd+wK4cY9cpY+a+/FaIdgs6yxXgN8brzCFoOgOH0mjqW6jsVSeGJ7mFjt7mpo+6MqmljQCWcZYnw1dr9mOJJMRq0mrkFlHSxwBeljDcNE4c/mph2af79scb64cNrCztX3cbvr5KczcvqpCHXoRws8AdJ9zHQeHu/3EWASPmLR+1P3pluX0jMjYBcXF9FW6tmbg6/oec3S1IQlJHdVlalAMK2lOTIdkCBqkxYqbjh5LIoHUztztVGopW3+j6gcXIc1C/NkRNljl2UT3eEn8LTVdcasanmQUZSrjvY4QTNsMjDEu2FTQ62bbA6+z0bTBzBDMGrMsCdlpi0BrP0Zah9ksD2wBOIjEfGYJWlfPtRNxb8LLlfK2kWKLXgw6LYGf5nP64rM4XJiH9J+WUowxWSjbIvePKnYZmrr/KPh+HT3jKMoVitCrFlPHDAEqD/h7zByOYhpz+qnY88wMjFkLolhCeOaswcBk2Ko5J4Tt44vDQShlog3yPPzCc52NnyPsE5Yz2zYHB3jiIvLn0s8LSnEecaonVnWIWF20Ojx9xYe4LvDWvKWyLTxbUd5eycu9vmFZ7xEIHvdngeEFkMWUMfWplHmcelnvjp0/I7IM3HaezyY8Ki2m7ePVkoEmxuA40IOLYcj/v/OIMIqVku4rVMR05b8NRXK5QxhI/l/4Bx8oXQ08Pc7jM76ZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin Acked-by: Shakeel Butt --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index bafd3cde4507..82828c51d2ea 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1047,19 +1047,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3004,7 +2991,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3013,16 +3000,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock();