From patchwork Tue Oct 10 00:09:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13414695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46850E95A9C for ; Tue, 10 Oct 2023 00:10:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFD238D009D; Mon, 9 Oct 2023 20:10:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CAC0A8D0089; Mon, 9 Oct 2023 20:10:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B27B58D009D; Mon, 9 Oct 2023 20:10:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A51B18D0089 for ; Mon, 9 Oct 2023 20:10:02 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7D660140356 for ; Tue, 10 Oct 2023 00:10:02 +0000 (UTC) X-FDA: 81327619044.04.693F27B Received: from out-206.mta0.migadu.com (out-206.mta0.migadu.com [91.218.175.206]) by imf18.hostedemail.com (Postfix) with ESMTP id CBE9A1C0004 for ; Tue, 10 Oct 2023 00:10:00 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ft97nbcy; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf18.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.206 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696896601; a=rsa-sha256; cv=none; b=Vu8wlI2d+qFGiz6926PbNUKAfkISexD6EDYfSWJUlo/h1zsD5Kx8iReT6KmpyRDREnNoWX qcsSYI6MNrlpDVi/H4G9X8bbfQEJlmKYtR+5AB5b4pVqsla1X03MEo/sHUcCVKKoIBnDuq mGrWTWrSnmVV7fazHb1wDSPdLasTh+0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ft97nbcy; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf18.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.206 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696896601; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6BDA26poRN0lVkuNk023R6ElLY4Pdz30rttSzJzWmU0=; b=SqvuqYWUnKaqTlWT3yjHxDrTkZhqQTblKEwy40mlUu9sPp7US/Ksu0qGEHsmVwM5F/yomH vPeT5JNLjpY4E+qF6x8RpvYRI7RK4NeV9ZJCSt2rsWxrsqm0XPoZj0A+vtcaOBJ8XgLXdf jzYedzWYRlCo10AlGnJV4fILKFo/+ZM= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6BDA26poRN0lVkuNk023R6ElLY4Pdz30rttSzJzWmU0=; b=ft97nbcyFPqeUqZ4lisCE8IdqHS0q8h0Wnsl4HIdSRB7hY45IRuQS+OuTRiaAGnlAil0tV 6n6m78HGgBIkfz0ByXmzPARF8sfyEGsDiqZLrbTdayMXeSR6UAx8qwH0gjpYcPSmPHKjtQ qzYOWVCL8FrVR9Z2F277gR71ZjBKiJU= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v2 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Mon, 9 Oct 2023 17:09:25 -0700 Message-ID: <20231010000929.450702-2-roman.gushchin@linux.dev> In-Reply-To: <20231010000929.450702-1-roman.gushchin@linux.dev> References: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CBE9A1C0004 X-Stat-Signature: 47kgt8wp94c8otrcd4et1qjtjjinuzoe X-HE-Tag: 1696896600-877202 X-HE-Meta: U2FsdGVkX18bcJYnSXL26nJoBAILfT9gXTCOLNLA/M5XbIB2xkJ9jbQsZ/diem1k7j2EKMqgjkG7EwgufWH0twIVAJD21eoPvLt2sYol0VbdYiXw5fTaHjQkWo4FoU1U77/imjILH1k6BWMXxCXjGLU3zZBGVO3jnaOap9agUEZ5k48WMFazotl9PQYtMjvwYxTwpPEvuJoGpO9hFvGInMA4k66Xb6Cwx807aQSaIUrlQq8R/FWPazilF1TOBThcI9RaRlHJ9tbrumdC3y+8VrPIr7RRn+LuTngnloAEyAGXn8ShDJJugTvRq8EcvJ3F9WkDG3ZTwHRhqb81DgWrq0iWi+D8bnFR0tPo/X7S+AmKyqRXU2Xp1fwukyfnQFPBLDpBCRhdWn3Wc431Q8osfgk32o7pioRc+vfTXHxXY45lkruuaYAXyohZeJ0Vxb2257dKVCIGr+N7zrFPM+9/zZchxyIZ11uJ97/CYcuLqck4QnxdBLPFEpLuYdpK5URpDp3vcStqvTTElwtJLy7L0OSJLog/imdkhNQRNYzVsWzjac6RFrHnraeRHtpiqY7xHubn999omrOiRJHoNkMPZhB2BT32dtRndyI5OH0Ye48rGceAorqNqqhav+7RLd6+FZ61RSBsPI0BcFEVfHCarbXLRDkOehdgVuNXp4bv1mdgcBtuHVpRfZrIgPRExCZuV/b96GKX8u9aZdlXemSSFOMXrjEKykIA3rhbIJQEKqLtIu2iibFDtN633nUxywB1Sg1mOrMlmekNfz7/UIagT2PthJ9+6NfAS8o00RhzwDpG9zYYetqt4hQsyQY5sAGDawRryg6FrGG6GWI3lqSnvLEUMSQMyXkAtFs7UMwYYSLVfREpPBM+o/qsOiCcNddjk//Q0dZe6HLJZBZTrFB7XAReVdtK6t+10wVU2v8PQIIrLXQOCiw5Zd832XNfzTV6+sZ33W+c2yLomLuE70b y420tM1u OgEbadXj5VguD1zjQKIoLbkPPgNV1lbbvHyXWQQb3E9ozARd8gooKy0B3rm6CkYzt9tdcF0wMGD4sbDJBxaOcNL2QZ1QCdGBeK6bonMWTMYrykskHk8taJgh6Hd2Lmy552wU+rW9VEN6hyMyL2q9Ay4I6K9JYeHHwErpummLqto2tvK1SCtqz1q5xMM/Y4rhqAFZ2sa5xOosNlRrpMrDxERKzrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock();