From patchwork Wed Sep 27 15:08:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13400987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 551FAE82CA6 for ; Wed, 27 Sep 2023 15:08:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C742E6B0163; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C24D16B0165; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC4DB6B0167; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9D5F76B0163 for ; Wed, 27 Sep 2023 11:08:58 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 781421C9588 for ; Wed, 27 Sep 2023 15:08:58 +0000 (UTC) X-FDA: 81282709956.04.490408E Received: from out-195.mta1.migadu.com (out-195.mta1.migadu.com [95.215.58.195]) by imf27.hostedemail.com (Postfix) with ESMTP id 8179640031 for ; Wed, 27 Sep 2023 15:08:56 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Q8hMpLXl; spf=pass (imf27.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.195 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695827336; a=rsa-sha256; cv=none; b=xA+M06NeQtA+M6JjBKZ7nKvFclrwP5snO2fk3zkiw+sFDilDpRPBhNhya6mjPpRph4bwXT MGDD2b0DAnzCq3jgDnbndBbF2Sm1UCeNfevnpUKCb7M1+pMV0koVQjPXJQTLf6jazI6iQT v2OJDX/4yoiw1IufY84YRTKwle1btxo= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Q8hMpLXl; spf=pass (imf27.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.195 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695827336; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=csA9lqKqSWug/CUi1CRNnTt3apPTiWEZn90EWKQHAwp5lsNX3kA6sAsuW4beNhxB8n+D2Q 86aCLuf+Qzd14Y4sssscRbNrv9w9PCy/ePF2PVGsZ7QX7dTC4638IyeH+MbmtPWDZ6Xs6I mxK1umW4NW2ka77j7scL9lyP9w7bbas= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=Q8hMpLXlwyHYuhYTY7QmN3ucJ4BymSYDpw0OZAW82OI1HjO9QhUqV7hIgxRE3/+F1Zdf1X 9pwvp7A1SXynG+K4IEOnW40uYr4B8pTu6BDJ26Y+WSYnLIfaJXQFLXbhSCwQWYyEHZxai0 mR7jQfl6X8WWs0zHRyt6d4SZPcg5cVY= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Wed, 27 Sep 2023 08:08:28 -0700 Message-ID: <20230927150832.335132-2-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8179640031 X-Stat-Signature: 8y9wo3gh1qkgoma89j76fsgwbco88fko X-Rspam-User: X-HE-Tag: 1695827336-418902 X-HE-Meta: U2FsdGVkX18TW1YQE9NePlw2rra9WjPYA6k8QQOti65JcK1pvh2lyvotJ2hT4MKFOIEDmi4LXSm8vCdmDyfIQlvFzJRdtekzpFjAmCnIz5IfLdFnJvRylHlOiNNBOHeIIaSY0V0DJjTxbiplEr8PuwR84NcIFp/rHt6QRv9u/b3kU6FdXj7oQ3VsQPDffL6XR6RYlEmnTh0ClUr5tT/tbq1iIzhXeq+RAj4rcB7U/Hr8uPT0+NjAnMLX0GgGW1aU2ot3cbMGBtJlaKtJoWHyWghZ92VQcslio9BP0cq4U/5Yd7RzQhJdcR6rHjTzeZFQ0NY6f49yv7HdvxSK1egLNzVavm6apTJj0ldjcC7DnJlieUCMhPWOQPaCZDEvNiP5dgK/ZmaPL5zLAWcHPte48Onyr6Qy0pwSLzTNbVju0KGHcawEZyixmkIrH/XhWrx7QJP35eLbswW5wRhR/DdQPVT1/zsfTFEEcqL5SPR/cCqTdbDTOvkIL8Sqze2L9GcMIbF09CGG/B2sEQQc+twF8767yc/IvI6+42ro8msUJa1uJ4mwCRhD1cho/RHLZehNsj5UuRpcnDXScngtGk6U159M/8mXKBLFKv9OG19A2/4O6BMWfztLVSKZtC4dbYnJAjbQWlNt0uVZ3X32UeuDhYyptSUI1G3gsqQbc7dlPhgH0K4qRkPFNqUEyifBxHzReuLiPoIwBhV76iFKDTxllXJiKNDB85/pDAF9c5I/jg6Zw6y/XY4vvqDosRHTjBTTWHFPEmci77dvIU/ZBoLEKOFZMwSQQd2GbymcWPuS8L5bHGGpzmkKpXa7E0YJugFRAVioPcIlJ5FGM9h5+GhcLjmKO52tSJXsQ8A9yGlI4fj00Iony9V9aixBOcAVM3UsjyS1tBHOrCs95ReN3ZnB4b/I6sb2KSNlh/2nkleqBmcvewnlaVFAsa6+W4uC3iQ6nfMlO3Xb+M7WqLy0cIq JkmtLOdL yi0KJJFm8x/Y4BVY4Uyt5a92Esfo7cY3KBtWJ5OGTQIVrmRgXM5moP5AfHyl3P1nTcagfpRBjwxlCl44EOM45ul0Vrji8UHINN6pgoQ8QLaJ09ZV7xJCmhAMF7kF7bZYxt7ErAnNMIR/eTA9gKKDm+j3KDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock();