From patchwork Thu Aug 27 17:52:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11741223 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8760913 for ; Thu, 27 Aug 2020 17:52:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B4305207CD for ; Thu, 27 Aug 2020 17:52:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="eeMlfq0O" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4305207CD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A26B6B0002; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 289398E0003; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0681D6B0006; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id DEE866B0003 for ; Thu, 27 Aug 2020 13:52:21 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A7344180AD811 for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) X-FDA: 77197092882.01.smell49_5f0bf782706f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 79A6B1004FA73 for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=550855a34e=guro@fb.com,,RULES_HIT:30012:30034:30054:30064,0,RBL:67.231.153.30:@fb.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04yrzw1bc7ouxaaghk55rk66eak6gypx43o1x7hard6yt66t4tdx6sf5g393u87.xkw56djkxi6tezh78hiqsnjjhw5ysskeet9rmgrgjpxzhcwyebiss3e7bbifush.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: smell49_5f0bf782706f X-Filterd-Recvd-Size: 5530 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Thu, 27 Aug 2020 17:52:20 +0000 (UTC) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07RHmgU5019297 for ; Thu, 27 Aug 2020 10:52:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=lBLp+M/M4q7MNxLrusnBwT4NnNHP3uVf51kHz/U8dPs=; b=eeMlfq0OG0057zNcGHGiR8Zj4dLqfdomvxEm5FiYzvkdMWMDF0dEimWZZDNU0KqhPeRC fEIZwsu3ngL0JDA/Iwf/jICb216OuzDZCDt/bXtjW1bjcOTClC1RuTpgOcdopMGXN3W6 5jR5M/LZ6l+LKbPBfvXiEhHm8kXXtqPU4AY= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 335up86mjs-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 27 Aug 2020 10:52:20 -0700 Received: from intmgw002.41.prn1.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 27 Aug 2020 10:52:19 -0700 Received: by devvm1096.prn0.facebook.com (Postfix, from userid 111017) id 2DB6D3922483; Thu, 27 Aug 2020 10:52:16 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm1096.prn0.facebook.com To: CC: Andrew Morton , =Shakeel Butt , Johannes Weiner , Michal Hocko , , , Roman Gushchin Smtp-Origin-Cluster: prn0c01 Subject: [PATCH RFC 1/4] mm: kmem: move memcg_kmem_bypass() calls to get_mem/obj_cgroup_from_current() Date: Thu, 27 Aug 2020 10:52:12 -0700 Message-ID: <20200827175215.319780-2-guro@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200827175215.319780-1-guro@fb.com> References: <20200827175215.319780-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-27_10:2020-08-27,2020-08-27 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 bulkscore=0 suspectscore=1 adultscore=0 mlxlogscore=999 spamscore=0 impostorscore=0 clxscore=1015 phishscore=0 priorityscore=1501 mlxscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008270134 X-FB-Internal: deliver X-Rspamd-Queue-Id: 79A6B1004FA73 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000109, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently memcg_kmem_bypass() is called before obtaining the current memory/obj cgroup using get_mem/obj_cgroup_from_current(). Moving memcg_kmem_bypass() into get_mem/obj_cgroup_from_current() reduces the number of call sites and allows further code simplifications. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 13 ++++++++----- mm/percpu.c | 3 +-- mm/slab.h | 3 --- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dc892a3c4b17..9c08d8d14bc0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1074,6 +1074,9 @@ EXPORT_SYMBOL(get_mem_cgroup_from_page); */ static __always_inline struct mem_cgroup *get_mem_cgroup_from_current(void) { + if (memcg_kmem_bypass()) + return NULL; + if (unlikely(current->active_memcg)) { struct mem_cgroup *memcg; @@ -2913,6 +2916,9 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + if (memcg_kmem_bypass()) + return NULL; + if (unlikely(!current->mm && !current->active_memcg)) return NULL; @@ -3039,19 +3045,16 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) struct mem_cgroup *memcg; int ret = 0; - if (memcg_kmem_bypass()) - return 0; - memcg = get_mem_cgroup_from_current(); - if (!mem_cgroup_is_root(memcg)) { + if (memcg && !mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge(memcg, gfp, 1 << order); if (!ret) { page->mem_cgroup = memcg; __SetPageKmemcg(page); return 0; } + css_put(&memcg->css); } - css_put(&memcg->css); return ret; } diff --git a/mm/percpu.c b/mm/percpu.c index f4709629e6de..9b07bd5bc45f 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1584,8 +1584,7 @@ static enum pcpu_chunk_type pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, { struct obj_cgroup *objcg; - if (!memcg_kmem_enabled() || !(gfp & __GFP_ACCOUNT) || - memcg_kmem_bypass()) + if (!memcg_kmem_enabled() || !(gfp & __GFP_ACCOUNT)) return PCPU_CHUNK_ROOT; objcg = get_obj_cgroup_from_current(); diff --git a/mm/slab.h b/mm/slab.h index 95e5cc1bb2a3..4a24e1702923 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -280,9 +280,6 @@ static inline struct obj_cgroup *memcg_slab_pre_alloc_hook(struct kmem_cache *s, { struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; - objcg = get_obj_cgroup_from_current(); if (!objcg) return NULL; From patchwork Thu Aug 27 17:52:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11741221 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86DCA913 for ; Thu, 27 Aug 2020 17:52:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4C91B22BEB for ; Thu, 27 Aug 2020 17:52:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="EAAmWl8C" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C91B22BEB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1C9386B0003; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 150F26B0002; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0D096B0007; Thu, 27 Aug 2020 13:52:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id D7D8B6B0002 for ; Thu, 27 Aug 2020 13:52:21 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 998CE4417 for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) X-FDA: 77197092882.02.sugar82_36047a02706f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 6B74810097AA0 for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=550855a34e=guro@fb.com,,RULES_HIT:4303:30054:30064,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04yrsrpy5kpfs1pay4wqc5u588fzjopw8zmsnytd3cdkb9ch1ykcw8hgxh9cks4.uo841ug7f6tqsqoy6ys1u7s6a4zcgxanoe4sb4rgad9od3x38ehwe4t4okqyx8t.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: sugar82_36047a02706f X-Filterd-Recvd-Size: 3702 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Thu, 27 Aug 2020 17:52:20 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07RHpw6K004687 for ; Thu, 27 Aug 2020 10:52:19 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=t0u1GdS/GsDLLXOwIOkkJ5krxovYtPBEXfJM9OWBdus=; b=EAAmWl8CFRNeBxb/vm8dscjZ6A+3HX3FxAB08KXrXDccC2vlxMAP7tc31Ww0D7D7przJ 0r7mpgscIpIeuCTVdJyTsGJ/zvL1HaFCSPPHPRUHdBiylUBuTc2ca2CctScdmrJWDpfb oyYMd26MmC2/S3xXrtgQEEZDo59HHauCK0s= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 335up9eq4w-20 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 27 Aug 2020 10:52:19 -0700 Received: from intmgw002.06.prn3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 27 Aug 2020 10:52:17 -0700 Received: by devvm1096.prn0.facebook.com (Postfix, from userid 111017) id 32FE23922485; Thu, 27 Aug 2020 10:52:16 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm1096.prn0.facebook.com To: CC: Andrew Morton , =Shakeel Butt , Johannes Weiner , Michal Hocko , , , Roman Gushchin Smtp-Origin-Cluster: prn0c01 Subject: [PATCH RFC 2/4] mm: kmem: remove redundant checks from get_obj_cgroup_from_current() Date: Thu, 27 Aug 2020 10:52:13 -0700 Message-ID: <20200827175215.319780-3-guro@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200827175215.319780-1-guro@fb.com> References: <20200827175215.319780-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-27_10:2020-08-27,2020-08-27 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 adultscore=0 clxscore=1015 suspectscore=1 bulkscore=0 malwarescore=0 impostorscore=0 mlxlogscore=614 lowpriorityscore=0 phishscore=0 mlxscore=0 spamscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008270134 X-FB-Internal: deliver X-Rspamd-Queue-Id: 6B74810097AA0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are checks for current->mm and current->active_memcg in get_obj_cgroup_from_current(), but these checks are redundant: memcg_kmem_bypass() called just above performs same checks. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9c08d8d14bc0..5d847257a639 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2919,9 +2919,6 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) if (memcg_kmem_bypass()) return NULL; - if (unlikely(!current->mm && !current->active_memcg)) - return NULL; - rcu_read_lock(); if (unlikely(current->active_memcg)) memcg = rcu_dereference(current->active_memcg); From patchwork Thu Aug 27 17:52:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11741225 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A0C8414F6 for ; Thu, 27 Aug 2020 17:52:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 699C3207CD for ; Thu, 27 Aug 2020 17:52:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="pmpCkPRE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 699C3207CD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 80BD96B0006; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 795B38E0003; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C6AC6B0007; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id 397468E0006 for ; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id F0139173084F for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) X-FDA: 77197092882.17.dress86_5f023a02706f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id BB450180D0181 for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=550855a34e=guro@fb.com,,RULES_HIT:30054:30064:30074,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04yf97mbrq65eez8iu48gng8uhgojypo9fphbmhryewhogqr8yxg6wm1jegost7.rpdwh39efoo9g1tcn6mc56szc1957khak6hs6rsffkg3y19ag6ahcwk1mk5p9mj.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: dress86_5f023a02706f X-Filterd-Recvd-Size: 7672 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07RHmdnh009176 for ; Thu, 27 Aug 2020 10:52:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=cTcroAjpBM+ohC44YY25rqXxYtNf61yTmh0X3sxMeu0=; b=pmpCkPREelgpOB5j+MekZgN61p0l5Yn8QNw1ItMZ18DEHuNHwNg17oid/Mkf6ij7ATui 4v/WWP89UWZKWGcYUVSCVMrGOyXLoll3ntb+RfFjNSGYns8VpM53VcWhlr5Ns8iIX4F9 ZnTiJX0lTcFs3DSyqnckYOOUi8fIy3tDltc= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 335up7ppnd-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 27 Aug 2020 10:52:20 -0700 Received: from intmgw002.41.prn1.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 27 Aug 2020 10:52:19 -0700 Received: by devvm1096.prn0.facebook.com (Postfix, from userid 111017) id 38DDD3922487; Thu, 27 Aug 2020 10:52:16 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm1096.prn0.facebook.com To: CC: Andrew Morton , =Shakeel Butt , Johannes Weiner , Michal Hocko , , , Roman Gushchin Smtp-Origin-Cluster: prn0c01 Subject: [PATCH RFC 3/4] mm: kmem: prepare remote memcg charging infra for interrupt contexts Date: Thu, 27 Aug 2020 10:52:14 -0700 Message-ID: <20200827175215.319780-4-guro@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200827175215.319780-1-guro@fb.com> References: <20200827175215.319780-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-27_10:2020-08-27,2020-08-27 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 bulkscore=0 phishscore=0 mlxscore=0 clxscore=1015 adultscore=0 suspectscore=1 malwarescore=0 mlxlogscore=500 lowpriorityscore=0 spamscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008270134 X-FB-Internal: deliver X-Rspamd-Queue-Id: BB450180D0181 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remote memcg charging API uses current->active_memcg to store the currently active memory cgroup, which overwrites the memory cgroup of the current process. It works well for normal contexts, but doesn't work for interrupt contexts: indeed, if an interrupt occurs during the execution of a section with an active memcg set, all allocations inside the interrupt will be charged to the active memcg set (given that we'll enable accounting for allocations from an interrupt context). But because the interrupt might have no relation to the active memcg set outside, it's obviously wrong from the accounting prospective. To resolve this problem, let's add a global percpu int_active_memcg variable, which will be used to store an active memory cgroup which will be sued from interrupt contexts. set_active_memcg() will transparently use current->active_memcg or int_active_memcg depending on the context. To make the read part simple and transparent for the caller, let's introduce two new functions: - struct mem_cgroup *active_memcg(void), - struct mem_cgroup *get_active_memcg(void). They are returning the active memcg if it's set, hiding all implementation details: where to get it depending on the current context. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/sched/mm.h | 13 +++++++++-- mm/memcontrol.c | 48 ++++++++++++++++++++++++++++------------ 2 files changed, 45 insertions(+), 16 deletions(-) diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 4c69a4349ac1..030a1cf77b8a 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -304,6 +304,7 @@ static inline void memalloc_nocma_restore(unsigned int flags) #endif #ifdef CONFIG_MEMCG +DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); /** * set_active_memcg - Starts the remote memcg charging scope. * @memcg: memcg to charge. @@ -318,8 +319,16 @@ static inline void memalloc_nocma_restore(unsigned int flags) static inline struct mem_cgroup * set_active_memcg(struct mem_cgroup *memcg) { - struct mem_cgroup *old = current->active_memcg; - current->active_memcg = memcg; + struct mem_cgroup *old; + + if (in_interrupt()) { + old = this_cpu_read(int_active_memcg); + this_cpu_write(int_active_memcg, memcg); + } else { + old = current->active_memcg; + current->active_memcg = memcg; + } + return old; } #else diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5d847257a639..a51a6066079e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -73,6 +73,9 @@ EXPORT_SYMBOL(memory_cgrp_subsys); struct mem_cgroup *root_mem_cgroup __read_mostly; +/* Active memory cgroup to use from an interrupt context */ +DEFINE_PER_CPU(struct mem_cgroup *, int_active_memcg); + /* Socket memory accounting disabled? */ static bool cgroup_memory_nosocket; @@ -1069,26 +1072,43 @@ struct mem_cgroup *get_mem_cgroup_from_page(struct page *page) } EXPORT_SYMBOL(get_mem_cgroup_from_page); -/** - * If current->active_memcg is non-NULL, do not fallback to current->mm->memcg. - */ -static __always_inline struct mem_cgroup *get_mem_cgroup_from_current(void) +static __always_inline struct mem_cgroup *active_memcg(void) { - if (memcg_kmem_bypass()) - return NULL; + if (in_interrupt()) + return this_cpu_read(int_active_memcg); + else + return current->active_memcg; +} - if (unlikely(current->active_memcg)) { - struct mem_cgroup *memcg; +static __always_inline struct mem_cgroup *get_active_memcg(void) +{ + struct mem_cgroup *memcg; - rcu_read_lock(); + rcu_read_lock(); + memcg = active_memcg(); + if (memcg) { /* current->active_memcg must hold a ref. */ - if (WARN_ON_ONCE(!css_tryget(¤t->active_memcg->css))) + if (WARN_ON_ONCE(!css_tryget(&memcg->css))) memcg = root_mem_cgroup; else memcg = current->active_memcg; - rcu_read_unlock(); - return memcg; } + rcu_read_unlock(); + + return memcg; +} + +/** + * If active memcg is set, do not fallback to current->mm->memcg. + */ +static __always_inline struct mem_cgroup *get_mem_cgroup_from_current(void) +{ + if (memcg_kmem_bypass()) + return NULL; + + if (unlikely(active_memcg())) + return get_active_memcg(); + return get_mem_cgroup_from_mm(current->mm); } @@ -2920,8 +2940,8 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return NULL; rcu_read_lock(); - if (unlikely(current->active_memcg)) - memcg = rcu_dereference(current->active_memcg); + if (unlikely(active_memcg())) + memcg = active_memcg(); else memcg = mem_cgroup_from_task(current); From patchwork Thu Aug 27 17:52:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11741227 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0CE5F14F6 for ; Thu, 27 Aug 2020 17:52:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CA319207CD for ; Thu, 27 Aug 2020 17:52:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="foGAK5Uq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA319207CD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 17F838E0006; Thu, 27 Aug 2020 13:52:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 041548E0003; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD6548E0006; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0003.hostedemail.com [216.40.44.3]) by kanga.kvack.org (Postfix) with ESMTP id C4DF88E0003 for ; Thu, 27 Aug 2020 13:52:22 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8525D4417 for ; Thu, 27 Aug 2020 17:52:22 +0000 (UTC) X-FDA: 77197092924.11.hair16_0406ce22706f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 46394180F8B82 for ; Thu, 27 Aug 2020 17:52:22 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=550855a34e=guro@fb.com,,RULES_HIT:30036:30054:30064,0,RBL:67.231.153.30:@fb.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100;04y8hhyqjj7ciur4nb5crqc9ft5fkypzjxxa9oj1fbbfwhet1p3qbakxw6ozq6q.zfax8fetn6fboyhi9xdb76azh459g5euqxme1hzyx81itwjzqpyykkt8ymtnxnk.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: hair16_0406ce22706f X-Filterd-Recvd-Size: 4968 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Thu, 27 Aug 2020 17:52:21 +0000 (UTC) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07RHpFFY009582 for ; Thu, 27 Aug 2020 10:52:21 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=GiSvtA9Y81I+jnIfUZtXBD50esug8S34EMBARaI7nmM=; b=foGAK5UqBbB+ZAGapE39yBPIE8Qge42QGP+OTLWgW6wstNydlIEUeOwHk3WUuLyx5qp6 Ircwsuy0TsTmB9v4GllxJEF+KpBYR2TGUBMEPLo3Rkgvp0nVtp4mkfcIId5xwK1jcsm2 2GU+ep1AEqxtK+aum6M0RjIdH6k/4x3agHA= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 335up66k17-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 27 Aug 2020 10:52:21 -0700 Received: from intmgw002.41.prn1.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 27 Aug 2020 10:52:19 -0700 Received: by devvm1096.prn0.facebook.com (Postfix, from userid 111017) id 3E5FD3922489; Thu, 27 Aug 2020 10:52:16 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm1096.prn0.facebook.com To: CC: Andrew Morton , =Shakeel Butt , Johannes Weiner , Michal Hocko , , , Roman Gushchin Smtp-Origin-Cluster: prn0c01 Subject: [PATCH RFC 4/4] mm: kmem: enable kernel memcg accounting from interrupt contexts Date: Thu, 27 Aug 2020 10:52:15 -0700 Message-ID: <20200827175215.319780-5-guro@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200827175215.319780-1-guro@fb.com> References: <20200827175215.319780-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-27_10:2020-08-27,2020-08-27 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 adultscore=0 impostorscore=0 spamscore=0 bulkscore=0 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=1 phishscore=0 mlxlogscore=798 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008270134 X-FB-Internal: deliver X-Rspamd-Queue-Id: 46394180F8B82 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a memcg to charge can be determined (using remote charging API), there are no reasons to exclude allocations made from an interrupt context from the accounting. Such allocations will pass even if the resulting memcg size will exceed the hard limit, but it will affect the application of the memory pressure and an inability to put the workload under the limit will eventually trigger the OOM. To use active_memcg() helper, memcg_kmem_bypass() is moved back to memcontrol.c. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 12 ------------ mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d0b036123c6a..924177502479 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1528,18 +1528,6 @@ static inline bool memcg_kmem_enabled(void) return static_branch_likely(&memcg_kmem_enabled_key); } -static inline bool memcg_kmem_bypass(void) -{ - if (in_interrupt()) - return true; - - /* Allow remote memcg charging in kthread contexts. */ - if ((!current->mm || (current->flags & PF_KTHREAD)) && - !current->active_memcg) - return true; - return false; -} - static inline int memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a51a6066079e..75cd1a1e66c8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1098,6 +1098,19 @@ static __always_inline struct mem_cgroup *get_active_memcg(void) return memcg; } +static __always_inline bool memcg_kmem_bypass(void) +{ + /* Allow remote memcg charging from any context. */ + if (unlikely(active_memcg())) + return false; + + /* Memcg to charge can't be determined. */ + if (in_interrupt() || !current->mm || (current->flags & PF_KTHREAD)) + return true; + + return false; +} + /** * If active memcg is set, do not fallback to current->mm->memcg. */