From patchwork Thu Sep 5 21:45:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11134105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66C391593 for ; Thu, 5 Sep 2019 22:24:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 420712070C for ; Thu, 5 Sep 2019 22:24:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="CNOo37C+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 420712070C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E51756B0003; Thu, 5 Sep 2019 18:24:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DDB126B0005; Thu, 5 Sep 2019 18:24:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA1866B0007; Thu, 5 Sep 2019 18:24:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id 9F7E56B0003 for ; Thu, 5 Sep 2019 18:24:36 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 0EF5A180AD7C3 for ; Thu, 5 Sep 2019 22:24:36 +0000 (UTC) X-FDA: 75902297352.25.shop72_2816ca3ee5f53 X-Spam-Summary: 2,0,0,8eac21262eeecfe9,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:1:2:41:355:379:472:541:800:960:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1605:1730:1747:1777:1792:1981:2194:2195:2197:2198:2199:2200:2201:2202:2393:2553:2559:2562:2693:2731:2739:2892:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4052:4250:4321:4605:5007:6119:6120:6261:6653:7875:7901:7903:8603:9036:9168:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:12986:13141:13161:13166:13229:13230:13869:14096:14097:14394:21080:21450:21451:21627:21740:30005:30054:30064:30070:30075:30089:30090,0,RBL:67.231.153.30:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custo m_rules: X-HE-Tag: shop72_2816ca3ee5f53 X-Filterd-Recvd-Size: 12313 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 22:24:35 +0000 (UTC) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85Lhr2v032677 for ; Thu, 5 Sep 2019 14:46:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=F6+Y/3KaVJP+sTReowVqRgiVYePOKXbKfRKu1OS1cQw=; b=CNOo37C+wR2FT9w55v9ic/fR2lcHK0+5ibSpi3gCiet14MHztSwdQmTCxECKpkawlFrz cNT48JxVG1xWiacjxiu3FP4NIufgJ4FrCOm9jbaZw1XwFfP/ds34no/uvndPhO90KziS cd7Ko/mbtQPu4baCb4DIyzvuRUP8vLq0Q84= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2utksg62eu-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:09 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:08 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id A2A7A17229DF8; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 01/14] mm: memcg: subpage charging API Date: Thu, 5 Sep 2019 14:45:45 -0700 Message-ID: <20190905214553.1643060-2-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 malwarescore=0 clxscore=1015 impostorscore=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=999 priorityscore=1501 suspectscore=1 phishscore=0 adultscore=0 bulkscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce an API to charge subpage objects to the memory cgroup. The API will be used by the new slab memory controller. Later it can also be used to implement percpu memory accounting. In both cases, a single page can be shared between multiple cgroups (and in percpu case a single allocation is split over multiple pages), so it's not possible to use page-based accounting. The implementation is based on percpu stocks. Memory cgroups are still charged in pages, and the residue is stored in perpcu stock, or on the memcg itself, when it's necessary to flush the stock. Please, note, that unlike the generic page charging API, a subpage object is not holding a reference to the memory cgroup. It's because a more complicated indirect scheme is required in order to implement cheap reparenting. The percpu stock holds a single reference to the cached cgroup though. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 4 ++ mm/memcontrol.c | 129 +++++++++++++++++++++++++++++++++---- 2 files changed, 119 insertions(+), 14 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0c762e8ca6a6..120d39066148 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -214,6 +214,7 @@ struct mem_cgroup { /* Accounted resources */ struct page_counter memory; struct page_counter swap; + atomic_t nr_stocked_bytes; /* Legacy consumer-oriented counters */ struct page_counter memsw; @@ -1370,6 +1371,9 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, unsigned int nr_pages); +int __memcg_kmem_charge_subpage(struct mem_cgroup *memcg, size_t size, + gfp_t gfp); +void __memcg_kmem_uncharge_subpage(struct mem_cgroup *memcg, size_t size); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1c4c08b45e44..effefcec47b3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2149,6 +2149,10 @@ EXPORT_SYMBOL(unlock_page_memcg); struct memcg_stock_pcp { struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; + + struct mem_cgroup *subpage_cached; + unsigned int nr_bytes; + struct work_struct work; unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 @@ -2189,6 +2193,29 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) return ret; } +static bool consume_subpage_stock(struct mem_cgroup *memcg, + unsigned int nr_bytes) +{ + struct memcg_stock_pcp *stock; + unsigned long flags; + bool ret = false; + + if (nr_bytes > (MEMCG_CHARGE_BATCH << PAGE_SHIFT)) + return ret; + + local_irq_save(flags); + + stock = this_cpu_ptr(&memcg_stock); + if (memcg == stock->subpage_cached && stock->nr_bytes >= nr_bytes) { + stock->nr_bytes -= nr_bytes; + ret = true; + } + + local_irq_restore(flags); + + return ret; +} + /* * Returns stocks cached in percpu and reset cached information. */ @@ -2206,6 +2233,27 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } +static void drain_subpage_stock(struct memcg_stock_pcp *stock) +{ + struct mem_cgroup *old = stock->subpage_cached; + + if (stock->nr_bytes) { + unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT; + unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1); + + page_counter_uncharge(&old->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&old->memsw, nr_pages); + + atomic_add(nr_bytes, &old->nr_stocked_bytes); + stock->nr_bytes = 0; + } + if (stock->subpage_cached) { + css_put(&old->css); + stock->subpage_cached = NULL; + } +} + static void drain_local_stock(struct work_struct *dummy) { struct memcg_stock_pcp *stock; @@ -2218,8 +2266,11 @@ static void drain_local_stock(struct work_struct *dummy) local_irq_save(flags); stock = this_cpu_ptr(&memcg_stock); - drain_stock(stock); - clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); + if (test_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { + drain_stock(stock); + drain_subpage_stock(stock); + clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); + } local_irq_restore(flags); } @@ -2248,6 +2299,29 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) local_irq_restore(flags); } +static void refill_subpage_stock(struct mem_cgroup *memcg, + unsigned int nr_bytes) +{ + struct memcg_stock_pcp *stock; + unsigned long flags; + + local_irq_save(flags); + + stock = this_cpu_ptr(&memcg_stock); + if (stock->subpage_cached != memcg) { /* reset if necessary */ + drain_subpage_stock(stock); + css_get(&memcg->css); + stock->subpage_cached = memcg; + stock->nr_bytes = atomic_xchg(&memcg->nr_stocked_bytes, 0); + } + stock->nr_bytes += nr_bytes; + + if (stock->nr_bytes > (MEMCG_CHARGE_BATCH << PAGE_SHIFT)) + drain_subpage_stock(stock); + + local_irq_restore(flags); +} + /* * Drains all per-CPU charge caches for given root_memcg resp. subtree * of the hierarchy under it. @@ -2276,6 +2350,9 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) if (memcg && stock->nr_pages && mem_cgroup_is_descendant(memcg, root_memcg)) flush = true; + memcg = stock->subpage_cached; + if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) + flush = true; rcu_read_unlock(); if (flush && @@ -2500,8 +2577,9 @@ void mem_cgroup_handle_over_high(void) } static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, - unsigned int nr_pages) + unsigned int amount, bool subpage) { + unsigned int nr_pages = subpage ? ((amount >> PAGE_SHIFT) + 1) : amount; unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages); int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; struct mem_cgroup *mem_over_limit; @@ -2514,7 +2592,9 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, if (mem_cgroup_is_root(memcg)) return 0; retry: - if (consume_stock(memcg, nr_pages)) + if (subpage && consume_subpage_stock(memcg, amount)) + return 0; + else if (!subpage && consume_stock(memcg, nr_pages)) return 0; if (!do_memsw_account() || @@ -2632,14 +2712,22 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); - css_get_many(&memcg->css, nr_pages); + + if (subpage) + refill_subpage_stock(memcg, (nr_pages << PAGE_SHIFT) - amount); + else + css_get_many(&memcg->css, nr_pages); return 0; done_restock: - css_get_many(&memcg->css, batch); - if (batch > nr_pages) - refill_stock(memcg, batch - nr_pages); + if (subpage && (batch << PAGE_SHIFT) > amount) { + refill_subpage_stock(memcg, (batch << PAGE_SHIFT) - amount); + } else if (!subpage) { + css_get_many(&memcg->css, batch); + if (batch > nr_pages) + refill_stock(memcg, batch - nr_pages); + } /* * If the hierarchy is above the normal consumption range, schedule @@ -2942,7 +3030,7 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct page_counter *counter; int ret; - ret = try_charge(memcg, gfp, nr_pages); + ret = try_charge(memcg, gfp, nr_pages, false); if (ret) return ret; @@ -3020,6 +3108,18 @@ void __memcg_kmem_uncharge(struct page *page, int order) css_put_many(&memcg->css, nr_pages); } + +int __memcg_kmem_charge_subpage(struct mem_cgroup *memcg, size_t size, + gfp_t gfp) +{ + return try_charge(memcg, gfp, size, true); +} + +void __memcg_kmem_uncharge_subpage(struct mem_cgroup *memcg, size_t size) +{ + refill_subpage_stock(memcg, size); +} + #endif /* CONFIG_MEMCG_KMEM */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -5267,7 +5367,8 @@ static int mem_cgroup_do_precharge(unsigned long count) int ret; /* Try a single bulk charge without reclaim first, kswapd may wake */ - ret = try_charge(mc.to, GFP_KERNEL & ~__GFP_DIRECT_RECLAIM, count); + ret = try_charge(mc.to, GFP_KERNEL & ~__GFP_DIRECT_RECLAIM, count, + false); if (!ret) { mc.precharge += count; return ret; @@ -5275,7 +5376,7 @@ static int mem_cgroup_do_precharge(unsigned long count) /* Try charges one by one with reclaim, but do not retry */ while (count--) { - ret = try_charge(mc.to, GFP_KERNEL | __GFP_NORETRY, 1); + ret = try_charge(mc.to, GFP_KERNEL | __GFP_NORETRY, 1, false); if (ret) return ret; mc.precharge++; @@ -6487,7 +6588,7 @@ int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm, if (!memcg) memcg = get_mem_cgroup_from_mm(mm); - ret = try_charge(memcg, gfp_mask, nr_pages); + ret = try_charge(memcg, gfp_mask, nr_pages, false); css_put(&memcg->css); out: @@ -6866,10 +6967,10 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) mod_memcg_state(memcg, MEMCG_SOCK, nr_pages); - if (try_charge(memcg, gfp_mask, nr_pages) == 0) + if (try_charge(memcg, gfp_mask, nr_pages, false) == 0) return true; - try_charge(memcg, gfp_mask|__GFP_NOFAIL, nr_pages); + try_charge(memcg, gfp_mask|__GFP_NOFAIL, nr_pages, false); return false; } From patchwork Thu Sep 5 21:45:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133969 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6B0991599 for ; Thu, 5 Sep 2019 21:46:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3311F20828 for ; Thu, 5 Sep 2019 21:46:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="b0xo5sei" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3311F20828 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 03CE36B0003; Thu, 5 Sep 2019 17:46:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F309A6B0005; Thu, 5 Sep 2019 17:46:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF78A6B0007; Thu, 5 Sep 2019 17:46:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0075.hostedemail.com [216.40.44.75]) by kanga.kvack.org (Postfix) with ESMTP id BD15C6B0003 for ; Thu, 5 Sep 2019 17:46:10 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 5A775180AD7C3 for ; Thu, 5 Sep 2019 21:46:10 +0000 (UTC) X-FDA: 75902200500.08.pipe14_8d1e2a1eb390b X-Spam-Summary: 2,0,0,83edf7d759c261d0,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:1:2:41:355:379:541:800:960:966:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2693:2731:2741:2918:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4050:4321:4385:4605:5007:6119:6120:6261:6653:7901:7903:8784:9010:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:12986:13141:13161:13229:13230:14096:14097:14394:21080:21433:21450:21451:21627:30034:30054:30064:30080,0,RBL:67.231.153.30:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: pipe14_8d1e2a1eb390b X-Filterd-Recvd-Size: 10809 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:09 +0000 (UTC) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.16.0.42/8.16.0.42) with SMTP id x85LgVK3015053 for ; Thu, 5 Sep 2019 14:46:09 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=jUYA9V6/76PQuUuUoRBz8Z4hZkBQOhxwB/mAKjBuibo=; b=b0xo5seiQRBzPeolJLt20bUGceN0SA+D2w/06RSqqA+HMXoaYd6WRwRbLZYWDneA149q VPBE0N6l+ymMTHo1prbnguhUIGr1kM2zd/X5hGkOuv81ZHRzn5vbm1n0TlqELpIW/tRh PZaFZPhk3kh5CbM+NrCTLNRCVc5vYyEvHxo= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net with ESMTP id 2utxy63a75-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:08 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:07 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id A65A717229DFA; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 02/14] mm: memcg: introduce mem_cgroup_ptr Date: Thu, 5 Sep 2019 14:45:46 -0700 Message-ID: <20190905214553.1643060-3-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 mlxscore=0 clxscore=1015 priorityscore=1501 spamscore=0 lowpriorityscore=0 adultscore=0 malwarescore=0 suspectscore=4 bulkscore=0 phishscore=0 mlxlogscore=999 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050202 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit introduces mem_cgroup_ptr structure and corresponding API. It implements a pointer to a memory cgroup with a built-in reference counter. The main goal of it is to implement reparenting efficiently. If a number of objects (e.g. slab pages) have to keep a pointer and a reference to a memory cgroup, they can use mem_cgroup_ptr instead. On reparenting, only one mem_cgroup_ptr->memcg pointer has to be changed, instead of walking over all accounted objects. mem_cgroup_ptr holds a single reference to the corresponding memory cgroup. Because it's initialized before the css reference counter, css's refcounter can't be bumped at allocation time. Instead, it's bumped on reparenting which happens during offlining. A cgroup is never released online, so it's fine. mem_cgroup_ptr is released using rcu, so memcg->kmem_memcg_ptr can be accessed in a rcu read section. On reparenting it's atomically switched to NULL. If the reader gets NULL, it can just read parent's kmem_memcg_ptr instead. Each memory cgroup contains a list of kmem_memcg_ptrs. On reparenting the list is spliced into the parent's list. The list is protected using the css set lock. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 50 ++++++++++++++++++++++ mm/memcontrol.c | 87 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 133 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 120d39066148..dd5ebfe5a86c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -23,6 +23,7 @@ #include struct mem_cgroup; +struct mem_cgroup_ptr; struct page; struct mm_struct; struct kmem_cache; @@ -197,6 +198,22 @@ struct memcg_cgwb_frn { int memcg_id; /* memcg->css.id of foreign inode */ u64 at; /* jiffies_64 at the time of dirtying */ struct wb_completion done; /* tracks in-flight foreign writebacks */ +} + +/* + * A pointer to a memory cgroup with a built-in reference counter. + * For a use as an intermediate object to simplify reparenting of + * objects charged to the cgroup. The memcg pointer can be switched + * to the parent cgroup without a need to modify all objects + * which hold the reference to the cgroup. + */ +struct mem_cgroup_ptr { + struct percpu_ref refcnt; + struct mem_cgroup *memcg; + union { + struct list_head list; + struct rcu_head rcu; + }; }; /* @@ -312,6 +329,8 @@ struct mem_cgroup { int kmemcg_id; enum memcg_kmem_state kmem_state; struct list_head kmem_caches; + struct mem_cgroup_ptr __rcu *kmem_memcg_ptr; + struct list_head kmem_memcg_ptr_list; #endif int last_scanned_node; @@ -440,6 +459,21 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; } +static inline bool mem_cgroup_ptr_tryget(struct mem_cgroup_ptr *ptr) +{ + return percpu_ref_tryget(&ptr->refcnt); +} + +static inline void mem_cgroup_ptr_get(struct mem_cgroup_ptr *ptr) +{ + percpu_ref_get(&ptr->refcnt); +} + +static inline void mem_cgroup_ptr_put(struct mem_cgroup_ptr *ptr) +{ + percpu_ref_put(&ptr->refcnt); +} + static inline void mem_cgroup_put(struct mem_cgroup *memcg) { if (memcg) @@ -1433,6 +1467,22 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) return memcg ? memcg->kmemcg_id : -1; } +static inline struct mem_cgroup_ptr * +mem_cgroup_get_kmem_ptr(struct mem_cgroup *memcg) +{ + struct mem_cgroup_ptr *memcg_ptr; + + rcu_read_lock(); + do { + memcg_ptr = rcu_dereference(memcg->kmem_memcg_ptr); + if (memcg_ptr && mem_cgroup_ptr_tryget(memcg_ptr)) + break; + } while ((memcg = parent_mem_cgroup(memcg))); + rcu_read_unlock(); + + return memcg_ptr; +} + #else static inline int memcg_kmem_charge(struct page *page, gfp_t gfp, int order) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index effefcec47b3..cb9adb31360e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -266,6 +266,77 @@ struct cgroup_subsys_state *vmpressure_to_css(struct vmpressure *vmpr) } #ifdef CONFIG_MEMCG_KMEM +extern spinlock_t css_set_lock; + +static void memcg_ptr_release(struct percpu_ref *ref) +{ + struct mem_cgroup_ptr *ptr = container_of(ref, struct mem_cgroup_ptr, + refcnt); + unsigned long flags; + + spin_lock_irqsave(&css_set_lock, flags); + list_del(&ptr->list); + spin_unlock_irqrestore(&css_set_lock, flags); + + mem_cgroup_put(ptr->memcg); + percpu_ref_exit(ref); + kfree_rcu(ptr, rcu); +} + +static int memcg_init_kmem_memcg_ptr(struct mem_cgroup *memcg) +{ + struct mem_cgroup_ptr *kmem_memcg_ptr; + int ret; + + kmem_memcg_ptr = kmalloc(sizeof(struct mem_cgroup_ptr), GFP_KERNEL); + if (!kmem_memcg_ptr) + return -ENOMEM; + + ret = percpu_ref_init(&kmem_memcg_ptr->refcnt, memcg_ptr_release, + 0, GFP_KERNEL); + if (ret) { + kfree(kmem_memcg_ptr); + return ret; + } + + kmem_memcg_ptr->memcg = memcg; + INIT_LIST_HEAD(&kmem_memcg_ptr->list); + rcu_assign_pointer(memcg->kmem_memcg_ptr, kmem_memcg_ptr); + list_add(&kmem_memcg_ptr->list, &memcg->kmem_memcg_ptr_list); + return 0; +} + +static void memcg_reparent_kmem_memcg_ptr(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + unsigned int nr_reparented = 0; + struct mem_cgroup_ptr *memcg_ptr = NULL; + + rcu_swap_protected(memcg->kmem_memcg_ptr, memcg_ptr, true); + percpu_ref_kill(&memcg_ptr->refcnt); + + /* + * kmem_memcg_ptr is initialized before css refcounter, so until now + * it doesn't hold a reference to the memcg. Bump it here. + */ + css_get(&memcg->css); + + spin_lock_irq(&css_set_lock); + list_for_each_entry(memcg_ptr, &memcg->kmem_memcg_ptr_list, list) { + xchg(&memcg_ptr->memcg, parent); + nr_reparented++; + } + if (nr_reparented) + list_splice(&memcg->kmem_memcg_ptr_list, + &parent->kmem_memcg_ptr_list); + spin_unlock_irq(&css_set_lock); + + if (nr_reparented) { + css_get_many(&parent->css, nr_reparented); + css_put_many(&memcg->css, nr_reparented); + } +} + /* * This will be the memcg's index in each cache's ->memcg_params.memcg_caches. * The main reason for not using cgroup id for this: @@ -3554,7 +3625,7 @@ static void memcg_flush_percpu_vmevents(struct mem_cgroup *memcg) #ifdef CONFIG_MEMCG_KMEM static int memcg_online_kmem(struct mem_cgroup *memcg) { - int memcg_id; + int memcg_id, ret; if (cgroup_memory_nokmem) return 0; @@ -3566,6 +3637,12 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) if (memcg_id < 0) return memcg_id; + ret = memcg_init_kmem_memcg_ptr(memcg); + if (ret) { + memcg_free_cache_id(memcg_id); + return ret; + } + static_branch_inc(&memcg_kmem_enabled_key); /* * A memory cgroup is considered kmem-online as soon as it gets @@ -3601,12 +3678,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* - * Deactivate and reparent kmem_caches. Then flush percpu - * slab statistics to have precise values at the parent and - * all ancestor levels. It's required to keep slab stats + * Deactivate and reparent kmem_caches and reparent kmem_memcg_ptr. + * Then flush percpu slab statistics to have precise values at the + * parent and all ancestor levels. It's required to keep slab stats * accurate after the reparenting of kmem_caches. */ memcg_deactivate_kmem_caches(memcg, parent); + memcg_reparent_kmem_memcg_ptr(memcg, parent); memcg_flush_percpu_vmstats(memcg, true); kmemcg_id = memcg->kmemcg_id; @@ -5171,6 +5249,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->socket_pressure = jiffies; #ifdef CONFIG_MEMCG_KMEM memcg->kmemcg_id = -1; + INIT_LIST_HEAD(&memcg->kmem_memcg_ptr_list); #endif #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&memcg->cgwb_list); From patchwork Thu Sep 5 21:45:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11134017 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B75F01593 for ; Thu, 5 Sep 2019 22:18:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA65820825 for ; Thu, 5 Sep 2019 22:18:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="JGwogWPy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA65820825 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 68CC46B0003; Thu, 5 Sep 2019 18:18:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 63E256B0005; Thu, 5 Sep 2019 18:18:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 552876B0007; Thu, 5 Sep 2019 18:18:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0208.hostedemail.com [216.40.44.208]) by kanga.kvack.org (Postfix) with ESMTP id 2E9A16B0003 for ; Thu, 5 Sep 2019 18:18:57 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id C43C645A0 for ; Thu, 5 Sep 2019 22:18:56 +0000 (UTC) X-FDA: 75902283072.10.offer06_883e9df151d30 X-Spam-Summary: 2,0,0,012d83ae5bbb5d03,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:2693:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:4117:4321:4605:5007:6119:6120:6261:6653:7875:7901:7904:9036:10004:10400:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13255:14096:14097:14181:14394:14721:21080:21433:21627:21740:30054:30064:30070,0,RBL:67.231.153.30:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: offer06_883e9df151d30 X-Filterd-Recvd-Size: 6090 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 22:18:56 +0000 (UTC) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85Lhr2x032677 for ; Thu, 5 Sep 2019 14:46:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=0TGz2gJ0tZNu6OTGW7RsyezpkFK8N/gfzLdnuPDN3vA=; b=JGwogWPyFdChIpr5UQoPD8vIQmZBoI9ulUQlI5gsywzal9KVz5XhTkSqS6kzaPVPjRRW o8D3ol+fXetjeit793DtVfuF5hZyV1k855W1KCwqKXwC3kgJmevhWZ6U56uLjBs8Sbhy ygm182ka0KVvmsbqReBs2pDUGFxYIN2D0HA= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2utksg62eu-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:10 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:08 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id AA85717229DFC; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 03/14] mm: vmstat: use s32 for vm_node_stat_diff in struct per_cpu_nodestat Date: Thu, 5 Sep 2019 14:45:47 -0700 Message-ID: <20190905214553.1643060-4-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 malwarescore=0 clxscore=1015 impostorscore=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=999 priorityscore=1501 suspectscore=1 phishscore=0 adultscore=0 bulkscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently s8 type is used for per-cpu caching of per-node statistics. It works fine because the overfill threshold can't exceed 125. But if some counters are in bytes (and the next commit in the series will convert slab counters to bytes), it's not gonna work: value in bytes can easily exceed s8 without exceeding the threshold converted to bytes. So to avoid overfilling per-cpu caches and breaking vmstats correctness, let's use s32 instead. This doesn't affect per-zone statistics. There are no plans to use zone-level byte-sized counters, so no reasons to change anything. Signed-off-by: Roman Gushchin --- include/linux/mmzone.h | 2 +- mm/vmstat.c | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index bda20282746b..6e8233d52971 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -353,7 +353,7 @@ struct per_cpu_pageset { struct per_cpu_nodestat { s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; #endif /* !__GENERATING_BOUNDS.H */ diff --git a/mm/vmstat.c b/mm/vmstat.c index 6afc892a148a..98f43725d910 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -337,7 +337,7 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long x; long t; @@ -395,13 +395,13 @@ void __inc_zone_state(struct zone *zone, enum zone_stat_item item) void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -439,13 +439,13 @@ void __dec_zone_state(struct zone *zone, enum zone_stat_item item) void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); @@ -538,7 +538,7 @@ static inline void mod_node_state(struct pglist_data *pgdat, enum node_stat_item item, int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; do { From patchwork Thu Sep 5 21:45:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133971 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3EE2476 for ; Thu, 5 Sep 2019 21:46:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1192020825 for ; Thu, 5 Sep 2019 21:46:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="VOKv3gR+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1192020825 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 485CE6B0005; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 40E866B0007; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FB196B0008; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 00A496B0005 for ; Thu, 5 Sep 2019 17:46:12 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 86B5E180AD7C3 for ; Thu, 5 Sep 2019 21:46:12 +0000 (UTC) X-FDA: 75902200584.18.cord93_8d6ca519e6f1f X-Spam-Summary: 2,0,0,a98a58b1bfbeb43a,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:41:69:327:355:379:541:800:960:966:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2690:2693:2731:2895:2914:3138:3139:3140:3141:3142:3353:3369:3865:3866:3867:3868:3870:3871:3872:3874:4041:4250:4321:4385:4605:5007:6119:6120:6261:6653:7901:7904:8603:8660:8957:9036:10004:10954:11026:11232:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13148:13161:13229:13230:14096:14097:14394:14877:21080:21324:21325:21433:21450:21451:21611:21627:21740:30012:30054:30055:30056:30064:30070,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custo m_rules: X-HE-Tag: cord93_8d6ca519e6f1f X-Filterd-Recvd-Size: 21591 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:11 +0000 (UTC) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85LjfHJ025642 for ; Thu, 5 Sep 2019 14:46:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=KRUpM9B4qzOJCnYFrMw0m++EnlrcohwDNfLzf2h8uX0=; b=VOKv3gR+QImTqU3Qiq9sn3Eo03piZeOdDu5hKWx7ldkNKJ6nSjssClu535r+J4m+VXMa Lq6TunGBNgTPVah7NRR7VZuRpnYyef4y4fEDMfj82o0ZzgOoRSXsiJO1mRXTWcGB3/kE OC+V1yfMvq6UzRiNWCSK7xBPUkJkt/p7B9A= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2uu8mdrhe7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:10 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:09 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id AEECA17229DFE; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 04/14] mm: vmstat: convert slab vmstat counter to bytes Date: Thu, 5 Sep 2019 14:45:48 -0700 Message-ID: <20190905214553.1643060-5-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 adultscore=0 spamscore=0 suspectscore=3 priorityscore=1501 impostorscore=0 bulkscore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 lowpriorityscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to prepare for per-object slab memory accounting, convert NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE vmstat items to bytes. To make sure that these vmstats are in bytes, rename them to NR_SLAB_RECLAIMABLE_B and NR_SLAB_UNRECLAIMABLE_B (similar to NR_KERNEL_STACK_KB). The size of slab memory shouldn't exceed 4Gb on 32-bit machines, so it will fit into atomic_long_t we use for vmstats. Signed-off-by: Roman Gushchin --- drivers/base/node.c | 11 ++++++++--- fs/proc/meminfo.c | 4 ++-- include/linux/mmzone.h | 10 ++++++++-- include/linux/vmstat.h | 8 ++++++++ kernel/power/snapshot.c | 2 +- mm/memcontrol.c | 29 ++++++++++++++++------------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 ++++---- mm/slab.h | 15 ++++++++------- mm/slab_common.c | 4 ++-- mm/slob.c | 12 ++++++------ mm/slub.c | 8 ++++---- mm/vmscan.c | 3 ++- mm/vmstat.c | 22 +++++++++++++++++++--- mm/workingset.c | 6 ++++-- 15 files changed, 93 insertions(+), 51 deletions(-) diff --git a/drivers/base/node.c b/drivers/base/node.c index 296546ffed6c..56664222f3fd 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -368,8 +368,8 @@ static ssize_t node_read_meminfo(struct device *dev, unsigned long sreclaimable, sunreclaimable; si_meminfo_node(&i, nid); - sreclaimable = node_page_state(pgdat, NR_SLAB_RECLAIMABLE); - sunreclaimable = node_page_state(pgdat, NR_SLAB_UNRECLAIMABLE); + sreclaimable = node_page_state_pages(pgdat, NR_SLAB_RECLAIMABLE_B); + sunreclaimable = node_page_state_pages(pgdat, NR_SLAB_UNRECLAIMABLE_B); n = sprintf(buf, "Node %d MemTotal: %8lu kB\n" "Node %d MemFree: %8lu kB\n" @@ -495,9 +495,14 @@ static ssize_t node_read_vmstat(struct device *dev, int i; int n = 0; - for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) + for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { + unsigned long x = sum_zone_node_page_state(nid, i); + + if (vmstat_item_in_bytes(i)) + x >>= PAGE_SHIFT; n += sprintf(buf+n, "%s %lu\n", vmstat_text[i], sum_zone_node_page_state(nid, i)); + } #ifdef CONFIG_NUMA for (i = 0; i < NR_VM_NUMA_STAT_ITEMS; i++) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index ac9247371871..87afa8683c1b 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -53,8 +53,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v) pages[lru] = global_node_page_state(NR_LRU_BASE + lru); available = si_mem_available(); - sreclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE); - sunreclaim = global_node_page_state(NR_SLAB_UNRECLAIMABLE); + sreclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B); + sunreclaim = global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B); show_val_kb(m, "MemTotal: ", i.totalram); show_val_kb(m, "MemFree: ", i.freeram); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 6e8233d52971..2dbc2d042ef6 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -215,8 +215,8 @@ enum node_stat_item { NR_INACTIVE_FILE, /* " " " " " */ NR_ACTIVE_FILE, /* " " " " " */ NR_UNEVICTABLE, /* " " " " " */ - NR_SLAB_RECLAIMABLE, /* Please do not reorder this item */ - NR_SLAB_UNRECLAIMABLE, /* and this one without looking at + NR_SLAB_RECLAIMABLE_B, /* Please do not reorder this item */ + NR_SLAB_UNRECLAIMABLE_B,/* and this one without looking at * memcg_flush_percpu_vmstats() first. */ NR_ISOLATED_ANON, /* Temporary isolated pages from anon lru */ NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */ @@ -247,6 +247,12 @@ enum node_stat_item { NR_VM_NODE_STAT_ITEMS }; +static __always_inline bool vmstat_item_in_bytes(enum node_stat_item item) +{ + return (item == NR_SLAB_RECLAIMABLE_B || + item == NR_SLAB_UNRECLAIMABLE_B); +} + /* * We do arithmetic on the LRU lists in various places in the code, * so it is important to keep the active lists LRU_ACTIVE higher in diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index bdeda4b079fe..e5460e597e0c 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -194,6 +194,12 @@ static inline unsigned long global_node_page_state(enum node_stat_item item) return x; } +static inline +unsigned long global_node_page_state_pages(enum node_stat_item item) +{ + return global_node_page_state(item) >> PAGE_SHIFT; +} + static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) { @@ -234,6 +240,8 @@ extern unsigned long sum_zone_node_page_state(int node, extern unsigned long sum_zone_numa_state(int node, enum numa_stat_item item); extern unsigned long node_page_state(struct pglist_data *pgdat, enum node_stat_item item); +extern unsigned long node_page_state_pages(struct pglist_data *pgdat, + enum node_stat_item item); #else #define sum_zone_node_page_state(node, item) global_zone_page_state(item) #define node_page_state(node, item) global_node_page_state(item) diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index 83105874f255..ce9e5686e745 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1659,7 +1659,7 @@ static unsigned long minimum_image_size(unsigned long saveable) { unsigned long size; - size = global_node_page_state(NR_SLAB_RECLAIMABLE) + size = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B) + global_node_page_state(NR_ACTIVE_ANON) + global_node_page_state(NR_INACTIVE_ANON) + global_node_page_state(NR_ACTIVE_FILE) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb9adb31360e..e4af9810b59e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -759,13 +759,16 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) { - long x; + long x, threshold = MEMCG_CHARGE_BATCH; if (mem_cgroup_disabled()) return; + if (vmstat_item_in_bytes(idx)) + threshold <<= PAGE_SHIFT; + x = val + __this_cpu_read(memcg->vmstats_percpu->stat[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + if (unlikely(abs(x) > threshold)) { struct mem_cgroup *mi; /* @@ -807,7 +810,7 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, pg_data_t *pgdat = lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pn; struct mem_cgroup *memcg; - long x; + long x, threshold = MEMCG_CHARGE_BATCH; /* Update node */ __mod_node_page_state(pgdat, idx, val); @@ -824,8 +827,11 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, /* Update lruvec */ __this_cpu_add(pn->lruvec_stat_local->count[idx], val); + if (vmstat_item_in_bytes(idx)) + threshold <<= PAGE_SHIFT; + x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + if (unlikely(abs(x) > threshold)) { struct mem_cgroup_per_node *pi; for (pi = pn; pi; pi = parent_nodeinfo(pi, pgdat->node_id)) @@ -1478,9 +1484,8 @@ static char *memory_stat_format(struct mem_cgroup *memcg) (u64)memcg_page_state(memcg, MEMCG_KERNEL_STACK_KB) * 1024); seq_buf_printf(&s, "slab %llu\n", - (u64)(memcg_page_state(memcg, NR_SLAB_RECLAIMABLE) + - memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE)) * - PAGE_SIZE); + (u64)(memcg_page_state(memcg, NR_SLAB_RECLAIMABLE_B) + + memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE_B))); seq_buf_printf(&s, "sock %llu\n", (u64)memcg_page_state(memcg, MEMCG_SOCK) * PAGE_SIZE); @@ -1514,11 +1519,9 @@ static char *memory_stat_format(struct mem_cgroup *memcg) PAGE_SIZE); seq_buf_printf(&s, "slab_reclaimable %llu\n", - (u64)memcg_page_state(memcg, NR_SLAB_RECLAIMABLE) * - PAGE_SIZE); + (u64)memcg_page_state(memcg, NR_SLAB_RECLAIMABLE_B)); seq_buf_printf(&s, "slab_unreclaimable %llu\n", - (u64)memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE) * - PAGE_SIZE); + (u64)memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE_B)); /* Accumulated memory events */ @@ -3564,8 +3567,8 @@ static void memcg_flush_percpu_vmstats(struct mem_cgroup *memcg, bool slab_only) int min_idx, max_idx; if (slab_only) { - min_idx = NR_SLAB_RECLAIMABLE; - max_idx = NR_SLAB_UNRECLAIMABLE; + min_idx = NR_SLAB_RECLAIMABLE_B; + max_idx = NR_SLAB_UNRECLAIMABLE_B; } else { min_idx = 0; max_idx = MEMCG_NR_STAT; diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 314ce1a3cf25..61476bf0f147 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -183,7 +183,7 @@ static bool is_dump_unreclaim_slabs(void) global_node_page_state(NR_ISOLATED_FILE) + global_node_page_state(NR_UNEVICTABLE); - return (global_node_page_state(NR_SLAB_UNRECLAIMABLE) > nr_lru); + return (global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B) > nr_lru); } /** diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c5d62f1c2851..9da8ee92c226 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5100,8 +5100,8 @@ long si_mem_available(void) * items that are in use, and cannot be freed. Cap this estimate at the * low watermark. */ - reclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE) + - global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE); + reclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B) + + global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE); available += reclaimable - min(reclaimable / 2, wmark_low); if (available < 0) @@ -5245,8 +5245,8 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) global_node_page_state(NR_FILE_DIRTY), global_node_page_state(NR_WRITEBACK), global_node_page_state(NR_UNSTABLE_NFS), - global_node_page_state(NR_SLAB_RECLAIMABLE), - global_node_page_state(NR_SLAB_UNRECLAIMABLE), + global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B), + global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B), global_node_page_state(NR_FILE_MAPPED), global_node_page_state(NR_SHMEM), global_zone_page_state(NR_PAGETABLE), diff --git a/mm/slab.h b/mm/slab.h index 68e455f2b698..7c5577c2b9ea 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -272,7 +272,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **); static inline int cache_vmstat_idx(struct kmem_cache *s) { return (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; + NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; } #ifdef CONFIG_MEMCG_KMEM @@ -360,7 +360,7 @@ static __always_inline int memcg_charge_slab(struct page *page, if (unlikely(!memcg || mem_cgroup_is_root(memcg))) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - (1 << order)); + (PAGE_SIZE << order)); percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); return 0; } @@ -370,7 +370,7 @@ static __always_inline int memcg_charge_slab(struct page *page, goto out; lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); - mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), PAGE_SIZE << order); /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); @@ -394,11 +394,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, memcg = READ_ONCE(s->memcg_params.memcg); if (likely(!mem_cgroup_is_root(memcg))) { lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); - mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), + -(PAGE_SIZE << order)); memcg_kmem_uncharge_memcg(page, order, memcg); } else { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - -(1 << order)); + -(PAGE_SIZE << order)); } rcu_read_unlock(); @@ -482,7 +483,7 @@ static __always_inline int charge_slab_page(struct page *page, { if (is_root_cache(s)) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - 1 << order); + PAGE_SIZE << order); return 0; } @@ -494,7 +495,7 @@ static __always_inline void uncharge_slab_page(struct page *page, int order, { if (is_root_cache(s)) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - -(1 << order)); + -(PAGE_SIZE << order)); return; } diff --git a/mm/slab_common.c b/mm/slab_common.c index c29f03adca91..79695d9c34f3 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1303,8 +1303,8 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) page = alloc_pages(flags, order); if (likely(page)) { ret = page_address(page); - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - 1 << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); } ret = kasan_kmalloc_large(ret, size, flags); /* As ret might get tagged, call kmemleak hook after KASAN. */ diff --git a/mm/slob.c b/mm/slob.c index fa53e9f73893..8b7b56235438 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -202,8 +202,8 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) if (!page) return NULL; - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - 1 << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); return page_address(page); } @@ -214,8 +214,8 @@ static void slob_free_pages(void *b, int order) if (current->reclaim_state) current->reclaim_state->reclaimed_slab += 1 << order; - mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE, - -(1 << order)); + mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); __free_pages(sp, order); } @@ -550,8 +550,8 @@ void kfree(const void *block) slob_free(m, *m + align); } else { unsigned int order = compound_order(sp); - mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE, - -(1 << order)); + mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); __free_pages(sp, order); } diff --git a/mm/slub.c b/mm/slub.c index c9856a9807f1..0873b77727bf 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3825,8 +3825,8 @@ static void *kmalloc_large_node(size_t size, gfp_t flags, int node) page = alloc_pages_node(node, flags, order); if (page) { ptr = page_address(page); - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - 1 << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); } return kmalloc_large_node_hook(ptr, size, flags); @@ -3957,8 +3957,8 @@ void kfree(const void *x) BUG_ON(!PageCompound(page)); kfree_hook(object); - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - -(1 << order)); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); __free_pages(page, order); return; } diff --git a/mm/vmscan.c b/mm/vmscan.c index ee47bbcb99b5..283da9a39de2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4272,7 +4272,8 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order) * unmapped file backed pages. */ if (node_pagecache_reclaimable(pgdat) <= pgdat->min_unmapped_pages && - node_page_state(pgdat, NR_SLAB_RECLAIMABLE) <= pgdat->min_slab_pages) + node_page_state_pages(pgdat, NR_SLAB_RECLAIMABLE_B) <= + pgdat->min_slab_pages) return NODE_RECLAIM_FULL; /* diff --git a/mm/vmstat.c b/mm/vmstat.c index 98f43725d910..d04f53997fd2 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -344,6 +344,8 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, x = delta + __this_cpu_read(*p); t = __this_cpu_read(pcp->stat_threshold); + if (vmstat_item_in_bytes(item)) + t <<= PAGE_SHIFT; if (unlikely(x > t || x < -t)) { node_page_state_add(x, pgdat, item); @@ -555,6 +557,8 @@ static inline void mod_node_state(struct pglist_data *pgdat, * for all cpus in a node. */ t = this_cpu_read(pcp->stat_threshold); + if (vmstat_item_in_bytes(item)) + t <<= PAGE_SHIFT; o = this_cpu_read(*p); n = delta + o; @@ -999,6 +1003,12 @@ unsigned long node_page_state(struct pglist_data *pgdat, #endif return x; } + +unsigned long node_page_state_pages(struct pglist_data *pgdat, + enum node_stat_item item) +{ + return node_page_state(pgdat, item) >> PAGE_SHIFT; +} #endif #ifdef CONFIG_COMPACTION @@ -1547,10 +1557,13 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, if (is_zone_first_populated(pgdat, zone)) { seq_printf(m, "\n per-node stats"); for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) { + unsigned long x = node_page_state(pgdat, i); + + if (vmstat_item_in_bytes(i)) + x >>= PAGE_SHIFT; seq_printf(m, "\n %-12s %lu", vmstat_text[i + NR_VM_ZONE_STAT_ITEMS + - NR_VM_NUMA_STAT_ITEMS], - node_page_state(pgdat, i)); + NR_VM_NUMA_STAT_ITEMS], x); } } seq_printf(m, @@ -1679,8 +1692,11 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos) v += NR_VM_NUMA_STAT_ITEMS; #endif - for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) + for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) { v[i] = global_node_page_state(i); + if (vmstat_item_in_bytes(i)) + v[i] >>= PAGE_SHIFT; + } v += NR_VM_NODE_STAT_ITEMS; global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD, diff --git a/mm/workingset.c b/mm/workingset.c index c963831d354f..675792387e58 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -430,8 +430,10 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, NR_LRU_BASE + i); - pages += lruvec_page_state_local(lruvec, NR_SLAB_RECLAIMABLE); - pages += lruvec_page_state_local(lruvec, NR_SLAB_UNRECLAIMABLE); + pages += lruvec_page_state_local( + lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT; + pages += lruvec_page_state_local( + lruvec, NR_SLAB_UNRECLAIMABLE_B) >> PAGE_SHIFT; } else #endif pages = node_present_pages(sc->nid); From patchwork Thu Sep 5 21:45:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133981 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4347C76 for ; Thu, 5 Sep 2019 21:46:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2798220828 for ; Thu, 5 Sep 2019 21:46:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="WqjKf11s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2798220828 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F5566B000C; Thu, 5 Sep 2019 17:46:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A56C6B000D; Thu, 5 Sep 2019 17:46:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56BF26B000E; Thu, 5 Sep 2019 17:46:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id 3657F6B000C for ; Thu, 5 Sep 2019 17:46:15 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id DA50C824CA36 for ; Thu, 5 Sep 2019 21:46:14 +0000 (UTC) X-FDA: 75902200668.17.base94_8dc9709d2af18 X-Spam-Summary: 2,0,0,41c4b026fee057fa,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1535:1544:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:2731:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4385:4605:5007:6119:6120:6261:6653:7901:7903:10004:11026:11233:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:12986:13138:13161:13229:13231:13255:14096:14097:14181:14394:14721:21080:21450:21627:30054:30064,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: base94_8dc9709d2af18 X-Filterd-Recvd-Size: 6964 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:14 +0000 (UTC) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85LjfHO025642 for ; Thu, 5 Sep 2019 14:46:13 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=UwPjL9OaS6l+zw5+AffGQKehTN9tHhdRtYm7b2UrvFk=; b=WqjKf11sa3e1l8/TmjXpBgZNL7N1F+aC2Zd32rcPerdm+34rvdxiq45xBiZJV88F5cnD C7gK69ICIObQDdJ3mLVEBzmKblwoxcjl85brhwtnUMVpRE6k45YJG5qphEdVqkFbS78T 9lhlExt+asGT6SDMJM61c/wNGask0dLNmRE= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2uu8mdrhe7-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:13 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:09 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id B3CAA17229E00; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 05/14] mm: memcg/slab: allocate space for memcg ownership data for non-root slabs Date: Thu, 5 Sep 2019 14:45:49 -0700 Message-ID: <20190905214553.1643060-6-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 adultscore=0 spamscore=0 suspectscore=3 priorityscore=1501 impostorscore=0 bulkscore=0 mlxlogscore=749 clxscore=1015 mlxscore=0 lowpriorityscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocate and release memory for storing the memcg ownership data. For each slab page allocate space sufficient for number_of_objects pointers to struct mem_cgroup_vec. The mem_cgroup field of the struct page isn't used for slab pages, so let's use the space for storing the pointer for the allocated space. This commit makes sure that the space is ready for use, but nobody is actually using it yet. Following commits in the series will fix it. Signed-off-by: Roman Gushchin --- include/linux/mm_types.h | 5 ++++- mm/slab.c | 3 ++- mm/slab.h | 37 ++++++++++++++++++++++++++++++++++++- mm/slub.c | 2 +- 4 files changed, 43 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 25395481d2ae..510cb170c4b8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -199,7 +199,10 @@ struct page { atomic_t _refcount; #ifdef CONFIG_MEMCG - struct mem_cgroup *mem_cgroup; + union { + struct mem_cgroup *mem_cgroup; + struct mem_cgroup_ptr **mem_cgroup_vec; + }; #endif /* diff --git a/mm/slab.c b/mm/slab.c index 9df370558e5d..f0833f287dcf 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1369,7 +1369,8 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { + if (charge_slab_page(page, flags, cachep->gfporder, cachep, + cachep->num)) { __free_pages(page, cachep->gfporder); return NULL; } diff --git a/mm/slab.h b/mm/slab.h index 7c5577c2b9ea..16d7ea30a2d3 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -406,6 +406,23 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } +static inline int memcg_alloc_page_memcg_vec(struct page *page, gfp_t gfp, + unsigned int objects) +{ + page->mem_cgroup_vec = kmalloc(sizeof(struct mem_cgroup_ptr *) * + objects, gfp | __GFP_ZERO); + if (!page->mem_cgroup_vec) + return -ENOMEM; + + return 0; +} + +static inline void memcg_free_page_memcg_vec(struct page *page) +{ + kfree(page->mem_cgroup_vec); + page->mem_cgroup_vec = NULL; +} + extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); @@ -455,6 +472,16 @@ static inline void memcg_uncharge_slab(struct page *page, int order, { } +static inline int memcg_alloc_page_memcg_vec(struct page *page, gfp_t gfp, + unsigned int objects) +{ + return 0; +} + +static inline void memcg_free_page_memcg_vec(struct page *page) +{ +} + static inline void slab_init_memcg_params(struct kmem_cache *s) { } @@ -479,14 +506,21 @@ static inline struct kmem_cache *virt_to_cache(const void *obj) static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, - struct kmem_cache *s) + struct kmem_cache *s, + unsigned int objects) { + int ret; + if (is_root_cache(s)) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), PAGE_SIZE << order); return 0; } + ret = memcg_alloc_page_memcg_vec(page, gfp, objects); + if (ret) + return ret; + return memcg_charge_slab(page, gfp, order, s); } @@ -499,6 +533,7 @@ static __always_inline void uncharge_slab_page(struct page *page, int order, return; } + memcg_free_page_memcg_vec(page); memcg_uncharge_slab(page, order, s); } diff --git a/mm/slub.c b/mm/slub.c index 0873b77727bf..3014158c100d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1517,7 +1517,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, else page = __alloc_pages_node(node, flags, order); - if (page && charge_slab_page(page, flags, order, s)) { + if (page && charge_slab_page(page, flags, order, s, oo_objects(oo))) { __free_pages(page, order); page = NULL; } From patchwork Thu Sep 5 21:45:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133973 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C478776 for ; Thu, 5 Sep 2019 21:46:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2745206DE for ; Thu, 5 Sep 2019 21:46:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="Wtjh7Yvp" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2745206DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A9D196B0007; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A4DBA6B0008; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 914F46B000A; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0187.hostedemail.com [216.40.44.187]) by kanga.kvack.org (Postfix) with ESMTP id 6A9966B0007 for ; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id AC7314835 for ; Thu, 5 Sep 2019 21:46:12 +0000 (UTC) X-FDA: 75902200584.17.ear18_8d73c17e54a13 X-Spam-Summary: 2,0,0,151025dc06caa956,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1534:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2538:2559:2562:2693:2731:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3872:4250:4321:4385:4605:5007:6120:6261:6653:7875:7901:8603:10004:10400:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13141:13230:14096:14097:14181:14394:14721:21080:21450:21451:21627:30034:30054:30064,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ff,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: ear18_8d73c17e54a13 X-Filterd-Recvd-Size: 4927 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:11 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85LjAkP007158 for ; Thu, 5 Sep 2019 14:46:11 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=rZedqTwzLOw9+QlYlJrNiqvvmmCUA+1LduWzC6DbQlU=; b=Wtjh7YvprPq/wx77nE+Im9sKl64kliQH1v9NIYFx/meAYcS+D9g83JFWQyDteeH9rgWO 7Pfp/4k9C4tq69SqkYxQK/afcOh4mGNO2E8iC65qXzeloau5DQH36TnkhmKKnlFXWQjl SXWMVIwbpLb54fNCQVzW3FjP4E9BR4Ns2TQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2utkkxwup2-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:11 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:08 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id B7F3917229E02; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 06/14] mm: slub: implement SLUB version of obj_to_index() Date: Thu, 5 Sep 2019 14:45:50 -0700 Message-ID: <20190905214553.1643060-7-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 malwarescore=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxlogscore=840 mlxscore=0 clxscore=1015 adultscore=0 suspectscore=3 impostorscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit implements SLUB version of the obj_to_index() function, which will be required to calculate the offset of memcg_ptr in the mem_cgroup_vec to store/obtain the memcg ownership data. To make it faster, let's repeat the SLAB's trick introduced by commit 6a2d7a955d8d ("[PATCH] SLAB: use a multiply instead of a divide in obj_to_index()") and avoid an expensive division. Signed-off-by: Roman Gushchin --- include/linux/slub_def.h | 9 +++++++++ mm/slub.c | 1 + 2 files changed, 10 insertions(+) diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index d2153789bd9f..200ea292f250 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -8,6 +8,7 @@ * (C) 2007 SGI, Christoph Lameter */ #include +#include enum stat_item { ALLOC_FASTPATH, /* Allocation from cpu slab */ @@ -86,6 +87,7 @@ struct kmem_cache { unsigned long min_partial; unsigned int size; /* The size of an object including metadata */ unsigned int object_size;/* The size of an object without metadata */ + struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ @@ -182,4 +184,11 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, return result; } +static inline unsigned int obj_to_index(const struct kmem_cache *cache, + const struct page *page, void *obj) +{ + return reciprocal_divide(kasan_reset_tag(obj) - page_address(page), + cache->reciprocal_size); +} + #endif /* _LINUX_SLUB_DEF_H */ diff --git a/mm/slub.c b/mm/slub.c index 3014158c100d..b043cfb673c9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3587,6 +3587,7 @@ static int calculate_sizes(struct kmem_cache *s, int forced_order) */ size = ALIGN(size, s->align); s->size = size; + s->reciprocal_size = reciprocal_value(size); if (forced_order >= 0) order = forced_order; else From patchwork Thu Sep 5 21:45:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133975 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B29776 for ; Thu, 5 Sep 2019 21:46:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5FA04206DE for ; Thu, 5 Sep 2019 21:46:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="UtZZyZA3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5FA04206DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F31AF6B0008; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EE6596B000A; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D34D46B000C; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0249.hostedemail.com [216.40.44.249]) by kanga.kvack.org (Postfix) with ESMTP id AC74A6B000A for ; Thu, 5 Sep 2019 17:46:13 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id 4F4344419 for ; Thu, 5 Sep 2019 21:46:13 +0000 (UTC) X-FDA: 75902200626.20.fowl22_8d8ceb8910153 X-Spam-Summary: 2,0,0,64f1a4998d375f94,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:1:41:355:379:541:800:960:966:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2636:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:4605:5007:6120:6261:6653:7901:8603:8660:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12679:12683:12895:12986:13148:13161:13166:13229:13230:13255:14096:14097:14394:21080:21222:21433:21450:21451:21611:21627:21796:30012:30036:30054:30064,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:35,LUA_SUMMARY:none X-HE-Tag: fowl22_8d8ceb8910153 X-Filterd-Recvd-Size: 13106 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:12 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85LjAkQ007158 for ; Thu, 5 Sep 2019 14:46:11 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=n22EwJTVKLxIpi5IStATWjFYKvAR7CWIFYBmP0R5+4M=; b=UtZZyZA3eSqmEQvlEKbwypax1MB6y22iErMnVILfPF5WXZ+6ISD5dFDpFIKmb3pPvciD 2UA148CEzKMUekkg0GTKGVAULgwceoBqPt4BCHI4y+Pn3aNsw7kXoMNxOHyLZHsLJnVx cMVsI9MHVdF5kmculaaZihRgWGaDpAKx9Oo= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2utkkxwup2-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:11 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:08 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id BC95E17229E04; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 07/14] mm: memcg/slab: save memcg ownership data for non-root slab objects Date: Thu, 5 Sep 2019 14:45:51 -0700 Message-ID: <20190905214553.1643060-8-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 malwarescore=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxlogscore=999 mlxscore=0 clxscore=1015 adultscore=0 suspectscore=3 impostorscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Store a memcg_ptr in the corresponding place of the mem_cgroup_vec for each allocated non-root slab object. Make sure that each allocated object holds a reference to the mem_cgroup_ptr. To get the memcg_ptr in the post alloc hook, we need a memcg pointer. Because all memory cgroup will soon share the same set of kmem_caches, let's not use the kmem_cache->memcg_params.memcg. Instead, let's pass the pointer directly from memcg_kmem_get_cache(). This will guarantee that we will use the same cgroup in pre- and post-alloc hooks. Please, note that the code is a bit bulky now, because we have to manage 3 types of objects with reference counters: memcg, kmem_cache and memcg_ptr. The following commits in the series will simplify it. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 3 +- mm/memcontrol.c | 8 +++-- mm/slab.c | 18 ++++++----- mm/slab.h | 64 ++++++++++++++++++++++++++++++++++---- mm/slub.c | 14 ++++++--- 5 files changed, 86 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index dd5ebfe5a86c..b9643d758fc9 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1395,7 +1395,8 @@ static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, } #endif -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep); +struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep, + struct mem_cgroup **memcgp); void memcg_kmem_put_cache(struct kmem_cache *cachep); #ifdef CONFIG_MEMCG_KMEM diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e4af9810b59e..761b646eb968 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3015,7 +3015,8 @@ static inline bool memcg_kmem_bypass(void) * done with it, memcg_kmem_put_cache() must be called to release the * reference. */ -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) +struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep, + struct mem_cgroup **memcgp) { struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; @@ -3071,8 +3072,11 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) */ if (unlikely(!memcg_cachep)) memcg_schedule_kmem_cache_create(memcg, cachep); - else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) + else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) { + css_get(&memcg->css); + *memcgp = memcg; cachep = memcg_cachep; + } out_unlock: rcu_read_unlock(); return cachep; diff --git a/mm/slab.c b/mm/slab.c index f0833f287dcf..956aab3b9035 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3222,9 +3222,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, unsigned long save_flags; void *ptr; int slab_node = numa_mem_id(); + struct mem_cgroup *memcg = NULL; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, flags); + cachep = slab_pre_alloc_hook(cachep, &memcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3260,7 +3261,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); - slab_post_alloc_hook(cachep, flags, 1, &ptr); + slab_post_alloc_hook(cachep, memcg, flags, 1, &ptr); return ptr; } @@ -3301,9 +3302,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) { unsigned long save_flags; void *objp; + struct mem_cgroup *memcg = NULL; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, flags); + cachep = slab_pre_alloc_hook(cachep, &memcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3317,7 +3319,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); - slab_post_alloc_hook(cachep, flags, 1, &objp); + slab_post_alloc_hook(cachep, memcg, flags, 1, &objp); return objp; } @@ -3439,6 +3441,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); + memcg_slab_free_hook(cachep, virt_to_head_page(objp), objp); /* * Skip calling cache_free_alien() when the platform is not numa. @@ -3504,8 +3507,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) { size_t i; + struct mem_cgroup *memcg = NULL; - s = slab_pre_alloc_hook(s, flags); + s = slab_pre_alloc_hook(s, &memcg, size, flags); if (!s) return 0; @@ -3528,13 +3532,13 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); - slab_post_alloc_hook(s, flags, size, p); + slab_post_alloc_hook(s, memcg, flags, size, p); /* FIXME: Trace call missing. Christoph would like a bulk variant */ return size; error: local_irq_enable(); cache_alloc_debugcheck_after_bulk(s, flags, i, p, _RET_IP_); - slab_post_alloc_hook(s, flags, i, p); + slab_post_alloc_hook(s, memcg, flags, i, p); __kmem_cache_free_bulk(s, i, p); return 0; } diff --git a/mm/slab.h b/mm/slab.h index 16d7ea30a2d3..39d93c970839 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -423,6 +423,45 @@ static inline void memcg_free_page_memcg_vec(struct page *page) page->mem_cgroup_vec = NULL; } +static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct mem_cgroup *memcg, + size_t size, void **p) +{ + struct mem_cgroup_ptr *memcg_ptr; + struct page *page; + unsigned long off; + size_t i; + + memcg_ptr = mem_cgroup_get_kmem_ptr(memcg); + for (i = 0; i < size; i++) { + if (likely(p[i])) { + page = virt_to_head_page(p[i]); + off = obj_to_index(s, page, p[i]); + mem_cgroup_ptr_get(memcg_ptr); + page->mem_cgroup_vec[off] = memcg_ptr; + } + } + mem_cgroup_ptr_put(memcg_ptr); + mem_cgroup_put(memcg); + + memcg_kmem_put_cache(s); +} + +static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, + void *p) +{ + struct mem_cgroup_ptr *memcg_ptr; + unsigned int off; + + if (!memcg_kmem_enabled() || is_root_cache(s)) + return; + + off = obj_to_index(s, page, p); + memcg_ptr = page->mem_cgroup_vec[off]; + page->mem_cgroup_vec[off] = NULL; + mem_cgroup_ptr_put(memcg_ptr); +} + extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); @@ -482,6 +521,17 @@ static inline void memcg_free_page_memcg_vec(struct page *page) { } +static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct mem_cgroup *memcg, + size_t size, void **p) +{ +} + +static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, + void *p) +{ +} + static inline void slab_init_memcg_params(struct kmem_cache *s) { } @@ -591,7 +641,8 @@ static inline size_t slab_ksize(const struct kmem_cache *s) } static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, - gfp_t flags) + struct mem_cgroup **memcgp, + size_t size, gfp_t flags) { flags &= gfp_allowed_mask; @@ -605,13 +656,14 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, if (memcg_kmem_enabled() && ((flags & __GFP_ACCOUNT) || (s->flags & SLAB_ACCOUNT))) - return memcg_kmem_get_cache(s); + return memcg_kmem_get_cache(s, memcgp); return s; } -static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, - size_t size, void **p) +static inline void slab_post_alloc_hook(struct kmem_cache *s, + struct mem_cgroup *memcg, + gfp_t flags, size_t size, void **p) { size_t i; @@ -623,8 +675,8 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, s->flags, flags); } - if (memcg_kmem_enabled()) - memcg_kmem_put_cache(s); + if (!is_root_cache(s)) + memcg_slab_post_alloc_hook(s, memcg, size, p); } #ifndef CONFIG_SLOB diff --git a/mm/slub.c b/mm/slub.c index b043cfb673c9..e8c44144a85f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2686,8 +2686,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct kmem_cache_cpu *c; struct page *page; unsigned long tid; + struct mem_cgroup *memcg = NULL; - s = slab_pre_alloc_hook(s, gfpflags); + s = slab_pre_alloc_hook(s, &memcg, 1, gfpflags); if (!s) return NULL; redo: @@ -2767,7 +2768,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); - slab_post_alloc_hook(s, gfpflags, 1, &object); + slab_post_alloc_hook(s, memcg, gfpflags, 1, &object); return object; } @@ -2972,6 +2973,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s, void *tail_obj = tail ? : head; struct kmem_cache_cpu *c; unsigned long tid; + + memcg_slab_free_hook(s, page, head); redo: /* * Determine the currently cpus per cpu slab. @@ -3149,9 +3152,10 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, { struct kmem_cache_cpu *c; int i; + struct mem_cgroup *memcg = NULL; /* memcg and kmem_cache debug support */ - s = slab_pre_alloc_hook(s, flags); + s = slab_pre_alloc_hook(s, &memcg, size, flags); if (unlikely(!s)) return false; /* @@ -3193,11 +3197,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, } /* memcg and kmem_cache debug support */ - slab_post_alloc_hook(s, flags, size, p); + slab_post_alloc_hook(s, memcg, flags, size, p); return i; error: local_irq_enable(); - slab_post_alloc_hook(s, flags, i, p); + slab_post_alloc_hook(s, memcg, flags, i, p); __kmem_cache_free_bulk(s, i, p); return 0; } From patchwork Thu Sep 5 21:45:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133977 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C4A91599 for ; Thu, 5 Sep 2019 21:46:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5FCD7206DE for ; Thu, 5 Sep 2019 21:46:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="SLK8TMs9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5FCD7206DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8306C6B0010; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 719326B000A; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E3B36B000E; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0152.hostedemail.com [216.40.44.152]) by kanga.kvack.org (Postfix) with ESMTP id 304D06B000C for ; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id D1B41180AD7C3 for ; Thu, 5 Sep 2019 21:46:13 +0000 (UTC) X-FDA: 75902200626.19.top49_8da5645d5370c X-Spam-Summary: 2,0,0,fba7b255ebb4c90f,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:2731:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3871:3872:4321:4605:5007:6120:6261:6653:7901:9592:10004:10400:11026:11658:11914:12043:12296:12297:12438:12555:12895:13069:13161:13229:13311:13357:14096:14097:14181:14394:14721:14819:14877:21080:21092:21451:21627:21740:30012:30054:30064,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: top49_8da5645d5370c X-Filterd-Recvd-Size: 4434 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:13 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85LgktN016046 for ; Thu, 5 Sep 2019 14:46:12 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=a0xRfNvZ6A20DAq+kUq8yz7y4FX28qaCwqEaBNzxJZ8=; b=SLK8TMs9JaS/3zhY99IsTN2IIENV2AQXlVDY2z+/yhYKIx4OgbHAEfEqr1uYpigRgLtK GoEbj8M1xVgaVndUUvNS14FrWlYrsNEGkWu73HdOODA5zHqN7eSVKBYI9DgkUn9f8bjx WqqT6qiWf/web1F2ts101CGDda/ox/gb1t0= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2uu93b0drb-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:12 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:08 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id C0C5817229E06; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 08/14] mm: memcg: move memcg_kmem_bypass() to memcontrol.h Date: Thu, 5 Sep 2019 14:45:52 -0700 Message-ID: <20190905214553.1643060-9-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 spamscore=0 clxscore=1015 lowpriorityscore=0 suspectscore=1 impostorscore=0 adultscore=0 bulkscore=0 phishscore=0 malwarescore=0 mlxscore=0 mlxlogscore=982 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050202 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To make the memcg_kmem_bypass() function available outside of the memcontrol.c, let's move it to memcontrol.h. The function is small and nicely fits into static inline sort of functions. It will be used from the slab code. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 7 +++++++ mm/memcontrol.c | 7 ------- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b9643d758fc9..8f1d7161579f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1430,6 +1430,13 @@ static inline bool memcg_kmem_enabled(void) return static_branch_unlikely(&memcg_kmem_enabled_key); } +static inline bool memcg_kmem_bypass(void) +{ + if (in_interrupt() || !current->mm || (current->flags & PF_KTHREAD)) + return true; + return false; +} + static inline int memcg_kmem_charge(struct page *page, gfp_t gfp, int order) { if (memcg_kmem_enabled()) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 761b646eb968..d57f95177aec 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2992,13 +2992,6 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg, queue_work(memcg_kmem_cache_wq, &cw->work); } -static inline bool memcg_kmem_bypass(void) -{ - if (in_interrupt() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - return false; -} - /** * memcg_kmem_get_cache: select the correct per-memcg cache for allocation * @cachep: the original global kmem cache From patchwork Thu Sep 5 21:45:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11133979 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4CAD81599 for ; Thu, 5 Sep 2019 21:46:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 29F2A20870 for ; Thu, 5 Sep 2019 21:46:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="KVrfu9Ck" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 29F2A20870 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A5C946B000A; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 82F416B000C; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D70A6B000D; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0206.hostedemail.com [216.40.44.206]) by kanga.kvack.org (Postfix) with ESMTP id 327BB6B000D for ; Thu, 5 Sep 2019 17:46:14 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id D34D134A3 for ; Thu, 5 Sep 2019 21:46:13 +0000 (UTC) X-FDA: 75902200626.13.sink61_8da50b2fb3022 X-Spam-Summary: 2,0,0,f3dbda0952b7e64b,d41d8cd98f00b204,prvs=4151d11f3c=guro@fb.com,::mhocko@kernel.org:hannes@cmpxchg.org:linux-kernel@vger.kernel.org:kernel-team@fb.com:shakeelb@google.com:vdavydov.dev@gmail.com:longman@redhat.com:guro@fb.com,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1261:1277:1313:1314:1345:1359:1437:1516:1518:1535:1544:1605:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3872:3874:4118:4250:4321:4385:4605:5007:6120:6261:6653:7875:7901:7903:9592:10004:11026:11473:11658:11914:12043:12114:12291:12296:12297:12438:12555:12683:12895:12986:13227:13229:14096:14097:14181:14394:14721:21080:21450:21627:30045:30054:30064:30070,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.2.0.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ff,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: sink61_8da50b2fb3022 X-Filterd-Recvd-Size: 7603 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Thu, 5 Sep 2019 21:46:13 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x85LjCGc007194 for ; Thu, 5 Sep 2019 14:46:12 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=SigM+i1toavMqTPh1qC3TfkVYP0HGpMJnAT2A/4oDIs=; b=KVrfu9CkBaur46aD/cHEmbEqERZE7J2QdSzmIO8FItCMn8BwDW15rYqMGmTavwiNVxLr kYC+h7RYjOn+Gm3A1sZW6SK6chgynstaMGoEAB65qFQw1KR6xpDBXvt+TrLcsVicFAZ9 EwD2zZC0xee6Wlcs53akE+QJ61E8OtsbwRk= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2utkkxwup9-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 05 Sep 2019 14:46:12 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 5 Sep 2019 14:46:09 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id C4A0717229E08; Thu, 5 Sep 2019 14:46:06 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: CC: Michal Hocko , Johannes Weiner , , , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH RFC 09/14] mm: memcg: introduce __mod_lruvec_memcg_state() Date: Thu, 5 Sep 2019 14:45:53 -0700 Message-ID: <20190905214553.1643060-10-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190905214553.1643060-1-guro@fb.com> References: <20190905214553.1643060-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-05_08:2019-09-04,2019-09-05 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 malwarescore=0 lowpriorityscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxlogscore=701 mlxscore=0 clxscore=1015 adultscore=0 suspectscore=3 impostorscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909050203 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for per-object accounting of slab objects, let's introduce __mod_lruvec_memcg_state() and mod_lruvec_memcg_state() helpers, which are similar to mod_lruvec_state(), but do not update global node counters, only lruvec and per-cgroup. It's necessary because soon node slab counters will be used for accounting of all memory used by slab pages, however on memcg level only the actually used memory will be counted. Free space will be shared between all cgroups, so it can't be accounted to any. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 22 ++++++++++++++++++++++ mm/memcontrol.c | 37 +++++++++++++++++++++++++++---------- 2 files changed, 49 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8f1d7161579f..cef8a9c51482 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -739,6 +739,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); +void __mod_lruvec_memcg_state(struct lruvec *lruvec, enum node_stat_item idx, + int val); void __mod_lruvec_slab_state(void *p, enum node_stat_item idx, int val); static inline void mod_lruvec_state(struct lruvec *lruvec, @@ -751,6 +753,16 @@ static inline void mod_lruvec_state(struct lruvec *lruvec, local_irq_restore(flags); } +static inline void mod_lruvec_memcg_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ + unsigned long flags; + + local_irq_save(flags); + __mod_lruvec_memcg_state(lruvec, idx, val); + local_irq_restore(flags); +} + static inline void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { @@ -1143,6 +1155,16 @@ static inline void mod_lruvec_state(struct lruvec *lruvec, mod_node_page_state(lruvec_pgdat(lruvec), idx, val); } +static inline void __mod_lruvec_memcg_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ +} + +static inline void mod_lruvec_memcg_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ +} + static inline void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d57f95177aec..89a892ef7699 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -795,16 +795,16 @@ parent_nodeinfo(struct mem_cgroup_per_node *pn, int nid) } /** - * __mod_lruvec_state - update lruvec memory statistics + * __mod_lruvec_memcg_state - update lruvec memory statistics * @lruvec: the lruvec * @idx: the stat item * @val: delta to add to the counter, can be negative * * The lruvec is the intersection of the NUMA node and a cgroup. This - * function updates the all three counters that are affected by a - * change of state at this level: per-node, per-cgroup, per-lruvec. + * function updates the two of three counters that are affected by a + * change of state at this level: per-cgroup and per-lruvec. */ -void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, +void __mod_lruvec_memcg_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { pg_data_t *pgdat = lruvec_pgdat(lruvec); @@ -812,12 +812,6 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, struct mem_cgroup *memcg; long x, threshold = MEMCG_CHARGE_BATCH; - /* Update node */ - __mod_node_page_state(pgdat, idx, val); - - if (mem_cgroup_disabled()) - return; - pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); memcg = pn->memcg; @@ -841,6 +835,29 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, __this_cpu_write(pn->lruvec_stat_cpu->count[idx], x); } +/** + * __mod_lruvec_state - update lruvec memory statistics + * @lruvec: the lruvec + * @idx: the stat item + * @val: delta to add to the counter, can be negative + * + * The lruvec is the intersection of the NUMA node and a cgroup. This + * function updates the all three counters that are affected by a + * change of state at this level: per-node, per-cgroup, per-lruvec. + */ +void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val) +{ + pg_data_t *pgdat = lruvec_pgdat(lruvec); + + /* Update node */ + __mod_node_page_state(pgdat, idx, val); + + /* Update per-cgroup and per-lruvec stats */ + if (!mem_cgroup_disabled()) + __mod_lruvec_memcg_state(lruvec, idx, val); +} + void __mod_lruvec_slab_state(void *p, enum node_stat_item idx, int val) { struct page *page = virt_to_head_page(p);