From patchwork Thu Aug 31 16:56:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13371726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1E36C83F32 for ; Thu, 31 Aug 2023 16:56:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DF158D0019; Thu, 31 Aug 2023 12:56:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 169398D0001; Thu, 31 Aug 2023 12:56:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F25AC8D0019; Thu, 31 Aug 2023 12:56:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E02D88D0001 for ; Thu, 31 Aug 2023 12:56:18 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C2753160301 for ; Thu, 31 Aug 2023 16:56:18 +0000 (UTC) X-FDA: 81185002836.20.64493CB Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id F2B3418001D for ; Thu, 31 Aug 2023 16:56:16 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="l5/SLQg6"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3MMbwZAoKCK0lbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3MMbwZAoKCK0lbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693500977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6NYJxZAD84nnU9JwFvW1AuIl8sWdu9iyZk6Te6FK6mM=; b=IXz6tEqtDiZ4auF2sEK9tAlh5ST4RJfFMLICYvqZ+6Vd/PLe2DDcMLK3KpZ4zLuckGeyVk tyMv53tPHTL5t6qgYeDVbwOaWzw5i8n3H0uV6ioJKvnBYFNuSRYtdYpmhNXAbSOx5iDU8o WQXjMeVpEI2czw93430eeIzvfQJGDaQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="l5/SLQg6"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3MMbwZAoKCK0lbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3MMbwZAoKCK0lbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693500977; a=rsa-sha256; cv=none; b=3UaP0QA63lfer2/4Q7d6cpufwTxJHiPe2v02/QU16swUlCGe19zj6itb1XaAIyuE4/4ecA wyRDkQH45yzvE2h1emLCRfKYXe6Yyky+4F5D3+2P8Nh8ngqV1Jxm8oIEufOKy4+Af8Z0lg nZgRRUBhayn1yKYCndkVejnsTKwwVY0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-58f9db8bc1dso13233107b3.3 for ; Thu, 31 Aug 2023 09:56:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693500976; x=1694105776; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6NYJxZAD84nnU9JwFvW1AuIl8sWdu9iyZk6Te6FK6mM=; b=l5/SLQg69ONmScRgPnnTXDSyf6lA/gtT7IPvq5ieaZ7zw7FLE4E8JLmqoRAusmEe/C bVLMAoXou4vB4QCC4GQMEL7HjwztN6RD9UzyIABmiOBurfPoGy+n/b3kyTgWon0Dwa2B rl7WCvFaM2Xuevj7sKZD4aLdqPV7YrkZCGaCorCs0RVvpl9HaEb37Ibw+YX8AfxcGbPO BTDGwHo0f0ab+5kg3EPHlGA/yFJY66WOJlrvDmeTJCPbt9kcLXiXigYaGD16WP/ju/XZ XdgwC919XOyeDoI6HhmghL23JHpFgIsmrqXA9MMkdzoRIDTXDE2DNQRWxmTefEMv7Pow //aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693500976; x=1694105776; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6NYJxZAD84nnU9JwFvW1AuIl8sWdu9iyZk6Te6FK6mM=; b=IAfmXX+3v5orMndFSS9+v7mQFdIbJn0wLCQAZ1LnCKLug/J2u8mczB58KB+ZuqxQC3 b732reYCOJJQ1FJfRWoe+biccqqoKoZtRi/QUws8cyMuKA4b449dKOqUZwxNWcKKvyL9 ZGhly6e7OFRgs89Uxdbrcms5a78d48lUoHa3lzot9y+M35j29fm09BPce2PIXv/Qerf1 zivTgiKyFRCL/gjWuETDbcivpbZpk1U20rypXe8ENWQXHWBfqJDC2VcLI9mSECrakw13 4DZUG7n6Bu1zNC2rZr0j+ir+jH11q3V4a7C+/nNy4jLePv5GBX4DPdx+ZFCuGEnVrNIg yXgg== X-Gm-Message-State: AOJu0YxjG8IJw2xm9+EhicN2uXCTonLlj+tvgvm/5hJJK11owJtDydGB VTxlbdI0VA06vMWvCWRtkEcTKD5VrGGObMrU X-Google-Smtp-Source: AGHT+IG+OUEj8IqQB30ouOiPIGWocFJhxTzlAMy3JDUj6cQD4hmj7r5Edc+cYhwbUNs50K1U4Cy6rv2ZeUvHgJsN X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a5b:d4a:0:b0:d73:bcb7:7282 with SMTP id f10-20020a5b0d4a000000b00d73bcb77282mr7521ybr.8.1693500976143; Thu, 31 Aug 2023 09:56:16 -0700 (PDT) Date: Thu, 31 Aug 2023 16:56:08 +0000 In-Reply-To: <20230831165611.2610118-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230831165611.2610118-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230831165611.2610118-2-yosryahmed@google.com> Subject: [PATCH v4 1/4] mm: memcg: properly name and document unified stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: F2B3418001D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 6hcagpnq1xwqbot67jzgie4r1cpb9gw4 X-HE-Tag: 1693500976-517274 X-HE-Meta: U2FsdGVkX18U27p3cg9sAyZDcOIwRmutM3DjEcQYuKCVMvVIHIk+I/Rt6U7IDXtEMVkT9RZy7eHbTn+YWUT4jMsAkTJ6lT9yX/5Wy6s1FGUp+R9PHNiH4Wdp+cxqj9/y2DihbmaLiR4CJYXxABblC7lzZ7Z5kbqWr65z6uo5qQPYTmFhmILJEcGdbQsEDhqq/lx8hY+GFgZ0Xua4U+tRVkoNHjpD5csAnHplRW7HL9AWan6ilH3KdBYrHotD0bKN6t7tBtBfGOmNbPe/MxhUW0g+9Wdxsnt2juA8sWsp64JbCBx1Borung0icwTlai74PA6Txth0/ShBiCbwdg4rzjsDxphBvj1yAXL49YrTMTu6NhHOCJ5acnFssvbjr5oJ1m0L1+bQzU2PWJo5iT+zLhqqZLOm0owSshxA8nEMG5HGLNNe5NrhIG4m22i/8qLIagxGRTXW4z0OrLIt0TZgmJIH4xZmxLSC43hxsig581EUSttwfht8ljf/T6Vtuqgn8l64EQDAZ3EhcqEZ5I5NcOh0S2UXeZGAEQuudG5gXXbQE0hkogxOiC+Fn2VezK+2bW6LSyCIPtDCqxolN2IsD+yh8uVrKpRL05xX9FlTIRtWYkfqFzLJFWsmSNKdH7lDzzzpVnSyKcOQqhT74GrY/QItp8e61LB2g3Ztx6DFjv6gPjgB2fY1wSLthcAqZiFLyFI5JBsYFHTH6/smilhH/ENYzbCdY3i+0ppvpfhkxckyqrSCZb0RkI24dd2MaEIFaVLjiCyfn9LoqrgiTrQxSDt6ef390Qv9yANxITZ7hckJKQYgwKZ9NZCRGfloNY4djUWWlMBfR+KnWh7xgsmyV3lSqGVfPEP+Klq2i2wW8/iYBcMe9G6HtxvrM16FY90GgHU528XuFecUX99Wqi2xVqyKeoiHtZ7Cm1yw/m19qx3tT5vgfiNTmIylGrI5UGx3nqwlU5TF2Pbq2BbDO/D TtCvU/hI YEBvi5EaoT7VY1xC4FatY2y6hyJku9CZoL1z91SFLpK1A9CfK2bySf3RJ0qPiMubgJ2qu+CvoGUK77Y83kzDKJhfrVq08fTYmbE6P2ULvssFwIvNbrm9Q0M63AvJ61s+fcAdAOrA562wtwFygWZKdN0p7dMcxOj1bk+aB9gs043c/IPbd8fgIli8lKFWCB8/dFC8rR9jWpvmrPlBEVIrn3tcURivnELZeSYNNYGtUENvuaduI2QT52xLY4fkNIzIWYiGk4XGQUdhWdV58H2N4C15s5S9UNvYZVLlXTAUV/Kk/KTv/YpVejO0MvyJKi+sX7kIf1Cc6wtz6TFXvwnYS/VnOl7O6va4cdKX1+jxfpiIUwi8pCo9WbDy8iamNARsb/hu+A5HRzROtDOV8tapfmimwSdjwUxa8CSa+0/4bH26e7QexpejXWqeqAIp0vk4xjDGmK6LNU6M/rIY7uMgT26JLqqgeLiqhV+MONM1IxMp/1b/84qRme/3NFYufFTNYyEO1XepIlZMRhmXeXqnSyU877LirFT8PMnsraaxK+OiTDsJpYGaMn4KnecCg66zRZYjTNuPSeLCGFWlyoT/wHkTBbuBAgjWkHoaBdW3t7QQ8soGI0dYD6fX5xv7pUClJMJieLDd+sduDiU8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most contexts that flush memcg stats use "unified" flushing, where basically all flushers attempt to flush the entire hierarchy, but only one flusher is allowed at a time, others skip flushing. This is needed because we need to flush the stats from paths such as reclaim or refaults, which may have high concurrency, especially on large systems. Serializing such performance-sensitive paths can introduce regressions, hence, unified flushing offers a tradeoff between stats staleness and the performance impact of flushing stats. Document this properly and explicitly by renaming the common flushing helper from do_flush_stats() to do_unified_stats_flush(), and adding documentation to describe unified flushing. Additionally, rename flushing APIs to add "try" in the name, which implies that flushing will not always happen. Also add proper documentation. No functional change intended. Signed-off-by: Yosry Ahmed Acked-by: Michal Hocko --- include/linux/memcontrol.h | 8 ++--- mm/memcontrol.c | 61 +++++++++++++++++++++++++------------- mm/vmscan.c | 2 +- mm/workingset.c | 4 +-- 4 files changed, 47 insertions(+), 28 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 11810a2cfd2d..d517b0cc5221 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1034,8 +1034,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_try_flush_stats(void); +void mem_cgroup_try_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1519,11 +1519,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_try_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_try_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cf57fe9318d5..2d0ec828a1c4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -588,7 +588,7 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); static DEFINE_PER_CPU(unsigned int, stats_updates); -static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); +static atomic_t stats_unified_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -630,7 +630,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) /* * If stats_flush_threshold exceeds the threshold * (>num_online_cpus()), cgroup stats update will be triggered - * in __mem_cgroup_flush_stats(). Increasing this var further + * in mem_cgroup_try_flush_stats(). Increasing this var further * is redundant and simply adds overhead in atomic update. */ if (atomic_read(&stats_flush_threshold) <= num_online_cpus()) @@ -639,15 +639,19 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void do_flush_stats(void) +/* + * do_unified_stats_flush - do a unified flush of memory cgroup statistics + * + * A unified flush tries to flush the entire hierarchy, but skips if there is + * another ongoing flush. This is meant for flushers that may have a lot of + * concurrency (e.g. reclaim, refault, etc), and should not be serialized to + * avoid slowing down performance-sensitive paths. A unified flush may skip, and + * hence may yield stale stats. + */ +static void do_unified_stats_flush(void) { - /* - * We always flush the entire tree, so concurrent flushers can just - * skip. This avoids a thundering herd problem on the rstat global lock - * from memcg flushers (e.g. reclaim, refault, etc). - */ - if (atomic_read(&stats_flush_ongoing) || - atomic_xchg(&stats_flush_ongoing, 1)) + if (atomic_read(&stats_unified_flush_ongoing) || + atomic_xchg(&stats_unified_flush_ongoing, 1)) return; WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); @@ -655,19 +659,34 @@ static void do_flush_stats(void) cgroup_rstat_flush(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - atomic_set(&stats_flush_ongoing, 0); + atomic_set(&stats_unified_flush_ongoing, 0); } -void mem_cgroup_flush_stats(void) +/* + * mem_cgroup_try_flush_stats - try to flush memory cgroup statistics + * + * Try to flush the stats of all memcgs that have stat updates since the last + * flush. We do not flush the stats if: + * - The magnitude of the pending updates is below a certain threshold. + * - There is another ongoing unified flush (see do_unified_stats_flush()). + * + * Hence, the stats may be stale, but ideally by less than FLUSH_TIME due to + * periodic flushing. + */ +void mem_cgroup_try_flush_stats(void) { if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - do_flush_stats(); + do_unified_stats_flush(); } -void mem_cgroup_flush_stats_ratelimited(void) +/* + * Like mem_cgroup_try_flush_stats(), but only flushes if the periodic flusher + * is late. + */ +void mem_cgroup_try_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -676,7 +695,7 @@ static void flush_memcg_stats_dwork(struct work_struct *w) * Always flush here so that flushing in latency-sensitive paths is * as cheap as possible. */ - do_flush_stats(); + do_unified_stats_flush(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1576,7 +1595,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4018,7 +4037,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4093,7 +4112,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4595,7 +4614,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6610,7 +6629,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; diff --git a/mm/vmscan.c b/mm/vmscan.c index c7c149cb8d66..457a18921fda 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2923,7 +2923,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index da58a26d0d4d..affb8699e58d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -520,7 +520,7 @@ void workingset_refault(struct folio *folio, void *shadow) } /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_try_flush_stats_ratelimited(); rcu_read_lock(); @@ -664,7 +664,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, From patchwork Thu Aug 31 16:56:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13371727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A463FC83F35 for ; Thu, 31 Aug 2023 16:56:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 255AC8D001A; Thu, 31 Aug 2023 12:56:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 204D58D0001; Thu, 31 Aug 2023 12:56:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07E458D001A; Thu, 31 Aug 2023 12:56:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E8A7B8D0001 for ; Thu, 31 Aug 2023 12:56:21 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B392D4029C for ; Thu, 31 Aug 2023 16:56:21 +0000 (UTC) X-FDA: 81185002962.23.04A3B11 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf18.hostedemail.com (Postfix) with ESMTP id E96221C0033 for ; Thu, 31 Aug 2023 16:56:19 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=57Tt0ekS; spf=pass (imf18.hostedemail.com: domain of 3MsbwZAoKCK8ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3MsbwZAoKCK8ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693500980; a=rsa-sha256; cv=none; b=t+RztL8f/qTlL9JoC7F6V6Lc3mTOJKKZOiCWFHLuxhkKad6BmqRjAAH9qQCSES1hH8k/aJ YD84/ptQ3j7XPKmC6WA7D9A7v3678gt/6pymmYLdUflv5eA+wzwZmKntinT9/KvZzPDFZw R4BCWHKOR4KKgQmm7lz0pZWq9fNab5c= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=57Tt0ekS; spf=pass (imf18.hostedemail.com: domain of 3MsbwZAoKCK8ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3MsbwZAoKCK8ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693500980; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rnl+F2n1kOs4P011p6EJFvzkt9Ylx9T9gDi6gAnNyhk=; b=4GvpNtU8SirkjcUe8D9Pd3/AROgG/g8f+1HWJ8Snr3qiISU4hMQCu+3IfNWTOLfqbA46ZG xapzQ5IsMBn8+wI0zt2JoJtqzYGw/X4LkuFkyLqmxgFfE1M6LJy8c3itR/Iw5y5JQ7Fi2H tJOVONEe1owCjBO/7pd1ouqTYZwYUQU= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-68bf47ff13cso1182738b3a.1 for ; Thu, 31 Aug 2023 09:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693500979; x=1694105779; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rnl+F2n1kOs4P011p6EJFvzkt9Ylx9T9gDi6gAnNyhk=; b=57Tt0ekSFpbTNTkqbfDC4EYKBgTusUH/5M88a9hgGDbEIzgaSkkGOLyQilh7BR2Pfv AaOuKVJbSGU0JLWxYqSgnkvXk2lBJkOy8h3kDcVkD9fIuPIU68yWCAH9pHywrXbqOj6d lQTYmQaIpoziGNcaJ0hdr9KtxmxO4/9Uq1NnNeBG87eLl93jZSCCEE2a/CmsLuxZiBNr Axw8glAy7pHwc7+wuG2pd+Z07LWbSLH53TqujvBUa1P+K/6AMKpKBf6F048Msg36c9Rc XcraSECSOZrNmAeDTNmWlBZTubhrSyYsjrpe7gdE8ZyPK+FqnlTBL6RVycdN2JzdMMaD TpYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693500979; x=1694105779; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rnl+F2n1kOs4P011p6EJFvzkt9Ylx9T9gDi6gAnNyhk=; b=JcaCQ3SD3EFG3Na8YJgASRw8OynJQdsK2K5pYo8fnllU+zdM1PdsHYIwEwvEij34Yp kDakfXi83rktEhZPQeMDVkfcDN6Lndnwa4KQgeQZ3dOrFluLPdqg+tg7IeuBZoriNqjG Dlo23C2hM5XzZLJ5raVNAODfAABq+ukGbWTdU4f+Bj2VJ9l9GVvbDWrtNKNo66xWjw5+ aq0MmkuqtAnBe+zOzokv3IG0OswiNLlRcmgx0ivkp2jBOSUCkCK1OmBBIMaxk7NS9frZ ddWITiVhtoBBnsODdWyYbzQkOEhJ6y9gMfczKkv0o7qtrP3OQh3pw3UhYKnlwT/9vjDc PATg== X-Gm-Message-State: AOJu0YzQzoMg+kfGYvFla8/rjIE0BF/wRF21uDszXcJ3V+IZ3HZysWSu z6jtKlDDBhJHe98gpgYn4+dedmkdnm1mhvPW X-Google-Smtp-Source: AGHT+IGQaFV/eoJPD0E5YObQdt63kn1VXtZb+DnEHQpo8u+eLis24QVhJvy1qf6oTqJysoROPjRG5kn0lET0eVSr X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2d87:b0:68b:dbbc:dd00 with SMTP id fb7-20020a056a002d8700b0068bdbbcdd00mr86084pfb.0.1693500978729; Thu, 31 Aug 2023 09:56:18 -0700 (PDT) Date: Thu, 31 Aug 2023 16:56:09 +0000 In-Reply-To: <20230831165611.2610118-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230831165611.2610118-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230831165611.2610118-3-yosryahmed@google.com> Subject: [PATCH v4 2/4] mm: memcg: add a helper for non-unified stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E96221C0033 X-Stat-Signature: swnqctgox57t6a5aqydkuynh9p7idncu X-Rspam-User: X-HE-Tag: 1693500979-46266 X-HE-Meta: U2FsdGVkX1+1vdW08BwR424u9eDdlui1LLd0idUP0CPWDeDavlhTNn/YVxmrjPvpYYiSfbVF6FZ+HcNH/ZExP6t84iui0CWyo/FoijBL8KEVca/zURQlWbKcsNftFHltudzFHWtWyORvtOP+SePpGaIEsKqjDbPXY/XeCihRKceatRoi+krd7F0/duKwQN1UjHcIFsahZ8hhbFwTzTvINpXWVmTva6hiyGs5ofvP/mHiPC6b/k2Kbn04XB25ht2W1FRklzWI/OxR3GZfT2mkuv2g/GXhFcGLbvBywqpoK6zE2dQr+f5Ew0gxdinrxO6/Q6F+e+SvU5uPFJF30J2laAiGlHwL86GV/56W0dYLYAPQJ0hDRTUfLs1tvWv+LAjqfs/eQbnBRYeaselHHKWKFeuhn1qDvybCEhVoKuA/uOwoZ6Bc4hducBSluJnDkv5FCUs3NvO/QX+TWj4U2I/QTIGF8FrHs7jvgCh30tD/H5BnimC3UV0NKJah+jrH/GACHOI5Hi5n1u/8raUH/L/Qh50pTFuDil1FCvrJaOhJXbtrapTJFxO0EJVuUITl5XRiEw0nEL0Xm6gD/FZ5Jh1lNYGU8yf7fS9a0F0m7zjcj33vR0NRAFlqvhwEKi/50gfOfSzfY8YFKcVnx2GdbJ9ZM4aR4FXB1muSsqBiBHc9Bt11VZOBJ6ANOw8LXV23c02KUTLkFHyp0qaabn8emlI5GWEA9zusS4dm62S8z+viSLu/kUc4CI1QA1fzW95OPCWtrpOGp/lpH7NLaQeXJEMT1bIB+NpVeH4CNa6CjoVBsxJsjOxws9mDahaK3Z/yPdS8OcjidBkE01aV4BEz2PiVGR4ZY004ocqmzCgScsaXoVteNA5XnZijTQc/Vmaju7ZAuw48uioy/HK6PoOMMIZNSdvRmXnoFM44TZWhr9Mkp5JpDR0ztXEvOwSvNaCPvfKgJo1YRpT0zcTm6fdYg5e 1LEqYa5x 34oKb+sA/vxfF9psm3a7dgN3KRHOCGqGZPSB5HiUZxVv46fncGYkcAQBI3+oXxPntg8VVRZwGPiub6+Wy0bpSEPHvOCvkmbwI2Hdm8otjm7LD2aTFGTQ59CVVQlshfg5Q2WWqWRxYWWjcGJrdR/otQCKLuSnQ84THMdRvdsY1U2X4CRMPjo/fDBxcfYueRkbW6t5guqazGYAEut4M/Dqfg0PEw/K7tcbs6z9wqTkkUJPbaxEw7Vhi4PpSdykbU7AFWrFTXuBONgBKvh3HpThwooeMwxhOWsCA/XLqjBlWBS4GkiaGQ08TEteRk9dIqRaNR/ExGHebbYOwdvhyLfWO3oUY0lnjX8vbHNfHRZ1Q+ArqXzkI9+G6SOR9RzVdZTrCxZCudUALckZDBctEhF5tiF2H3kFna/rKshSG8ZevldFer/vENGKUh4l/OI2E8BYEUikNCETQ1DH8GTKZj0ZkdgteTlcHYmKdQXHfgho0R0TBA7+2+g0d+1Ei30VXTFAa7Fn4TytPZ3dvbMqhtlf/xvKEXzuc2RNg14lSRgzduG36z/AgAwb37r3HctV8rOODNyXwcqlPWLE8WGYsKvw6HHUuuar9QmTGE/mL04JO2tk5+yrjlKVKlzxJ7PUIp1+A2QZ0h24+svoTZ8AvAcYINhipk/KO9TfRoKJb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some contexts flush memcg stats outside of unified flushing, directly using cgroup_rstat_flush(). Add a helper for non-unified flushing, a counterpart for do_unified_stats_flush(), and use it in those contexts, as well as in do_unified_stats_flush() itself. This abstracts the rstat API and makes it easy to introduce modifications to either unified or non-unified flushing functions without changing callers. No functional change intended. Signed-off-by: Yosry Ahmed Acked-by: Michal Hocko --- mm/memcontrol.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d0ec828a1c4..8c046feeaae7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -639,6 +639,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } +/* + * do_stats_flush - do a flush of the memory cgroup statistics + * @memcg: memory cgroup to flush + * + * Only flushes the subtree of @memcg, does not skip under any conditions. + */ +static void do_stats_flush(struct mem_cgroup *memcg) +{ + cgroup_rstat_flush(memcg->css.cgroup); +} + /* * do_unified_stats_flush - do a unified flush of memory cgroup statistics * @@ -656,7 +667,7 @@ static void do_unified_stats_flush(void) WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + do_stats_flush(root_mem_cgroup); atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_unified_flush_ongoing, 0); @@ -7790,7 +7801,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + do_stats_flush(memcg); pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -7855,8 +7866,10 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); - return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); + struct mem_cgroup *memcg = mem_cgroup_from_css(css); + + do_stats_flush(memcg); + return memcg_page_state(memcg, MEMCG_ZSWAP_B); } static int zswap_max_show(struct seq_file *m, void *v) From patchwork Thu Aug 31 16:56:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13371728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1CE6C83F01 for ; Thu, 31 Aug 2023 16:56:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 785718D001B; Thu, 31 Aug 2023 12:56:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70F8C8D0001; Thu, 31 Aug 2023 12:56:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 587C28D001B; Thu, 31 Aug 2023 12:56:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 389958D0001 for ; Thu, 31 Aug 2023 12:56:24 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0EF521C98C4 for ; Thu, 31 Aug 2023 16:56:24 +0000 (UTC) X-FDA: 81185003088.09.7C6526E Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf08.hostedemail.com (Postfix) with ESMTP id 2F73B160028 for ; Thu, 31 Aug 2023 16:56:21 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=zRkLlJYF; spf=pass (imf08.hostedemail.com: domain of 3NMbwZAoKCLEpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3NMbwZAoKCLEpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693500982; a=rsa-sha256; cv=none; b=mPDAn+JHBlH/bXYNHBkU9RB3eNHuNWXvxYAWwN2apaF2GMtqbhXX0dfevuWM24tL9kyfab yn5TlihvN07W6SjlhYoUA3j8W/Btp0wTqXS5WPbxRACQp2f2pSaKSLs1R5nyTKCtAXm/J4 cWYbZ70y0txvTfAQRUwd2XIq1W+iEH4= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=zRkLlJYF; spf=pass (imf08.hostedemail.com: domain of 3NMbwZAoKCLEpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3NMbwZAoKCLEpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693500982; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dJkwDc7Wxmywu9Yo0ZIK2ppmyNcYkDsKDS48R26U4fo=; b=1n+3i8HaAznoaGTDM8YOTx4Gt70dVNUzHV90L1Nj/hCi55mERJROdvIzNXZfMpkcy9LRmS Yjc11VA2DvaOJmtlYkCYxvFAcqOBL90k8bMK8fXnXDARDK/xVCn5yJuYleQ9F9C1xsdKK3 wIiN9aA5W+1F3SwiznbPi6kPNCeLQ/M= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-26d49dd574aso1090856a91.1 for ; Thu, 31 Aug 2023 09:56:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693500981; x=1694105781; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=dJkwDc7Wxmywu9Yo0ZIK2ppmyNcYkDsKDS48R26U4fo=; b=zRkLlJYFYG57ipyrhkSIPSaqiou5gZt/+2eOf8u/tjgAbYjEVdK3dfkxbi1bUw6QOz LfZXorKuox/MR+BGtcA8EgcVoFBDTHxcwudhpw5+JGVQV4+0rh45dtBkw891PVIVkVnb hE9eoLlIvOB3mQIETYYTirUoB1nN8cSQMx1wFWLLV4MiPmmeHow95God3QmUlfBKNs7Y PF0ZZo8G/itUtHhHOn/RJR7K2VDfVMb7+kvjRCvFB57LNbzUCrDZ159yQPqiYnTX/Pwj 5+SRBy55nrJTT/RoqOVx14MPrsxsmWoG/o95A96KnNRKSdoMBXwwJhdYlwGUgVZGStk9 xz1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693500981; x=1694105781; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=dJkwDc7Wxmywu9Yo0ZIK2ppmyNcYkDsKDS48R26U4fo=; b=d7ayFaHyZ8g7AjdDkiSMYUPsQ/1Za/NpnE8+L75QAU10XqFAp1ifLHF3STilmz8nLL NZwQmuE4fHQAE23D379r8r4u1A0VN5CakN6JPdrkcIX3SJFd0M1naS8Puh42d3xz3WJT iQQjTmqyF/3thsIkbtoIKfq8zlpCp+rFo7F+kWkF87G7MAXSIMdbxNpooOQLcSBpNbOD 5UODAOlpUf2nI7ypDKmyVAGRPrlIO7k+8AFw6ExAWMyr6KHYWLsCDI4oj646xras8QoN i8qebV5J6EuMl09Vt0CjB9yUjJ9G42Se+bPaEuwhJbkLo0Iq0amTBT6PD+lilFiAmB4g k+1A== X-Gm-Message-State: AOJu0Yx1os5FD04lGmathttn37JQJeZhKJhlrAvXDW2JM9ReKeLac/1t GuvH1ifhHOUgwnmViOCDGTBNUEi0c36p5H6a X-Google-Smtp-Source: AGHT+IGBXws1kwasa2qMZ2oV47hWaKQbyN2jJKglhtHqwCQYSWOwpg/OejdryTSoKa2sWR5HEgqm/Qoc3DqRPg8i X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:c68c:b0:268:9cfa:171c with SMTP id n12-20020a17090ac68c00b002689cfa171cmr61794pjt.4.1693500980771; Thu, 31 Aug 2023 09:56:20 -0700 (PDT) Date: Thu, 31 Aug 2023 16:56:10 +0000 In-Reply-To: <20230831165611.2610118-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230831165611.2610118-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230831165611.2610118-4-yosryahmed@google.com> Subject: [PATCH v4 3/4] mm: memcg: let non-unified root stats flushes help unified flushes From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2F73B160028 X-Stat-Signature: ycekp87qfoeib9bt54urcypgsdr1f57z X-Rspam-User: X-HE-Tag: 1693500981-967942 X-HE-Meta: U2FsdGVkX1/CvCA0OXo0/GskqkQGE0D535QgdJ36vr81qM0vP3W8pVG8UihMr6UcHh4A93Qu/QUK6wp5uADaZ1Uce/rY+7VzOsKtMI3bgbkIuWAUsCzvMJIQ6/KaWqDZVgSXeqKIODBkW+JqrdLBUdjOWxfCXpm96oUR0rbEswwiAFa5S6jSH2TH7LoCq5NwnV1tJVP8pyxQtXPxP+lUcIGL+w2ZW2Tnuisb8WXryCcMptINwR5qLJKknFTyVAK2FCwJqI3pcBUYpKrPKdLXa/38vH5I05O3jU4udNCF4SAgKszNuKOZcIi2FRCylEacvrI1NezC+4Iq4QVudUfD/P1RwRAEmaGlHA7blKBGuA2xy4Cj/vrU2zoO2AlJUnBBzuQ9DUQSWw5h4j1OLGzIO3I662PdxJGwGc/F/8UEj0qnwlIjRWCRtd4gtC5cc5mTrRNrjIaQyE/ahXS5FJ3nSMNFAzceP8nJmYPobFuj616TOTMskxvMh4vlJ72ObRBR0sUujMrTOqvpUwcehtsQJgWvAMpmSmuQ8w01c8rUVQM0kCsSV2rOOflHi4w7PNVp27K22ljcQ97FkTacVyy4NLYWpoQ7FiI5PSmBE2nEBOIrWbb7P/5uZFY6KEUdUFo9jN4DzPNlPU9nO8bpYxPuGe98FcWoFcBjem8VZw/oesCXebdlxvF94SfBmQQ02h6SLoBbVdroWeUqnRidetw43hYxGgRGJ/Wd+lVEK+UOpjb4F7NdJKwBpZEKMrHHftQCvkgV/dCdQOcmV5CINcwseRkkOEXTzi/DZfhZT2quI6gv4J8GxuJ2Z5/pJhDkayGmRuLwytY3eJX+j0yk7JwjhkMSfihl4XoGspFiC+K/2s5/qmlEBu7HxuRBluqg8rfa1tqKslDSsmEC5xkHWOMRIZbi6Suz805Q6RFYe9KFp/3z1RUAwfpfF4V8Z90IcfO5LPCho8FxesUMgcOW0yM yNa6Ptid S/hMutzfPVqwfEXwLlhUewoygPkXJGXW0qyfHJVYLaZD0ln49gLsokTxNwPF2W/UWDYer0egTFUJBMUr4TlAFk89RE2KH+M3xsXQ9/d6NTIyhjUvX8Uh3ull2AmFew5PsJjJhYa/76+uVg5ES639MafJEtD/pbZxtm4lcWod9BzKAwljlBWnDrv3Fj0fjQTyFaSGzz/Bbe9k9MXT2Ui8+1mlz7OiaJjRESYQ/Jmm34oeSaXQDWGIgHK8O72rZ4jndIuzAx2syZM2BOyKcW/oYiBvAy/Eg7zSO84VsDwkJ4zBg907TBGsHqNYXhepxExVkxXTevOhi6eGF62cAmNdYgDuF+84oDO4nvKRy2sjP72kwyF0wp0iIkiKDy2P7vGDujZA7yBBl9VR0gzwZkmb81dDI6TLJhi13cx3llDgXghK8pHZxTts+Rd9ncCzWx7hXTGlCJvYvS2UvqFqOl29YYx1WMHIMcPWMlGKFqH0e0zhree8lQLkw2StcaplkK2VotmRTc3KvTOfiSzClyIRVANYO+I0uhXjWGMpvXE/H+U8sMGWak3AZ4SnqKZZYD6CzIncLZ4MpMUt0jwa1jp5Cp4ulBtUvewA+PCUos77AvLm366p7vkcCzPGITxcpFoxa606jiAtURgGxeFaNwTpPmj/PYySySEZe49lCCBh8stb/ABZmdIIiCUC8R0lcKxoNbVHoyh4ug+4FSJk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unified flushing of memcg stats keeps track of the magnitude of pending updates, and only allows a flush if that magnitude exceeds a threshold. It also keeps track of the time at which ratelimited flushing should be allowed as flush_next_time. A non-unified flush on the root memcg has the same effect as a unified flush, so let it help unified flushing by resetting pending updates and kicking flush_next_time forward. Move the logic into the common do_stats_flush() helper, and do it for all root flushes, unified or not. There is a subtle change here, we reset stats_flush_threshold before a flush rather than after a flush. This probably okay because: (a) For flushers: only unified flushers check stats_flush_threshold, and those flushers skip anyway if there is another unified flush ongoing. Having them also skip if there is an ongoing non-unified root flush is actually more consistent. (b) For updaters: Resetting stats_flush_threshold early may lead to more atomic updates of stats_flush_threshold, as we start updating it earlier. This should not be significant in practice because we stop updating stats_flush_threshold when it reaches the threshold anyway. If we start early and stop early, the number of atomic updates remain the same. The only difference is the scenario where we reset stats_flush_threshold early, start doing atomic updates early, and then the periodic flusher kicks in before we reach the threshold. In this case, we will have done more atomic updates. However, since the threshold wasn't reached, then we did not do a lot of updates anyway. Suggested-by: Michal Koutný Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8c046feeaae7..94d5a6751a9e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -647,6 +647,11 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) */ static void do_stats_flush(struct mem_cgroup *memcg) { + /* for unified flushing, root non-unified flushing can help as well */ + if (mem_cgroup_is_root(memcg)) { + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); + atomic_set(&stats_flush_threshold, 0); + } cgroup_rstat_flush(memcg->css.cgroup); } @@ -665,11 +670,8 @@ static void do_unified_stats_flush(void) atomic_xchg(&stats_unified_flush_ongoing, 1)) return; - WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - do_stats_flush(root_mem_cgroup); - atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_unified_flush_ongoing, 0); } From patchwork Thu Aug 31 16:56:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13371729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61450C83F2E for ; Thu, 31 Aug 2023 16:56:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA4598D001C; Thu, 31 Aug 2023 12:56:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B380F8D0001; Thu, 31 Aug 2023 12:56:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 933F98D001C; Thu, 31 Aug 2023 12:56:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7CC8B8D0001 for ; Thu, 31 Aug 2023 12:56:25 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5970E403A5 for ; Thu, 31 Aug 2023 16:56:25 +0000 (UTC) X-FDA: 81185003130.25.DB12B81 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 6443280016 for ; Thu, 31 Aug 2023 16:56:23 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="tz39G/2E"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3NsbwZAoKCLMrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3NsbwZAoKCLMrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693500983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lVjKd06WEqwLUbomPNwpng6u2PnrisAzbeEHsBpSLQA=; b=ogMBcUVktvkMYkGirP1NT/dUGRryPJUFn64NJtlpbHsS4Z7oIIen8NHyoCB6UmnXE8EkDN JJBRMaW2wCkeRbWdGYSSY2U4WA3KviXNuSM/KAIQ7rAj1yfL/TcyfOFVTr7CNdiquScC+O bMtuYCSkal4f5W6hlMDErYDEuskt1S0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="tz39G/2E"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3NsbwZAoKCLMrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3NsbwZAoKCLMrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693500983; a=rsa-sha256; cv=none; b=nE1hmWs/9+mQ8LpCjo5ijnga0moIKQQq9DX/+5mUR+Y0X75rQzJ7xU4aQeB//raFynZX7p B7tmti6Lrq4CBL31YaaAuF5IXiwN2cwsOty20rLy5K6CZirnAadZqkxpqGJGe6Svpw48O5 hULWGhwisIjZ0CkURFxDyofRwBxy+7I= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d748b6f1077so899129276.1 for ; Thu, 31 Aug 2023 09:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693500982; x=1694105782; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lVjKd06WEqwLUbomPNwpng6u2PnrisAzbeEHsBpSLQA=; b=tz39G/2EFfFqPC7xETaRyDIHdldvN3VUhOkdGeY2MQhASDJF+Ug+VvWv+VtzkCdC1q priEAgU/sNtnsufIrzgcbHFtPSwwmkPtCM0Wh3jYn29WagIRmNT1rOB3/3Mf950jKLz0 wBffDKUCvmST+A4jPW8bNP08JREYrCnOPIiuZyijoT09f8qhpAXVRSO368qISqK6Af43 /aVLTxtxE1CW5vMQiXI5WJDQg8SEJQNqV21qDfmziLkT2nAftTg7SwLqqjP6MtGlXwsn 5edcHjKTztsyfHnEE1LNq+78wgDaBFlgU34e4ZXZNqQCFnJrbyoUjee4B974pvODXcHl o1XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693500982; x=1694105782; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lVjKd06WEqwLUbomPNwpng6u2PnrisAzbeEHsBpSLQA=; b=IhLrP7XwLbqrd+PjQ08pe+gy+BiI2t9yMz8TV8REXfY2kO3R9VbAkb1WM99TnPdSwB sOY+kW5bVYPweBSCqFlJ6z5KlWnh1qZqAvj3TuMLKA2/aXt2zVIX2QMq4fxfEnASwMek WKR4Gix0w/MkGchW/hw3puUjiBHZ7fqtojvN5lge+0Oz0lIXtCJ9297gJ0cTzhrU5V88 /qE00CO/quo7ODwgFEqcBjZug7mnGtmOjJD+qpoicwloEt5sEPtC/72tirJHvqe0pjLM 7fP+Z/49y8a7SAUZrAGgCzCfMCJLSrlYte6Q3guT1vNYGf9zFmqtASmKDvHfew+WtGMK 0pvQ== X-Gm-Message-State: AOJu0YxYPKF0+6dyiJvr8asN0Wot5LuyvHcUONZ40YYoWXGu8vuCSaDh Ya5DcULf5qQljoJUPwU8p32ceb8czmvz92zm X-Google-Smtp-Source: AGHT+IEy8G064MlfVtEPW2U+DbL+KGFNApMore566FGW/RELtpuRVjTr6mp/8DBxtEe5U/4TTDMHu8xHlFoQIToU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:ba90:0:b0:d72:8661:ee29 with SMTP id s16-20020a25ba90000000b00d728661ee29mr6359ybg.2.1693500982588; Thu, 31 Aug 2023 09:56:22 -0700 (PDT) Date: Thu, 31 Aug 2023 16:56:11 +0000 In-Reply-To: <20230831165611.2610118-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230831165611.2610118-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230831165611.2610118-5-yosryahmed@google.com> Subject: [PATCH v4 4/4] mm: memcg: use non-unified stats flushing for userspace reads From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Stat-Signature: xrft5wtnhhu7cjdp5jx4xmbshsamkrd6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6443280016 X-HE-Tag: 1693500983-472860 X-HE-Meta: U2FsdGVkX1+6xsU2Lfe312BPdcx7XUBDcIqr9qu6DhccJUAYhmEMVz9GbH7makMimKDsZz1UnOEKgyo+T0gdHP7Cr5Ig6uSJnYNEezHLPI4eYBg6rk8uanJb2GL+o6ZwEWdugNQUNnOsnOt/UHv7XV31oLoesE7Jf2jvx5qCVnmSwNDw+3q4iIMmwrtOMJpaL5eJElQ4biEhZGhRRY+hjpHIHkuY/WEVHzIPhAokV/oSQSMVP8pQkCKvxx1lv1W2DKJZdk2QXmxcS1c7YfLVhAJoLHcKgL2WEyYEwmQBNUe9/ibbEQE5rl4raGyYhZvYYfY20X1tbDFYSTDOZ4WW3m0+bwOwI8uzGl0GqDBGdXibmcmJHRTB0jyqLXvohWdtla+vn6id/P+mx7JsrLhRi/eskN1Z8wq8n+VAAu9PkjeSpQavqevswUYWoo8cBcBszzfgM2vPUynXxuSYl6p+FzyDpHnKYbdvJ9M/ZvXsJXzfpEg8Fdv/4ZgUKIKrexWSXNQ8HiCanGpQBBMckcMZDVCocY277BJX5wLmXs8dZGUZioyVM8VidH9pZwepjsy2R+zcYEAnKFeop5o7D0I8pckZY4521Ti3SGc7V5Bym1dhWRePV6ixhysxiNWE6DMk1D7V2CkPUE0OXNIkaBpUMIQSOAXrCXyWHWAAS8ZwKt5e6vtUM4aFdKf3S+jTosEUiApRByv7cYXPvm8PZGt2qNkFE2Y2OigX1ccR7cQkzer8qgX6mmJ0ls6DpcKolV0BZUzhdesX8dP319CR31rkfc1JogcVRI29Z6amfYBmpLHzWc/IrQx2iTxz0Hk6TmBPPekkBauTn6iAxAqN2Ninb4j9a4M033DasFHD6hJDQMAXIi7VGaNsZYsPKUGD01ijwnfIDiEDurRwuKSnyXy7AL/RbqXtVa54RnK/Z+mwmrstL5pJAOt2irC3poGC8XfF7SK0z/hiPeb6NIvb5kV trxKXSl3 b3lgS8X6mx0bJhxHTT9h9FlIKtAU8SDCPT5RLdJtj0FmK180LZXYtyveAWyECqOOrTlkWT0vPgtNJU32/pdYL/v5xZIkNxzLKmw02EYGWNwqqTTXsT6HK1KeIZF6YuFX3qV6TwuOgs0QcI+Zla9g1CwLT+aYQGgvRFnfZLEuvP9KKwVTSXfvZukdO0jiKuZEFKLUXPMW0uqp5xB8aOuvDniZa6kzUbVRhYVbgSPwxh8NIk+B3NOm4j9PV0Lel3OuEXPbdhmlbuDiqO6oPbHWsaTreu0lNxt/v61ULOppfTsqokEyjlyw3jztnoUuX1c9uhV1PjOchKDabVIKHHZ7olK2mUVho6Q7gqnZJL2sTGY8N1HeDOiGX4KH5lfIEz2057zdcjrZf2/Y2JDMwyMf3/HOVz2aAAkxy0vcafr+7rBE0YmYZ1Rq1iF1j57nuVmfQ1yNNe7Lim042Yqy77MuWKbINArL/pdyygPZaT8k8u8/UO/PfI8APlcOJmI5JWPcR00vGdBiNUQEmf++M5b22IhN8cWWtHaRBcOI3URmaSxcpdhgvaqTvIJ6viljhm6f9n7IalZ7iUpLvtH3R9md6ACvPgIKr0uWrRZitEWYCdjzu1k15tq3RCpGjQunXvDdTyBc7I/5HijuDkTXKnJ8luMmY9vkmU1tv2L29eR0giLBR2C9mN7zsnysIb6E2ZIb7fVbPzn04lR54/4/+5cZ3+WdioAYxYeBCOhmp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unified flushing allows for great concurrency for paths that attempt to flush the stats, at the expense of potential staleness and a single flusher paying the extra cost of flushing the full tree. This tradeoff makes sense for in-kernel flushers that may observe high concurrency (e.g. reclaim, refault). For userspace readers, stale stats may be unexpected and problematic, especially when such stats are used for critical paths such as userspace OOM handling. Additionally, a userspace reader will occasionally pay the cost of flushing the entire hierarchy, which also causes problems in some cases [1]. Opt userspace reads out of unified flushing. This makes the cost of reading the stats more predictable (proportional to the size of the subtree), as well as the freshness of the stats. Userspace readers are not expected to have similar concurrency to in-kernel flushers, serializing them among themselves and among in-kernel flushers should be okay. Nonetheless, for extra safety, introduce a mutex when flushing for userspace readers to make sure only a single userspace reader can compete with in-kernel flushers at a time. This takes away userspace ability to directly influence or hurt in-kernel lock contention. An alternative is to remove flushing from the stats reading path completely, and rely on the periodic flusher. This should be accompanied by making the periodic flushing period tunable, and providing an interface for userspace to force a flush, following a similar model to /proc/vmstat. However, such a change will be hard to reverse if the implementation needs to be changed because: - The cost of reading stats will be very cheap and we won't be able to take that back easily. - There are user-visible interfaces involved. Hence, let's go with the change that's most reversible first and revisit as needed. This was tested on a machine with 256 cpus by running a synthetic test script [2] that creates 50 top-level cgroups, each with 5 children (250 leaf cgroups). Each leaf cgroup has 10 processes running that allocate memory beyond the cgroup limit, invoking reclaim (which is an in-kernel unified flusher). Concurrently, one thread is spawned per-cgroup to read the stats every second (including root, top-level, and leaf cgroups -- so total 251 threads). No significant regressions were observed in the total run time, which means that userspace readers are not significantly affecting in-kernel flushers: Base (mm-unstable): real 0m22.500s user 0m9.399s sys 73m41.381s real 0m22.749s user 0m15.648s sys 73m13.113s real 0m22.466s user 0m10.000s sys 73m11.933s With this patch: real 0m23.092s user 0m10.110s sys 75m42.774s real 0m22.277s user 0m10.443s sys 72m7.182s real 0m24.127s user 0m12.617s sys 78m52.765s [1]https://lore.kernel.org/lkml/CABWYdi0c6__rh-K7dcM_pkf9BJdTRtAU08M43KO9ME4-dsgfoQ@mail.gmail.com/ [2]https://lore.kernel.org/lkml/CAJD7tka13M-zVZTyQJYL1iUAYvuQ1fcHbCjcOBZcz6POYTV-4g@mail.gmail.com/ Signed-off-by: Yosry Ahmed Acked-by: Michal Hocko --- mm/memcontrol.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 94d5a6751a9e..46a7abf71c73 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -588,6 +588,7 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); static DEFINE_PER_CPU(unsigned int, stats_updates); +static DEFINE_MUTEX(stats_user_flush_mutex); static atomic_t stats_unified_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -655,6 +656,21 @@ static void do_stats_flush(struct mem_cgroup *memcg) cgroup_rstat_flush(memcg->css.cgroup); } +/* + * mem_cgroup_user_flush_stats - do a stats flush for a user read + * @memcg: memory cgroup to flush + * + * Flush the subtree of @memcg. A mutex is used for userspace readers to gate + * the global rstat spinlock. This protects in-kernel flushers from userspace + * readers hogging the lock. + */ +static void mem_cgroup_user_flush_stats(struct mem_cgroup *memcg) +{ + mutex_lock(&stats_user_flush_mutex); + do_stats_flush(memcg); + mutex_unlock(&stats_user_flush_mutex); +} + /* * do_unified_stats_flush - do a unified flush of memory cgroup statistics * @@ -1608,7 +1624,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_try_flush_stats(); + mem_cgroup_user_flush_stats(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4050,7 +4066,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_try_flush_stats(); + mem_cgroup_user_flush_stats(memcg); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4125,7 +4141,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_try_flush_stats(); + mem_cgroup_user_flush_stats(memcg); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -6642,7 +6658,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_try_flush_stats(); + mem_cgroup_user_flush_stats(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid;