From patchwork Mon Aug 21 20:54:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13359806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E0AFEE4996 for ; Mon, 21 Aug 2023 20:55:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9DCD940011; Mon, 21 Aug 2023 16:55:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A27138E0012; Mon, 21 Aug 2023 16:55:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C82A940011; Mon, 21 Aug 2023 16:55:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 757868E0012 for ; Mon, 21 Aug 2023 16:55:21 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 412814035E for ; Mon, 21 Aug 2023 20:55:21 +0000 (UTC) X-FDA: 81149317242.01.D11CCA1 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 893AE8000D for ; Mon, 21 Aug 2023 20:55:19 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gDjS77jh; spf=pass (imf30.hostedemail.com: domain of 3Ns_jZAoKCJEJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Ns_jZAoKCJEJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692651319; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wvdq61L+aFGdt8QKKpNnJHBdE39S0UeYM1yvvHjD3xQ=; b=C/z1nFpShr4RYyrCjosr8B/wnyM83Ms+uvFVyOa9lCNw+7toc/kdRlLoNoK2DiZLVBbDZ1 sGBDmzJBc39a/lD4yYC89I99I4q3br3HfW+WwHu1ZWvkbZsToBSBbaI4SEB82JsuhKzd8c YAUXa94+7O0caPUxH7gD7o5JPE2WxLo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692651319; a=rsa-sha256; cv=none; b=2FsdF/t3352k86RoAPSU3ZRfMMCucdeAE8FFh/aovYtoylgLnyM6Pln9lacukZTok6kou4 4N4gfsA00oCXbio36e0e4dMjdwpAdAObNOpAkgMIeeGPO8WhqFS/2UERFbKibSkMGlvoP8 /APPvNMq3iipyjjZujRLJKh6HnG6/1Y= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gDjS77jh; spf=pass (imf30.hostedemail.com: domain of 3Ns_jZAoKCJEJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Ns_jZAoKCJEJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-583c49018c6so55880657b3.0 for ; Mon, 21 Aug 2023 13:55:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692651318; x=1693256118; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wvdq61L+aFGdt8QKKpNnJHBdE39S0UeYM1yvvHjD3xQ=; b=gDjS77jh86r8ZKs1/zKg7Cco96sopZmFLEz/zj9xB/n1q5vx7rO/YHELKGS+X9Ug4U YpGS8y0oXx1+lQOuCRWO4hJaQzYd+MTroVbrUvtNf91m8TASXjyJIAiKKFjFs2SvbEMF ETcYXj5DRyAg6uMI1Tg+PNCmWCLVT4wh3IA7ElB0FC39nvwl0DxqHUQMVF3L6VZeLxWD Igvb6FKlgK7kbWdQtRC2tXSUf35syDuAtXUdPI50X7vL1Y+pprzowUj3JjDeaCVIeOGE KhQl4U7drhKa+h5IQMVK0nmZmAfMy/Kou0T2fowQemhM6iwR2fCYPTNirZnXMfkhAzuf 7vZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692651318; x=1693256118; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wvdq61L+aFGdt8QKKpNnJHBdE39S0UeYM1yvvHjD3xQ=; b=VQcQJ9TfEbuEMMSZ4S+cyCWV+w6WmIXo/HF7FtevcrjkuzxMYieOq6MtrNPHgqGfeT PhRxNYTeJBg4CWRz7NXsT2W+vx7fISIBXIoK8d8jYmGntriZlbGe6WRdf7cCG3nPLDhQ AR6pVbwLCZEktc8FFDwuQbH0w9Hcra5kkt2Cbu703V4NuG1zlBCyWhyAUyrUXPNiPubE pUmr/jSRxnIuRp6DzspqbbnS2FsoBKebMFUbY250S8CkUa1OeTPOrBMXNh/8LDTKuc/A R84+Kx1ZbDGUWc2UqllOmiWhzNvh3ImdAmDJAIpRLqeHY/gct2ptjSxyhPRCrPpRYhWU X9OA== X-Gm-Message-State: AOJu0YzNZv9Vsmk3B/P+r5eGDTUmZafuXdeOGm/HH3ShazMRJOKzk5Xy FXNttOAQ7mFuFIF5tdvqMkg5YcyhFXyG3wDX X-Google-Smtp-Source: AGHT+IHI17WgDLA9h5213qI/4QCn9pZF+BZYijIyegDPoTXjNDSTPJibwpZtlmJmIAbTjzp62J1FhYw7o02Y6LmV X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:d604:0:b0:cf9:3564:33cc with SMTP id n4-20020a25d604000000b00cf9356433ccmr61637ybg.13.1692651318564; Mon, 21 Aug 2023 13:55:18 -0700 (PDT) Date: Mon, 21 Aug 2023 20:54:56 +0000 In-Reply-To: <20230821205458.1764662-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230821205458.1764662-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc1.204.g551eb34607-goog Message-ID: <20230821205458.1764662-2-yosryahmed@google.com> Subject: [PATCH 1/3] mm: memcg: properly name and document unified stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 893AE8000D X-Rspam-User: X-Stat-Signature: rb4ddqsf586iqbf13chdwbwr116mm4ze X-Rspamd-Server: rspam03 X-HE-Tag: 1692651319-977381 X-HE-Meta: U2FsdGVkX18TyIkaMmtQxrX0v7CviUJejKRzT/uDPQVps88Zd3KTN92gzr/KN82iejXb//sBxhwUPIMgKwAjqx290YvqBXcCSPbfRKW+L9ZsuQj8mr36HGwR+MwFrWcWksDw0FSsfKlTVj61UEgRBCn4i6jgwtCTSIUelalKwDSv+bKghsYpBntP83sJA0p1plTo1u8t02doQKRMn5wuZ7gSMMuIHhPQrI/lSHoJot3RKum1RgOT/ko7sP6kEztb6dYf8NtWDkpWwoZjHhtLcVrl/XqJAwde+l5egikIVR4WyJIjRowp8i87v6uIPRrDeqUvL8bJuJPgkHMXZKWV9MfRRqbzWG8/zRbrd+K281vQomKlNP1GUiHcb0pKgPDREZvC0sfc1E6uCSgA4lBHVWnsgfcJreP5WcamYNiXtdTwqbodGtIAAlB0GEUH0hrTHRK+0MLyMT7wrJFZKjaHtCr8qu5ypjoJClrl9XfoP9mFYYZao4z0CBtbyzvyrl0YmIb+AjxQzJZVF1MW3wFQQ46+IlwVzR+RNeHnKCCekC+lg+OEbk/x/7RRT69GzIYUBYszjnZ+HdwxlO/Y5PPJkiWYGQvIK9lC9J7Nl3aibmOy7A9bA5XIhjxQ+wVjK81aqOphyyKDq0Sa4XAwTOrvnivtT5V35Wo0akhzZXlPXTDP910H5XcxdbHFBckDr7s9TcMf0OyuCd5HiA/WWJ9g/Cisnrf1Q74Nb8VUggJZXv6lGnmKpRhUNxt9LOr+/tcnSU4SBENsaFW63WncnO8e7puIV2ap6At4lNAv9bicuwXg6WRLGUaEifmVF/0zULerhJrjSj70yF9nPg6XitZpYY2TzahupzcTtmb4rQb+/cyE4Jd6zPHBTkVSGbkfUBAuhPYl3sDgAoPqA7VeEPrr7h2a7b0aBQTsiNqS6e92V5YXiBKXs8ppwrsri+O0pF7ywT1gMSpgtUgy7zHIoM7 0L5zSXE1 lcp0WTX2EObpQTt002B/B33nGcP0cFHiWt/zOQ9rm0KeTnY0t3SuXVfsml6XvH/Q3Y/TFS5HIsX/Z/CpehDugORGaTxXb+xF+YobkRsdpTNA8tajvA7+p/Czt/cyudME0t+tkU/nXhebwWRRmJ2U2b0LxK/aZFF0YdMxCrRJ1TM9U4r93NTO9sFHlfoxiz4t7XcPeRrbF7a7Ry1fq0FO5zWLIdjZQLPHOA55HYboELDEhG3Xgz/zFSygQGTz6eZMn6oF6v8Bn+nZYAKSfNj3lesT7gsynkT9Df2VJtjNX3JPcxFYsb0LsitNDQaDYwh1Pxr29EmXfJFVsTsJnStSd6fOAuNPYhY/aIto9YviMAEoUflAFmZUh94UMvi5ECig18Jrc3a9O3J+XvWdC3g3NrS5uek5UNOG82P0JkVh2j1qi85sPKYzKr4B/vz/hkdJKtNA+zDdJM2TxJGPIqa25c2MsvmOyWwn9EDvV/SmUEpVh9xUCSblEdroL6Cmz9kPtmbX7BfQxUynUxz9oRGn8rYM4ULB6Y1Q2WCpgulgbtqp2ZbIc0GXg88OroVFIluJhoR07xBpTSuQ2AMpo6kRbRzMr/8ijyTJgouHs5VVh1dTaH5YSe03WRCT2U4SD1qMcPXJHAftWnFqxj3g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most contexts that flush memcg stats use "unified" flushing, where basically all flushers attempt to flush the entire hierarchy, but only one flusher is allowed at a time, others skip flushing. This is needed because we need to flush the stats from paths such as reclaim or refaults, which may have high concurrency, especially on large systems. Serializing such performance-sensitive paths can introduce regressions, hence, unified flushing offers a tradeoff between stats staleness and the performance impact of flushing stats. Document this properly and explicitly by renaming the common flushing helper from do_flush_stats() to do_unified_stats_flush(), and adding documentation to describe unified flushing. Additionally, rename flushing APIs to add "try" in the name, which implies that flushing will not always happen. Also add proper documentation. No functional change intended. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 8 +++--- mm/memcontrol.c | 53 ++++++++++++++++++++++++++------------ mm/vmscan.c | 2 +- mm/workingset.c | 4 +-- 4 files changed, 43 insertions(+), 24 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 11810a2cfd2d..d517b0cc5221 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1034,8 +1034,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_try_flush_stats(void); +void mem_cgroup_try_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1519,11 +1519,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_try_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_try_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cf57fe9318d5..c6150ea54d48 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -630,7 +630,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) /* * If stats_flush_threshold exceeds the threshold * (>num_online_cpus()), cgroup stats update will be triggered - * in __mem_cgroup_flush_stats(). Increasing this var further + * in mem_cgroup_try_flush_stats(). Increasing this var further * is redundant and simply adds overhead in atomic update. */ if (atomic_read(&stats_flush_threshold) <= num_online_cpus()) @@ -639,13 +639,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void do_flush_stats(void) +/* + * do_unified_stats_flush - do a unified flush of memory cgroup statistics + * + * A unified flush tries to flush the entire hierarchy, but skips if there is + * another ongoing flush. This is meant for flushers that may have a lot of + * concurrency (e.g. reclaim, refault, etc), and should not be serialized to + * avoid slowing down performance-sensitive paths. A unified flush may skip, and + * hence may yield stale stats. + */ +static void do_unified_stats_flush(void) { - /* - * We always flush the entire tree, so concurrent flushers can just - * skip. This avoids a thundering herd problem on the rstat global lock - * from memcg flushers (e.g. reclaim, refault, etc). - */ if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) return; @@ -658,16 +662,31 @@ static void do_flush_stats(void) atomic_set(&stats_flush_ongoing, 0); } -void mem_cgroup_flush_stats(void) +/* + * mem_cgroup_try_flush_stats - try to flush memory cgroup statistics + * + * Try to flush the stats of all memcgs that have stat updates since the last + * flush. We do not flush the stats if: + * - The magnitude of the pending updates is below a certain threshold. + * - There is another ongoing unified flush (see do_unified_stats_flush()). + * + * Hence, the stats may be stale, but ideally by less than FLUSH_TIME due to + * periodic flushing. + */ +void mem_cgroup_try_flush_stats(void) { if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - do_flush_stats(); + do_unified_stats_flush(); } -void mem_cgroup_flush_stats_ratelimited(void) +/* + * Like mem_cgroup_try_flush_stats(), but only flushes if the periodic flusher + * is late. + */ +void mem_cgroup_try_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -676,7 +695,7 @@ static void flush_memcg_stats_dwork(struct work_struct *w) * Always flush here so that flushing in latency-sensitive paths is * as cheap as possible. */ - do_flush_stats(); + do_unified_stats_flush(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1576,7 +1595,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4018,7 +4037,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4093,7 +4112,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4595,7 +4614,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6610,7 +6629,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; diff --git a/mm/vmscan.c b/mm/vmscan.c index c7c149cb8d66..457a18921fda 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2923,7 +2923,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index da58a26d0d4d..affb8699e58d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -520,7 +520,7 @@ void workingset_refault(struct folio *folio, void *shadow) } /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_try_flush_stats_ratelimited(); rcu_read_lock(); @@ -664,7 +664,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, From patchwork Mon Aug 21 20:54:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13359807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35C22EE49A6 for ; Mon, 21 Aug 2023 20:55:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFDDF940012; Mon, 21 Aug 2023 16:55:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B86588E0012; Mon, 21 Aug 2023 16:55:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D926940012; Mon, 21 Aug 2023 16:55:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8850B8E0012 for ; Mon, 21 Aug 2023 16:55:23 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 61AC7A0CD7 for ; Mon, 21 Aug 2023 20:55:23 +0000 (UTC) X-FDA: 81149317326.21.9F91D66 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 98A8C8000A for ; Mon, 21 Aug 2023 20:55:21 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BxE45txI; spf=pass (imf02.hostedemail.com: domain of 3OM_jZAoKCJMLBFELx49103BB381.zB985AHK-997Ixz7.BE3@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3OM_jZAoKCJMLBFELx49103BB381.zB985AHK-997Ixz7.BE3@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692651321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EWXap7NsEgOymMxjxsCXNzpaZzVX2vYEXhkkwglYb2w=; b=KZyI+LF6R7VBlYj43A2c+DLBj14YNCcW4LW24INgkf0e2IVOWQsw6i4rGNykGsDwJgOJj0 6cfFy+xT0y5pJE/xJAiztxKciRAy5mVROqP7xhLanLAaIpmj1Vc/fWWt/cw3WaywVvcX7I lQA3BmSgYFTqREIwATw/ov1SOJ6Lrgs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692651321; a=rsa-sha256; cv=none; b=zaVd6YrLRUGzdFpdMjEp7aa9eMN4JzQP9UlNSxFUmixhlIv7Hk6fchCSODbDhPhYtwZ0lM toREfDhmJK85qV7RUSScisxtmsPaNJnVHsvVspdiqGk0CFktyKrOeEnHLZ1P+0qPZDyde2 5le/SiO8FWLAAH1kwKWJMLOyJ796CXo= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BxE45txI; spf=pass (imf02.hostedemail.com: domain of 3OM_jZAoKCJMLBFELx49103BB381.zB985AHK-997Ixz7.BE3@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3OM_jZAoKCJMLBFELx49103BB381.zB985AHK-997Ixz7.BE3@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1bde7998451so57389525ad.2 for ; Mon, 21 Aug 2023 13:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692651320; x=1693256120; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EWXap7NsEgOymMxjxsCXNzpaZzVX2vYEXhkkwglYb2w=; b=BxE45txI7COiK2llaPLekWLKmnwKvbfF2s+ErO15DE6ofDgcY41orJSnC1lxnC5xSD ns0NI3CFDeBKOgHffDfz8J+snMVDi/Bm1Z7XQP+gB/BJurXrc210wa7+HakGaRIeyfrR gkyrBvxIdVL0w6/wEiEjuj80edhWhBOarSLoMLG6xilaOxakqnBgLMne6hlBqjVYfs4v ENXPjTfI0g332oAIouzQJ4gw4Eqbnd43rlLTg2T4GFT3PPeE9f1Qb7UfSFQpFlOSlXs8 BweEjiJ9C0STVkzlpyv7rd/z4Ep9F8B9BTFfyXqsH3AJsnekq8bnFOZfOryTXvrS+E+8 DCMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692651320; x=1693256120; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EWXap7NsEgOymMxjxsCXNzpaZzVX2vYEXhkkwglYb2w=; b=lnwG3YSN/RamYEjex41KBGXijStYM3pXaKJwT6VBbdncTr6oF1T/qbfCZOq31noXbQ gI5tLXu/JJfD939QOoGCSJJUoRjQjPITu6SvToJD5RCnEGw3Yhm6FJ5Yc+KLh29HzmAn 4U96AxG+rGRnqbikSQ+O3e7GkCJx02Q36km30ZOFMmdBo8PLmt+Y394sOLoCkhqDlYw5 5/8PCebS+lRiLdKYjjJy7jL1fEzE6LhiWFaEBY+29VnvDJdZ2bU06O7i9leL34apPMbF vsbv9fiiDe2N2N72iX7HxtFTuftFhEv56WmhJIHMWTUKm3hTZwKpGJPiwEw2n+on0LaN 7CdA== X-Gm-Message-State: AOJu0Yyw9X5mDGnPmCLzpJ05wVjwXOvEPt4hq7/88hw33ZSNMdwWtHgF 8W3VQKXnWeggIOHHCCT+6n+JrL/JcgwWhOM8 X-Google-Smtp-Source: AGHT+IFrW4DDu5hPoWYHSg5YRuSc/PnkgVYk/IScMtefDrv62gvp7vZ2XxWBYhOW7vd5z08FD5JBCBz/m5ZQr4DG X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:903:2806:b0:1bc:5182:1ddb with SMTP id kp6-20020a170903280600b001bc51821ddbmr2949941plb.3.1692651320527; Mon, 21 Aug 2023 13:55:20 -0700 (PDT) Date: Mon, 21 Aug 2023 20:54:57 +0000 In-Reply-To: <20230821205458.1764662-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230821205458.1764662-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc1.204.g551eb34607-goog Message-ID: <20230821205458.1764662-3-yosryahmed@google.com> Subject: [PATCH 2/3] mm: memcg: add a helper for non-unified stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 98A8C8000A X-Rspam-User: X-Stat-Signature: p1hfzbpccbdmkffezqmxpep9pe3piaht X-Rspamd-Server: rspam03 X-HE-Tag: 1692651321-259847 X-HE-Meta: U2FsdGVkX19kxw1e0o+Nt6xFZzeG/uGsxgInU3sGzwVbA8AUJGI3NTM3bpPm/WaXwObDk88yqfiYfyFSXklS1H6foklz+nofmUmcpTGCD+kBOEZwfBAJLGVR8DFSQZSsK+8zdFLHshsXdwSCrJnmpBacaGGQuWi+DgsEELx3D7FtmY2DbNxtliBE+F4z6lLQFATt18d3B9Gp92hOcEesyRseS5VwgqubJHKM0fHdAe45gV0o5R7NG2pKwRxcZG3wmxXAOojBayp0ZNbAm325guVhYvhUu6ATV9rhXuI8J5NKJ1eXwYbI594mGEynJMm1M4wrrjX9Zgo38ySL4HKTHLy5jbcwZ82RVUso6e4MD5Bm6lvDv0wQhi6FW1pBCBcYyk6yMauiXpBkLnUznc85wL1eZDq4ujb04Ab25rkWy9LBvpWnCQQvhdyfoO06k24WsJVSdXXDhuzwGKgA5bnDdnoIrZdnuj8X5KsuygoHEFW8H0z97RiYFT+txBNapcvh0B1EdRIR9RCs4cB1SWMNVk45NvjrE+wXI2cAkvkTXN15j1Tj4RhXUFLhDYqy0PmpE6N2gLvAu/np1/m8W2v0ndrO97soOvxaperYUPwSzlWokGAxNmAsAVbtAt1kekiGoAYTmd2seUOAJeMZ1rXUh23cA5bUcRL9ZNIl7jclXPw/fMq4zJCw2uZ54li3eWjkw44mEnx3ri+2RCnCbpwO4m3K8Y3KHRYvn210Ftu0j9VRxLY2UziAmS+Lq8CB32B0S8l6X/JaUl+xdTMkM02JM11NNsKq8U+6HF8mnAvXSfIs03ihjACIoj2Uh+UpBnNbOQAqaxFXkzJoQ37t+VwcJ+/D/uO3qDTZ4JGB+YB9ohD+E3AMOWZnypTgyOsAIcDMRE5cPpKOqTvWzntGWpYnobJcrGfwcen8/57/yXmGhnp2emLCKDEyTnX+yK9+fYoU1vsgWPlebrN0bnin0n1 6Lsn+ka4 Unrn0G7XB+dze+bJ7/hDtupbCz8jr4iNPRZFtR7zsJxU+DU0V+0Qg0LgXYH0IUOWzvj8r0/xtFArbJX0juu5Ox9wVpA//lTK5I4Et8biuvcPAZCsXIUM5x5WL4YkQS1muhShGQfxq4KZ7vDO0eTg3j7YYWXif/7z0ITyhNinbh8F4NCaxZn66LinTFuJ3Ewn3Jb7dbNvOM993PAq7wmy2JZ8YKLAZYT0zhAE6sFql2/0oHQY58rXodSxySi85nayIubwp2SQiubtIaJ3PzpeLQaZQLElw2jvYNZUyNY/yRjlgXaP3KOfUjBdMkgymyhnEShHusXoJfQ9//jOEmFo4sWbosKnmVUE2FjbB/dWQMSemnpK9wgOh89zonP/VFEYVVmVZ9sLEMcYi3hBd8tBCSBMFXdoeRfsknm5g1yHWKjtL7KDft75EMZ5/cVmk0bnsOIvwU9UeBKLJSvWu9IEe3sR33pDVqe08HIis09L+zzNxPRx5RwxZcw0eYb4un6aGSu3gPkRytQSo/8tWXGpYez2m1LvM5AKD/LaUEoeYI7g2EavZ1nnbUCKdnbD6fNLzBN14FzFMKk+P9RlICN51FODsMYLO2EOsBTyV7z0eHQuhP2QWDTKn+Sf2Wi2ar2TbaWKEiJdGt1kf+qNRMxzsmZozBOO0nlXiUb1U10LSJQBHtabADmCxlE6yxA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some contexts flush memcg stats outside of unified flushing, directly using cgroup_rstat_flush(). Add a helper for non-unified flushing, a counterpart for do_unified_stats_flush(), and use it in those contexts, as well as in do_unified_stats_flush() itself. This abstracts the rstat API and makes it easy to introduce modifications to either unified or non-unified flushing functions without changing callers. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c6150ea54d48..90f08b35fa77 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -639,6 +639,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } +/* + * do_stats_flush - do a flush of the memory cgroup statistics + * @memcg: memory cgroup to flush + * + * Only flushes the subtree of @memcg, does not skip under any conditions. + */ +static void do_stats_flush(struct mem_cgroup *memcg) +{ + cgroup_rstat_flush(memcg->css.cgroup); +} + /* * do_unified_stats_flush - do a unified flush of memory cgroup statistics * @@ -656,7 +667,7 @@ static void do_unified_stats_flush(void) WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + do_stats_flush(root_mem_cgroup); atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); @@ -7790,7 +7801,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + do_stats_flush(memcg); pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -7855,8 +7866,10 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); - return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); + struct mem_cgroup *memcg = mem_cgroup_from_css(css); + + do_stats_flush(memcg); + return memcg_page_state(memcg, MEMCG_ZSWAP_B); } static int zswap_max_show(struct seq_file *m, void *v) From patchwork Mon Aug 21 20:54:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13359808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91F67EE49AA for ; Mon, 21 Aug 2023 20:55:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1765E940013; Mon, 21 Aug 2023 16:55:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1000A8E0012; Mon, 21 Aug 2023 16:55:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6D41940013; Mon, 21 Aug 2023 16:55:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CEBAC8E0012 for ; Mon, 21 Aug 2023 16:55:25 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9F108140B1E for ; Mon, 21 Aug 2023 20:55:25 +0000 (UTC) X-FDA: 81149317410.03.76A9951 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf13.hostedemail.com (Postfix) with ESMTP id D30D220005 for ; Mon, 21 Aug 2023 20:55:23 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=KmH6O0vZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3Os_jZAoKCJUNDHGNz6B325DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3Os_jZAoKCJUNDHGNz6B325DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692651323; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=II24bAIuGVg2YyiqwiCMReE2mVTE25qmDUAuXBGJ4Ok=; b=7Ar+uzhTl5abwmTHFT4VzkFnnoESq6wP/e0laGCwFqcuAPIFOIyjw3n4wODgx1XbCBczw2 V3/W257pVwCSYf1QHR8bQwIeIirMTIOJOxmf3R6JVd3ApqwUHdl0MfC7eIKK+KE2uVov6s A14GIAOy30GlkPgqoHeZW/AA8mE9blM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=KmH6O0vZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3Os_jZAoKCJUNDHGNz6B325DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3Os_jZAoKCJUNDHGNz6B325DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692651323; a=rsa-sha256; cv=none; b=uUlcUEBJUfgbj3LFiyqEL00Y/sOB7KITYaBqSCLB+xpp8qfJDLYKqnuKLK3XgGubkxd1YD 73sY/SiwQ59Rr45PFrmEgQbZndV4C4w8JAxfowu5CWJ6zQni3WJIVNRhy1D7dgXJtx13oN NHef3eTnBqPU0+jwZu2MvbdXogOGQ8M= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1bdba3f0e73so41877095ad.1 for ; Mon, 21 Aug 2023 13:55:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692651323; x=1693256123; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=II24bAIuGVg2YyiqwiCMReE2mVTE25qmDUAuXBGJ4Ok=; b=KmH6O0vZ9sBp7HHBae0PwJCnRa8ujSumjQ0AlsGJ1pi5KbQIxpyrvcCO0krC5ztY+u N8ljzGOzIX6jer2HV1GxzZPCURYnOZC6D1lrlNXbpCHNDpRJB0nITXoN0mon6iaCy4e1 QkiLalZ25r1ZxplkbXN+NiDkoW8GDcA+Uj1Shac+wu5psshTskn8q8Lf6B5bKZffckoB jKrEuG2Vw03aNvOQfFER14s8IkriQqDS2U2JTgSKgCtg7TmcMJ9q3B18uyHoSMZM7lnj ioj21pXqwFO1fJbsQwZ0rFa+1Vc07Ob56riuci9Ea7Iw13vPBzZ96Hsd5+OoDw6zGDiR 3h5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692651323; x=1693256123; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=II24bAIuGVg2YyiqwiCMReE2mVTE25qmDUAuXBGJ4Ok=; b=KJvFNEMfM63rs0JCWHadM7MGEMNBJSa7DGFk1S+KsTjIb5vjLr+q7HZEb3hXusAzNH or9s5OjxtBsWrfJfLTBaGByFliRa57QD858HXzd/ZGV22w9iOpQdzFvbIIlZBh63mrd/ YetLPCsgf5kL1If734GCzV0Kg0Qe+8eAAkgGI8otXNP0DT55OhbHr4AlV5EnTADN9j83 EX/aARWOgO3ab93OUhyuubP6LmHJ4ZNmDLuELLLgkQZpGqwEWFdNMV23+CAVhfAIbDz+ SkfXazMOAMVzYw1h2NzVMWU2nlBeRLjqJxQ/1AW5WzLu8OaunPhWH4qw5nRfiZSIJXSe qD1Q== X-Gm-Message-State: AOJu0YwtHFc/fJ7qtTn6d4P18VLE9Cw65zHYLO/mLPbkL6GZTo2SPfQU 0BwRgV8MsLuwwwySGO/G+WD5x4fVCGZwRmTj X-Google-Smtp-Source: AGHT+IEvQmiYC84sZKnE5YzQy6JNP4dH1ZNkD129Wrs7O1rPHJU5tveJ2ce7KEKHH/DtaNVXLSxePpzIYxXdsGad X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:eccc:b0:1b8:a555:385d with SMTP id a12-20020a170902eccc00b001b8a555385dmr3847926plh.9.1692651322749; Mon, 21 Aug 2023 13:55:22 -0700 (PDT) Date: Mon, 21 Aug 2023 20:54:58 +0000 In-Reply-To: <20230821205458.1764662-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230821205458.1764662-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc1.204.g551eb34607-goog Message-ID: <20230821205458.1764662-4-yosryahmed@google.com> Subject: [PATCH 3/3] mm: memcg: use non-unified stats flushing for userspace reads From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: D30D220005 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: fixpoq4uwe98qqdmwpg7uh9k1n4737sa X-HE-Tag: 1692651323-429222 X-HE-Meta: U2FsdGVkX1+JJ+Aea7PEXAeinft69/s5Jczv582CIBmo6hH0yV3g10iQ3RldNs7TvrXVIS+7hrVrVAZdmDI1yWHhglAeUJoQsKElD/YjYYxxeWaDML3f0wMB7RoRWsrjPnDGCBr7YQjL0/yqVhhJzsejTrXQLt5gIdAZ6+djM69zYh09RNTbtasymb7yunNzDoKSdinh1mdgzKrbVQtrxwR8FlPHla3s5w+EsUX/4fb5rqUXAQS0plnwJR+JErTJAGEgdGdn2NI++wTMO2quq1j4DAJZPeRlCyKDSz5iMVUrYoym2hL1J337EbJE4KjvEDl4gfYS/ocs3Fs5cWQ+V4ZZdJwrKfMsc759pBhhNFNwc4iKFJJpswKjUcbHGViLmwO9VzuxHoJEtFdtRwQYFOCdx0DbyL3UZ4b67gwLLSsAIOY7/mB4zaoiAhj7TKVjBZtdHd4+Zh0p3Zg/gVb/gpcPJf9F9XRCmOgiDihiHCXbQRKgNp1DMfTsmwu9Xk/bpBfiKmbT2t3sA2L+x694mzpBJIui0KM11Rs53V1rapYS8FqR5SZBEQbvmJ6FnG8W+xRXFybF3YE8AllzR5mjbUnoqJ7fMRMnPpsic1rBOyANWV/vSOSQitzqOMuYp9CMy36OwXYM+SlZmjq6P8OCtdbr3rLLoLALyw051fIZXYFVDvADYsqOHcrsdIUwZNXvWy1sxfiQ7rxGyYSxz0KQsmfTXpYzKmzJSO8Fw0xAM6TK5z7AGbw+QNHoU9DeDteSuiYH4g0Ub4UzPnUjQAk1hkHKbNnQNXboYBflqmjQvzkJqJAVJEklH+u4ZFDNdZW14KlcrVa2uAmb5j5FiD9y5+kU4CDrGvR56ypXE7H1FUyG7VmJuw9ythYLn1sDB/BomyEWVm3R+ZP8dvkmiYaT91gJqqZZ8VolHDVox0nyIzq5xlIOgD4pB4PF2+kNy+uzFx5rNieaFR4jjxnuMsF FaVz+uZC lJFTvn5gDQn++hLpJQM9F68U/f7rNa7rfzfxRFhnX97xPVoHZl6AH9aRFO1brutcRVBPRmRV8dPIPPiPIibNruWNy0s/P/wJmjTbUM+7yvSq9dXmchPBrRYgsouTcr1TZdhgPQkpQ24UXGWYtAC/6guAa4rwAmhixB3K0IuA1/eXH2Z1udpMQljIxyKGCRJhUPTF8FJVaxq9MzTp7tIBygainKD0SAEj21v61QybuRU61MZOaorMkT80a3ovpCLjnYpiBaGkLTaTBxLPgIgJfqavEDnkGBMYN4Eqy+ChKVUCH1nBCdYYJgKMsuRPkqKACzdYaQm2rN24DuvtKfpJdo2YTyBngQFwOqs/UrLsxPvzP4Eq9jtLkiopxTTRwU3R3/hIrb2EKsFy5AwAMjEN6Z/lz+Usr1ZQKSTitrJF44qiR9lr4SKbG3rfIX0sOmOnKUlSrXMQCWynqgGdtIjt8nGq0Nj6lfIdq1Vt3UuFKeK+UnDL3N7WnENB4SLqeOtYfKDe8Eg1dZoy4cv0PAkMTwu1uoHHSZykAnFAFFPoQjgGsT6Xi+xmKWm+O/nfFSH6hlu57w+nj1bIXfAPjnWT1CF/z8kX078Gtw+U29v+qE7eFc+GDJrDPs7YjJ9stwPjRrjE98P/Rg3YhzdK1vTSZtjwN75VJL8Fe7P90/SP/U1F7Zffxwz+mMPcxyzH18PdtwoZsObo0VXLNdsV0bVD+V0Sjrg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unified flushing allows for great concurrency for paths that attempt to flush the stats, at the expense of potential staleness and a single flusher paying the extra cost of flushing the full tree. This tradeoff makes sense for in-kernel flushers that may observe high concurrency (e.g. reclaim, refault). For userspace readers, stale stats may be unexpected and problematic, especially when such stats are used for critical paths such as userspace OOM handling. Additionally, a userspace reader will occasionally pay the cost of flushing the entire hierarchy, which also causes problems in some cases [1]. Opt userspace reads out of unified flushing. This makes the cost of reading the stats more predictable (proportional to the size of the subtree), as well as the freshness of the stats. Since userspace readers are not expected to have similar concurrency to in-kernel flushers, serializing them among themselves and among in-kernel flushers should be okay. This was tested on a machine with 256 cpus by running a synthetic test The script that creates 50 top-level cgroups, each with 5 children (250 leaf cgroups). Each leaf cgroup has 10 processes running that allocate memory beyond the cgroup limit, invoking reclaim (which is an in-kernel unified flusher). Concurrently, one thread is spawned per-cgroup to read the stats every second (including root, top-level, and leaf cgroups -- so total 251 threads). No regressions were observed in the total running time; which means that non-unified userspace readers are not slowing down in-kernel unified flushers: Base (mm-unstable): real 0m18.228s user 0m9.463s sys 60m15.879s real 0m20.828s user 0m8.535s sys 70m12.364s real 0m19.789s user 0m9.177s sys 66m10.798s With this patch: real 0m19.632s user 0m8.608s sys 64m23.483s real 0m18.463s user 0m7.465s sys 60m34.089s real 0m20.309s user 0m7.754s sys 68m2.392s Additionally, the average latency for reading stats went from roughly 40ms to 5 ms, because we mostly read the stats of leaf cgroups in this script, so we only have to flush one cgroup, instead of *sometimes* flushing the entire tree with unified flushing. [1]https://lore.kernel.org/lkml/CABWYdi0c6__rh-K7dcM_pkf9BJdTRtAU08M43KO9ME4-dsgfoQ@mail.gmail.com/ Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 90f08b35fa77..d3b13a06224c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1606,7 +1606,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4048,7 +4048,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4123,7 +4123,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4625,7 +4625,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6640,7 +6640,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid;