From patchwork Thu Sep 30 04:47:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12527247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2D3AC433FE for ; Thu, 30 Sep 2021 04:47:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7FB9A61361 for ; Thu, 30 Sep 2021 04:47:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7FB9A61361 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 07D8A940089; Thu, 30 Sep 2021 00:47:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02CA694003A; Thu, 30 Sep 2021 00:47:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5EA6940089; Thu, 30 Sep 2021 00:47:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id D694794003A for ; Thu, 30 Sep 2021 00:47:28 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8A1AC8249980 for ; Thu, 30 Sep 2021 04:47:28 +0000 (UTC) X-FDA: 78643006176.10.6B97B9C Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 3E23510000B7 for ; Thu, 30 Sep 2021 04:47:28 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id v7-20020a1709029a0700b0013daaeaa33eso2551773plp.5 for ; Wed, 29 Sep 2021 21:47:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=8iSDbwVHcLjIGb+FNAeM8Z7fJcNOoEVp1tR+zgdQ3tk=; b=ZvVHnNiP3I5iA4kndR/KFrSjz/XRG3SNhzrYtsHrv6nYpWfgS11lbdgnwalrhTQxPw gUXnr0ppOFu/OYRztH34Ga469sm7p2eFREnC1lMUECpJ5nr4YP+T8w3w7+3M7khgpUcX sfsMqAlQ8bs6Kh2tVGwD40grHqL15xYDgsGKjR6MQE1ze//ztY8nOQYA8rByTD+o6PZS ZVXfGBXRK2lEUrvKaUmg/hv0j5WsnRwaPsHwOGXiy32X+x+9gHQly+aTQaKtTv7hedXf 66CdwUw8K6emOzFWGnJmkXA15O9MRO/Ld09tIIU6RdWFebmBB8R0BNXllcf2VvVn2vGs ZUNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=8iSDbwVHcLjIGb+FNAeM8Z7fJcNOoEVp1tR+zgdQ3tk=; b=K6K6IlIXj71wX5SdCKJmGSF0hQvCcGdaCJuic6dAK5d/LG+ir2sn6FJFtgBeYJ1wSz 0Z0dE1RC0kn275LY0QT0iVtm2B2MjponJ3+WHeH/DqBItEslgp624SmhIsC1YpMu1hMQ 1xKnfLjwDYzGYehJLFLalLYpsj9DNc5CI+eEDaY1NZnN3pt1LnmqYTScR9NrUhmrtqbf QMvCjLevspAGQTr9F+gIGsHQWPg5xCq5Fiw+HglaFr5qwnoX5TG5uU3Q8qW+PP6g8/wh bn3iQado0kwsJQ/B6U9ulsJv8MyH10B5Vt9TxLyEHPpXKcATglOoEJyB98PoY6r+5EBH n7lw== X-Gm-Message-State: AOAM531YdnpAEEkzhYDdCB3Y/SiU+0pWVU4l28qSDnqBiWjOhlfmUU/9 lH6G+5XgI9KC//Ex3WIRmY9700jLQz3NpA== X-Google-Smtp-Source: ABdhPJzYoIhmmUYXvt/79ki4qSrM4WaWtWsCWSaj/1EfMbrR4wE4IvcteqnEo8nZwY+9ax8V9SLla+e+lQbQ0w== X-Received: from shakeelb.svl.corp.google.com ([2620:15c:2cd:202:4f8a:fdf0:d4ae:6a30]) (user=shakeelb job=sendgmr) by 2002:a63:724b:: with SMTP id c11mr3184147pgn.9.1632977246883; Wed, 29 Sep 2021 21:47:26 -0700 (PDT) Date: Wed, 29 Sep 2021 21:47:10 -0700 Message-Id: <20210930044711.2892660-1-shakeelb@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.685.g46640cef36-goog Subject: [PATCH 1/2] memcg: flush stats only if updated From: Shakeel Butt To: Johannes Weiner , Michal Hocko Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 3E23510000B7 X-Stat-Signature: 8cf3ge3serpjjyme7jsejsoyj8dzgd1j Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZvVHnNiP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3XkFVYQgKCEc1qjtnnukpxxpun.lxvurw36-vvt4jlt.x0p@flex--shakeelb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3XkFVYQgKCEc1qjtnnukpxxpun.lxvurw36-vvt4jlt.x0p@flex--shakeelb.bounces.google.com X-HE-Tag: 1632977248-6591 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: At the moment, the kernel flushes the memcg stats on every refault and also on every reclaim iteration. Although rstat maintains per-cpu update tree but on the flush the kernel still has to go through all the cpu rstat update tree to check if there is anything to flush. This patch adds the tracking on the stats update side to make flush side more clever by skipping the flush if there is no update. The stats update codepath is very sensitive performance wise for many workloads and benchmarks. So, we can not follow what the commit aa48e47e3906 ("memcg: infrastructure to flush memcg stats") did which was triggering async flush through queue_work() and caused a lot performance regression reports. That got reverted by the commit 1f828223b799 ("memcg: flush lruvec stats in the refault"). In this patch we kept the stats update codepath very minimal and let the stats reader side to flush the stats only when the updates are over a specific threshold. For now the threshold is (nr_cpus * CHARGE_BATCH). To evaluate the impact of this patch, an 8 GiB tmpfs file is created on a system with swap-on-zram and the file was pushed to swap through memory.force_empty interface. On reading the whole file, the memcg stat flush in the refault code path is triggered. With this patch, we bserved 63% reduction in the read time of 8 GiB file. Signed-off-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6da5020a8656..933dde29c67b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -107,6 +107,8 @@ static bool do_memsw_account(void) static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); static DEFINE_SPINLOCK(stats_flush_lock); +static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_threshold = ATOMIC_INIT(0); #define THRESHOLDS_EVENTS_TARGET 128 #define SOFTLIMIT_EVENTS_TARGET 1024 @@ -635,6 +637,13 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) return mz; } +static inline void memcg_rstat_updated(struct mem_cgroup *memcg) +{ + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + if (!(__this_cpu_inc_return(stats_updates) % MEMCG_CHARGE_BATCH)) + atomic_inc(&stats_flush_threshold); +} + /** * __mod_memcg_state - update cgroup memory statistics * @memcg: the memory cgroup @@ -647,7 +656,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) return; __this_cpu_add(memcg->vmstats_percpu->state[idx], val); - cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + memcg_rstat_updated(memcg); } /* idx can be of type enum memcg_stat_item or node_stat_item. */ @@ -675,10 +684,12 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, memcg = pn->memcg; /* Update memcg */ - __mod_memcg_state(memcg, idx, val); + __this_cpu_add(memcg->vmstats_percpu->state[idx], val); /* Update lruvec */ __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val); + + memcg_rstat_updated(memcg); } /** @@ -780,7 +791,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, return; __this_cpu_add(memcg->vmstats_percpu->events[idx], count); - cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + memcg_rstat_updated(memcg); } static unsigned long memcg_events(struct mem_cgroup *memcg, int event) @@ -5341,15 +5352,22 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css) memcg_wb_domain_size_changed(memcg); } -void mem_cgroup_flush_stats(void) +static void __mem_cgroup_flush_stats(void) { if (!spin_trylock(&stats_flush_lock)) return; cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + atomic_set(&stats_flush_threshold, 0); spin_unlock(&stats_flush_lock); } +void mem_cgroup_flush_stats(void) +{ + if (atomic_read(&stats_flush_threshold) > num_online_cpus()) + __mem_cgroup_flush_stats(); +} + static void flush_memcg_stats_dwork(struct work_struct *w) { mem_cgroup_flush_stats(); From patchwork Thu Sep 30 04:47:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12527249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE028C433F5 for ; Thu, 30 Sep 2021 04:47:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 707EE6136A for ; Thu, 30 Sep 2021 04:47:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 707EE6136A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 11B4694008A; Thu, 30 Sep 2021 00:47:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CA5E94003A; Thu, 30 Sep 2021 00:47:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFB2B94008A; Thu, 30 Sep 2021 00:47:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0132.hostedemail.com [216.40.44.132]) by kanga.kvack.org (Postfix) with ESMTP id DFFFF94003A for ; Thu, 30 Sep 2021 00:47:39 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 967A9183DEA9F for ; Thu, 30 Sep 2021 04:47:39 +0000 (UTC) X-FDA: 78643006638.08.7AF7B9F Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf29.hostedemail.com (Postfix) with ESMTP id 51175900066C for ; Thu, 30 Sep 2021 04:47:39 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id cm10-20020a17090afa0a00b0019f222501baso4467256pjb.0 for ; Wed, 29 Sep 2021 21:47:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=3wCSVYwLyxFTgw/NEPcqwwI1Bpk2vqRKGXaqYC4gfms=; b=DtSDabIIC+wI82CTcYe7egJAGdF/UFE2BKRj3Yae1ZQd2V54f2o78Q4UrdWf6YKqcr 0wFzisOz2LGitNBfKkvecCzJqQWOrcIwALJwdVcDjpLyGBqXFuE0neUoeoIzr6FbF6Hc o/GZeEV5sqXh12jPc516sCjkzUNl/KaC1jmlcnsR9DUHvLoRX28wrFdu2nAtyXi95Hvj JXk4dtsivlft9MddRs1ZmyQrGJBxpzcG4/4iyfcnmAPPSQJWjr2je++2ppdKs91qx7mk 9KqgdC5GaMuCyA5Bb8eMtD1wBT82M0MOUL3kW3ohFKxivrEagmT9gRJhVace95S4FYHx tzcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=3wCSVYwLyxFTgw/NEPcqwwI1Bpk2vqRKGXaqYC4gfms=; b=UgeS3qvIwrzUa1jaLROu8g9hJK+6KiLGgddpUxg4072nuPbSnnP7uEQlM8eNg06I3e KxdQ9yYFnve6NQ4S/eKsTy4JX6DNrvTKgTdf/5L3KCHlTQcTnHXOoXaucmS/nv9NTo7/ TU4FZvj1fgQ4NPooA7urh2TC+J1ENhtX5cGTDeu2GJ1SLKCHKQkwmWBHXVEYfYFmMAo/ CIQBZB2CV0xAP2fHWQMqkRjKbHPYMIHsidcE3TzVj7DbPmzhY6o1ixw5HCFLcRQkxSZi 4REWYdqvs/uOA+PXxs6Nc0w4p5FCBldzbZx1L7VTuW2Y2aLj1+WIo27vhYZ9io5juqUK aoxQ== X-Gm-Message-State: AOAM5324oNCe/CCtlMxT736rQeCjl9szC5eWqi3kTPUQ1rdHkDrUWkmQ 8eFfvpkGD6DpV1UFPQCsI2QkIUJUTRwXxA== X-Google-Smtp-Source: ABdhPJyzzSFOJ25gEBsJ3rGoOkoxHHAwcnvoxBWUw9cUHF51oByMunT/xnn7z63a888V1JleYEgftdTbWgb12w== X-Received: from shakeelb.svl.corp.google.com ([2620:15c:2cd:202:4f8a:fdf0:d4ae:6a30]) (user=shakeelb job=sendgmr) by 2002:a17:90a:8b89:: with SMTP id z9mr4151194pjn.89.1632977258242; Wed, 29 Sep 2021 21:47:38 -0700 (PDT) Date: Wed, 29 Sep 2021 21:47:11 -0700 In-Reply-To: <20210930044711.2892660-1-shakeelb@google.com> Message-Id: <20210930044711.2892660-2-shakeelb@google.com> Mime-Version: 1.0 References: <20210930044711.2892660-1-shakeelb@google.com> X-Mailer: git-send-email 2.33.0.685.g46640cef36-goog Subject: [PATCH 2/2] memcg: unify memcg stat flushing From: Shakeel Butt To: Johannes Weiner , Michal Hocko Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DtSDabII; spf=pass (imf29.hostedemail.com: domain of 3akFVYQgKCFMD2v5zz6w19916z.x97638FI-775Gvx5.9C1@flex--shakeelb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3akFVYQgKCFMD2v5zz6w19916z.x97638FI-775Gvx5.9C1@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 51175900066C X-Stat-Signature: cqnxm7o96ipn7jatr1rp3yfhzqfsw4fr X-HE-Tag: 1632977259-398425 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The memcg stats can be flushed in multiple context and potentially in parallel too. For example multiple parallel user space readers for memcg stats will contend on the rstat locks with each other. There is no need for that. We just need one flusher and everyone else can benefit. In addition after aa48e47e3906 ("memcg: infrastructure to flush memcg stats") the kernel periodically flush the memcg stats from the root, so, the other flushers will potentially have much less work to do. Signed-off-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 933dde29c67b..688c891448dd 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1425,7 +1425,7 @@ static char *memory_stat_format(struct mem_cgroup *memcg) * * Current memory state: */ - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -3529,8 +3529,7 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - /* mem_cgroup_threshold() calls here from irqsafe context */ - cgroup_rstat_flush_irqsafe(memcg->css.cgroup); + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -3911,7 +3910,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -3983,7 +3982,7 @@ static int memcg_stat_show(struct seq_file *m, void *v) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4486,7 +4485,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - cgroup_rstat_flush_irqsafe(memcg->css.cgroup); + mem_cgroup_flush_stats(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -5354,12 +5353,14 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css) static void __mem_cgroup_flush_stats(void) { - if (!spin_trylock(&stats_flush_lock)) + unsigned long flag; + + if (!spin_trylock_irqsave(&stats_flush_lock, flag)) return; cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock(&stats_flush_lock); + spin_unlock_irqrestore(&stats_flush_lock, flag); } void mem_cgroup_flush_stats(void) @@ -6391,7 +6392,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid;