From patchwork Tue Mar 28 06:16:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190544 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65C09C77B61 for ; Tue, 28 Mar 2023 06:17:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232477AbjC1GRE (ORCPT ); Tue, 28 Mar 2023 02:17:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232446AbjC1GRB (ORCPT ); Tue, 28 Mar 2023 02:17:01 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46B492D75 for ; Mon, 27 Mar 2023 23:16:44 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id oa14-20020a17090b1bce00b0023d1b58d3baso2934657pjb.4 for ; Mon, 27 Mar 2023 23:16:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984204; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cs9g98LfB/5B5fgCHg0khEoDDLXk/FbX9zQPYHFGol8=; b=IrEj06HWrJaiEe47kle+4mjwycO8VkK4UYyVi89hOr/BT4ba3YoJaIkMxMmDu91Azt OCiIdpmqw/gfkGEUiinzv01MOm68VDd/zLRgX2gIU8f7UybpdpXtpixpCNAld7gzowZR +gyt1jsqLawCKRD+ojRnZ6c35iauByhdICeE6tA9/AW2kV9h46wsE+qITfJzQSmauG1W ySDyn3mwIkhUXYNihBaZeA4rjBJGjlP0HSdTCoy9A7k1vdMTk4m51J+idABUhgX1bKO7 He48ZTeiLxeRfpruo08A01IDO1BpVxjEsw+pmwGtbL4byIuTDU8yveD/qiON77jThzUh Vo3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984204; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cs9g98LfB/5B5fgCHg0khEoDDLXk/FbX9zQPYHFGol8=; b=nD9umEV0Gr4N+92ffy5dS6xK/C23Zc/vsQ1tlRQqX5m1ikvCEeuPqMjdYB6O6GCR8/ eo8l5AtNe3frfpnH1cMYI9BaIMHQS8FUceO3wmNjZ19IbFSJgKkcTNY2g0SnWMtQo6bJ IXD4TtJcFPzWdZHgUoatjJeTRfiHs+eGiD+axXOSmk5g9LWP7NR9yDakNnv3gCDuVSH5 GOaTPcVYdR9J0af8bcA3GofV/H88UEtXMD0P0mxvpbU9XKAv1geoejioTzaDSgdjOBHn MuQhj8d5MAQ3xH/zxRYOIfLl5BgkTAgtQMlDHv93F1LTYqkgzonh4Ae5I3gP5WrMqwsQ 2d9g== X-Gm-Message-State: AAQBX9eJbcIJfmvWn6u06pP/4IrPWvWJ2m5WuLJ4afk0PAFHd4OEP+ss nf1aaHOpZwbHmH0gOG5vojNETsFY4tYFf5WO X-Google-Smtp-Source: AKy350aYOPbMPoVcZ2xVfirHkGMy5US/bJka4xE6zLbD6emP+rv0j2dGtP2N8z3/3tcuS7Hx271FErAXLO/AbXkX X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:24c4:b0:629:f9a2:64e6 with SMTP id d4-20020a056a0024c400b00629f9a264e6mr7565323pfv.4.1679984203700; Mon, 27 Mar 2023 23:16:43 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:30 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-2-yosryahmed@google.com> Subject: [PATCH v1 1/9] cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org cgroup_rstat_flush_irqsafe() can be a confusing name. It may read as "irqs are disabled throughout", which is what the current implementation does (currently under discussion [1]), but is not the intention. The intention is that this function is safe to call from atomic contexts. Name it as such. Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/cgroup.h | 2 +- kernel/cgroup/rstat.c | 4 ++-- mm/memcontrol.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..885f5395fcd0 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..d3252b0416b6 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -241,12 +241,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) } /** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() + * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() * @cgrp: target cgroup * * This function can be called from any context. */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) +void cgroup_rstat_flush_atomic(struct cgroup *cgrp) { unsigned long flags; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..0205e58ea430 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -642,7 +642,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); spin_unlock_irqrestore(&stats_flush_lock, flag); } From patchwork Tue Mar 28 06:16:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190545 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25ABAC76196 for ; Tue, 28 Mar 2023 06:17:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232499AbjC1GRF (ORCPT ); Tue, 28 Mar 2023 02:17:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232456AbjC1GRC (ORCPT ); Tue, 28 Mar 2023 02:17:02 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EC8F30ED for ; Mon, 27 Mar 2023 23:16:46 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id z14-20020a170903018e00b001a1a2bee58dso7050565plg.3 for ; Mon, 27 Mar 2023 23:16:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wKfUWPDFjTfCFAmAu+PDSXoVaLfaEZa4F2xpA6M98WI=; b=ML25Z5sJR8k5t8kSQzNLjGMjEKmdcebxHFtYwKZcrTLZX2wK0gMrUstJ28IAfa/l/B dmJfAqHqeUNGDMqpjHMcUn/3rl6ZtTwoLtJc4kM6PKx7oTSnQD1aK8fu6wcMuvNpddNP JXz/aPGcRNjn7CdP1wNYEAXnpd0Uj2w+rzBilII9bcn8vsNoKap2/emH97XB9yYmQX0j zWlTQgN/LRjTLn7LSTnUMs5Zi8YiSMdP4rhUt8H9KINdPVTgc9z/VX9FnrviFmsMUsqK yY+aBb6I7/+gAXbDQAsJSzcOoGzk7Nj4yOSGJy0Va4z4/Vp/gmfRCs4LrcZySZjB+aj0 I08g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wKfUWPDFjTfCFAmAu+PDSXoVaLfaEZa4F2xpA6M98WI=; b=Ic5zQSiAOEMugLFvHwVyqafUCT1MARYTYeKyX/1Ozn/F4TTfPe4vvZvtj02QZ1hrNK EoLVzPkJvD5MkRfi+bTCLJ6YQSaK2iRa6Rc4agRd85aa5aK2JZyO0ZtC2Pje7agrT07n GhalJkf8JC0JZz7vWHUI3KYPyGS5OjyV1CmeM02lUqlS+AvvccFg0j0gGxsaA/SB3K6f bpmCPVzuPXvbpsjY5tetoYuNwJoN+PJRvNheoTzEhD2QzqpJGZm+udosEUbEGOZfLGex u53tJXmjcz8WVY7i9L1jzng2+n1FdtfeJO9XDkZTTmtHM4geHGf8kD7r1DSmak15N4n2 fhCQ== X-Gm-Message-State: AAQBX9fvhZWFTDMe3INyKlTXgGAp53oK6Ob9Qz0u1WIgyn42aToabmbU w5jQjmkyoyu8peX0gDGAiHbHWFCP+va9uSc7 X-Google-Smtp-Source: AKy350Z/+j+6P1wMetl7E14Y1prBzO4hFyvwBGRqSlF4uLs1fV1I04aur9Q5rKjp70R1U9RxRmWGSymD4DlPFw5z X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:7088:b0:19a:5953:e85c with SMTP id z8-20020a170902708800b0019a5953e85cmr5289230plk.1.1679984205498; Mon, 27 Mar 2023 23:16:45 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:31 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-3-yosryahmed@google.com> Subject: [PATCH v1 2/9] memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org mem_cgroup_flush_stats_delayed() suggests his is using a delayed_work, but this is actually sometimes flushing directly from the callsite. What it's doing is ratelimited calls. A better name would be mem_cgroup_flush_stats_ratelimited(). Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 2 +- mm/workingset.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..ac3f3b3a45e2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1535,7 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0205e58ea430..c3b6aae78901 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -653,7 +653,7 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, flush_next_time)) mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..af862c6738c3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 06:16:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190546 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E796C761A6 for ; Tue, 28 Mar 2023 06:17:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232487AbjC1GRL (ORCPT ); Tue, 28 Mar 2023 02:17:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232401AbjC1GRD (ORCPT ); Tue, 28 Mar 2023 02:17:03 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D71543A84 for ; Mon, 27 Mar 2023 23:16:47 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id ie21-20020a17090b401500b0023b4ba1e433so2942162pjb.0 for ; Mon, 27 Mar 2023 23:16:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wILXj9LqWwom+6PQC1+rRSpOfUrkR+GzdkCDGSSBhGY=; b=sq4WxgrXs35T1kNaqttQxLQHBBfHPzw9iKcGHtZRmWn7KoDZlSxKdNufMOgLyKUFr+ TMiE/wqtjNlMTixUEQeY/Vrw6MeR14OOP90t7VeDCccykNrGNGWLRshNs7Tr5dzPpQxM EqNAZipGy1I27uXZW4pV5ZH3WWGxC9s0suXWrK0/Qi+Fx4TfEA0shsUjup5r4I0T4jcO otnYwmVT7RvIIuwqMJTULG/AsEi9P5/3lltCZIuxMi9242jZnrgGHko3vAN7pXgBxOSv 6bmFPpVjK2j81kXxaYSwcWty18LJNFnzIpVeFZG9+e6GZQd6MIhYJNlu+ckzRRvaCz8L Rlqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wILXj9LqWwom+6PQC1+rRSpOfUrkR+GzdkCDGSSBhGY=; b=00ibUKIsu6rf4x8BjaDpIUDYyBRCrhLSV/Hm6pQLaWCRUggQNduLdtMV60pYDB2sds Hrua0T0eqk+7/oVldOosNhCwrwCRfF7ghL4ztqhe26MrufcXgtcFqhSR5ZxWk0R7OuLt 0/paJFjcCE0eIEs2pdh+FMxPu/IAoj+0ztIW1a7RHf7LVcYSJP1QNYBSo6byZr+i2tlh gBLKVhjrPZtLC4oVER21Mr88MP3p/aQ30L6ZewnJZUEncPCLM/DUpAA3YMSpfCabGMi8 BdcpMuF7Amxg1f94SJ45DyMcsIX7gECzJxdOvWLGsy1GSoDoOxOQVYph6q1QjtkVFbmx MHlw== X-Gm-Message-State: AO0yUKVD9ZBCuKON0r0UewX8MtyOIYbcO8liWLTGNYxLHhWMGntJMOLy gVjWR1hE9jgUNDPZVoCFjKbV9OFe3F/vpA03 X-Google-Smtp-Source: AK7set/NXJ9x9Sdrim9yKl61I9Pd5pvCD8XFm/ZA+2N5HaWD82pwRjDqKXOdg8RZFiQsg1c0bhIGvLI1ByyxVbFu X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:4510:b0:626:1710:9b7d with SMTP id cw16-20020a056a00451000b0062617109b7dmr8924786pfb.0.1679984207461; Mon, 27 Mar 2023 23:16:47 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:32 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-4-yosryahmed@google.com> Subject: [PATCH v1 3/9] memcg: do not flush stats in irq context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Currently, the only context in which we can invoke an rstat flush from irq context is through mem_cgroup_usage() on the root memcg when called from memcg_check_events(). An rstat flush is an expensive operation that should not be done in irq context, so do not flush stats and use the stale stats in this case. Arguably, usage threshold events are not reliable on the root memcg anyway since its usage is ill-defined. Suggested-by: Johannes Weiner Suggested-by: Shakeel Butt Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c3b6aae78901..ff39f78f962e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3669,7 +3669,21 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * We can reach here from irq context through: + * uncharge_batch() + * |--memcg_check_events() + * |--mem_cgroup_threshold() + * |--__mem_cgroup_threshold() + * |--mem_cgroup_usage + * + * rstat flushing is an expensive operation that should not be + * done from irq context; use stale stats in this case. + * Arguably, usage threshold events are not reliable on the root + * memcg anyway since its usage is ill-defined. + */ + if (in_task()) + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) From patchwork Tue Mar 28 06:16:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190547 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F047C76196 for ; Tue, 28 Mar 2023 06:17:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232401AbjC1GRM (ORCPT ); Tue, 28 Mar 2023 02:17:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232446AbjC1GRF (ORCPT ); Tue, 28 Mar 2023 02:17:05 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1BB83C1D for ; Mon, 27 Mar 2023 23:16:49 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id o10-20020a17090ac08a00b0023f3196fa6fso2973000pjs.2 for ; Mon, 27 Mar 2023 23:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+vNiceMoZ2PdzrNG/xVeRzEo7O+ovCZNHcQMp7VClk4=; b=FtVms547YGVdf9XAqjguoo0WST2Bomcq4pMeCW3qlg0ksZV5swPoVz0GmrxFu8vfrh HBzgJptVGGKy7geKpBPvC2PsWo0HmjT2F27i9dbdVA6zCF4EYemeZRNrh89DbSPqLObX h82tbSvZ/WCeLjLtii1Meb+k3BLVneGl+TdsHqmfTXT/vAFZIC+/S3Aw9k3ElSrpUy7X 9xgxQRUTlE19S9rQfATi59HRnf65TEpEpviTmJbLID2U9/qvSayVfcGFQGEl03zvUEu5 uDhGnK3XWBy1kJngfOB04Odcd1UeEO5WA33WjKyVDhleLQy8k50zMQSY0RXklX82o1Jn CmFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+vNiceMoZ2PdzrNG/xVeRzEo7O+ovCZNHcQMp7VClk4=; b=GfYV3R9xnh9l3z8WuqrnaUjQDVQvTGC8yegUinzI2k2dt/0hoU/nNlcN3G+3YEQ2L2 BCzZw8C6oAjMzs4vBD0erkiuE0IPq3NQwwPrbpuyYE47G7ZY+cLywGN35SHCcBPEmmRD IBRQ+rAIYY+JQda+e0d0KJhAgVQ/Dmah7Rgs5rHJY02EyOAfvHbCC+33SBHQWrMTh9Bk 2aRkgxN2Fnqo8p2XYLaudfHsmZ+3tq8iARQEjPP+SfE/sE/Od6qT63RFTeE6rDpH0/9X c8c2jQRxyMBBozIP8e1lbu9QFLthrIusq56ZXO/AHULshke5Y93/SUzQ6nrprrrjna9A nYoA== X-Gm-Message-State: AAQBX9fTkqNeGNosuDYTkQ2nXTEM2c/d27SCyUelSt8ruVJrLZk2Vw8w rmtr3kFFRu6naqSZSgsNbL7GkeNXSueyCZiP X-Google-Smtp-Source: AKy350az2ZQxKTBjGyFUiNza+YDGvWpTHjavWcHAugAuQk4rmOEm1wQoqfQkURL2u01jT9Cm3t+xr5muiADnxiOT X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:1a46:b0:625:c7de:48c1 with SMTP id h6-20020a056a001a4600b00625c7de48c1mr7269994pfv.4.1679984209151; Mon, 27 Mar 2023 23:16:49 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:33 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-5-yosryahmed@google.com> Subject: [PATCH v1 4/9] cgroup: rstat: add WARN_ON_ONCE() if flushing outside task context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org rstat flushing is too expensive to perform in irq context. The previous patch removed the only context that may invoke an rstat flush from irq context, add a WARN_ON_ONCE() to detect future violations, or those that we are not aware of. Signed-off-by: Yosry Ahmed Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index d3252b0416b6..c2571939139f 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -176,6 +176,8 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) { int cpu; + /* rstat flushing is too expensive for irq context */ + WARN_ON_ONCE(!in_task()); lockdep_assert_held(&cgroup_rstat_lock); for_each_possible_cpu(cpu) { From patchwork Tue Mar 28 06:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190548 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59E7BC76195 for ; Tue, 28 Mar 2023 06:17:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232521AbjC1GR2 (ORCPT ); Tue, 28 Mar 2023 02:17:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232526AbjC1GRI (ORCPT ); Tue, 28 Mar 2023 02:17:08 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A1C33C2B for ; Mon, 27 Mar 2023 23:16:51 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id t12-20020a65608c000000b005091ec4f2d4so3007672pgu.20 for ; Mon, 27 Mar 2023 23:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IdlOdGF0YCCYf+h2+Wx+Pr2285UVOLm52gCW9Jb/OEo=; b=dxPzci0vIh1xyJxlLBjWkEhAjm9YbPSdsUTO5u3B5j0mjvT8KDBeosrn5UiPm3Ksna ECFLKks3+i/JLwnVARkeiq9XCne6PDhOU+plEpcmqroZ0FIo5b3stpDp4azRz7RDEUXr fg0RX+xHWkbr/97SE9yMggMcSWIL9SDqbA2byVm27sTtHVM/RcaUTUFVwNm3hT6OyTW8 KGm1B3oJJOu2mLGqIrSENlsxO5AJatCTwMfnNnO9IVILzvLEURQL9lNGe4Pge9n4mWvC 01ODxHoNCu+DMqEmep2UjWrEts9KjVkOV7sWpi8aLYcEgH/6hlGK54jM+qhbvTAI+Y6D 5qIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IdlOdGF0YCCYf+h2+Wx+Pr2285UVOLm52gCW9Jb/OEo=; b=uDQ9XRsMUTGgMDvFVTpWR9V3ccDqXYLU4jvP7/5+itjevGZeoKk1YvP0NDxFNki7oJ u9wjoW0Ui+m/Fi5m5n5jrzttpYoL0bRaKavOnf+bkUbH+iXNF6m1Pt7NhG0O65jPSKpj ya8i1wlRppvmrHo7l75GQuS49gvSnzm3TlfwXXCL4ljH7aDSKlJON2doCCc6qIx5Chuv RvFAI1hizRRxVpvkHONK7VgO2POqX4qcCKjklGqL0eqRdFD3gb98gq9iUROKm7OeGQvm u2NtNDGQT4P7YEPoTgL49UlaVHCoXETlZ4pyOFAJOva9Ax5Egd+MXauoIBraRjrCdMLw uXBA== X-Gm-Message-State: AAQBX9cE4AO25Nu1paoVJ82VbJYvCbdvyU7vtPJecWX7l+i775nLerKX ACtZL7Pi0nfK1EN2QRPwywxk2uGj8s4N/3/t X-Google-Smtp-Source: AKy350ZiMUeqTdYquDazwFQP3GVswROLJyitcbpH2br9NmUFXZZud9qfmMVPueO4yaGFykMkMD2PV8f94RgXrI2o X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:74c9:b0:23f:1105:4295 with SMTP id p9-20020a17090a74c900b0023f11054295mr4298168pjl.7.1679984210825; Mon, 27 Mar 2023 23:16:50 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:34 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-6-yosryahmed@google.com> Subject: [PATCH v1 5/9] memcg: replace stats_flush_lock with an atomic From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org As Johannes notes in [1], stats_flush_lock is currently used to: (a) Protect updated to stats_flush_threshold. (b) Protect updates to flush_next_time. (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. However: 1. stats_flush_threshold is already an atomic 2. flush_next_time is not atomic. The writer is locked, but the reader is lockless. If the reader races with a flush, you could see this: if (time_after(jiffies, flush_next_time)) spin_trylock() flush_next_time = now + delay flush() spin_unlock() spin_trylock() flush_next_time = now + delay flush() spin_unlock() which means we already can get flushes at a higher frequency than FLUSH_TIME during races. But it isn't really a problem. The reader could also see garbled partial updates, so it needs at least READ_ONCE and WRITE_ONCE protection. 3. Serializing cgroup_rstat_flush() calls against the ratelimit factors is currently broken because of the race in 2. But the race is actually harmless, all we might get is the occasional earlier flush. If there is no delta, the flush won't do much. And if there is, the flush is justified. So the lock can be removed all together. However, the lock also served the purpose of preventing a thundering herd problem for concurrent flushers, see [2]. Use an atomic instead to serve the purpose of unifying concurrent flushers. [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner --- mm/memcontrol.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ff39f78f962e..64ff33e02c96 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -636,15 +636,18 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * We always flush the entire tree, so concurrent flushers can just + * skip. This avoids a thundering herd problem on the rstat global lock + * from memcg flushers (e.g. reclaim, refault, etc). + */ + if (atomic_xchg(&stats_flush_ongoing, 1)) return; - flush_next_time = jiffies_64 + 2*FLUSH_TIME; + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) @@ -655,7 +658,7 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, flush_next_time)) + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) mem_cgroup_flush_stats(); } From patchwork Tue Mar 28 06:16:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190549 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84590C76195 for ; Tue, 28 Mar 2023 06:17:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232658AbjC1GRn (ORCPT ); Tue, 28 Mar 2023 02:17:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232557AbjC1GRR (ORCPT ); Tue, 28 Mar 2023 02:17:17 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABF8435A7 for ; Mon, 27 Mar 2023 23:16:53 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id b8-20020a17090a488800b0023d1bf65c7eso2972471pjh.3 for ; Mon, 27 Mar 2023 23:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984213; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9IOPAG+bqxWxKt0wdRgDKHhdCXWwpmJ9UigZO44sKyE=; b=ntB4QsMVT8Fqv0//N2SyNJC8TO+SF5Nd//5MMCXPuMIlVWZztrbNiDvPnYTm36PsVJ 0K9UDAdVpl/jKPNVqGESnlXBby6gGh6KP9olFVGSWL7LZjX9hTDyTd2rcdVRYaALTq3d Svdw5D9xcS6f+R64pTOq4hP3DTwLDxc7s6WmPqIXIzObjUGQ7BCksYYcDbsn8CMNCe4l XYGLULvlYDvGRGQwaFkUUkpNiN3oSuW1tCmNSMxILm7ttbjB9+T2xkgSS2wr2ydIjDg8 ulb0ThWq/NbS6X5lqaGmn6zu6IKbfwWSX3vLp0eLAeAr3Ue3Dif4Fen7adboVvdM5kPM ZMpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984213; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9IOPAG+bqxWxKt0wdRgDKHhdCXWwpmJ9UigZO44sKyE=; b=FTOsBoz8Tx2Qx6wddpFihCG/g/SQJ++qzq1L6QrPXxCyC3bmnGkH2iQMxk0ucwKSBs KJND7RocvFNEkpWTJ0EyO1mQr/Yn9accKHVoKwrq3oAosRbiiujjqwHWoyGXp02vlymJ Hdsu/bad8Z/Ftl/DE8GYvzXX4guebSHK0ZP691xA4lSUeoI6Bv9npTwyV50CCfFRBneI kEWGrKy2tU5Cm2wiOwAQrJ98ekf8Ks1ic4LWj6IsUSVoYA9NiJxkAkthQLyTS7H148bP qXCue4RJunWtUJLukRfcx6ZRoRESPnbvytl70JH3uoYhT0/QbKuv4NzErG4ePk4SjjKp Wptg== X-Gm-Message-State: AAQBX9eBRtWXXcZh/qQgfv3kqagFbdAEV5mq7oadtYgPo4qRxU0Wi4b6 nPiCr/EoYScBKWAmGKwUlxyhL9fAAFwgQx+7 X-Google-Smtp-Source: AKy350aQyFDG3vc2bblz86MnnpZXVqNeZCl/Rb1kLJ1qs2lbXDpCUugeQMSOiQHvyL07sX6z7bmDfgTHunmuYJ4D X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:4a41:0:b0:507:46cb:f45b with SMTP id j1-20020a634a41000000b0050746cbf45bmr3969874pgl.1.1679984212630; Mon, 27 Mar 2023 23:16:52 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:35 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-7-yosryahmed@google.com> Subject: [PATCH v1 6/9] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Refactor the code to make mem_cgroup_flush_stats() non-atomic (aka sleepable), and provide a separate atomic version. The atomic version is used in reclaim, refault, writeback, and in mem_cgroup_usage(). All other code paths are left to use the non-atomic version. This includes callbacks for userspace reads and the periodic flusher. Since refault is the only caller of mem_cgroup_flush_stats_ratelimited(), this function is changed to call the atomic version of mem_cgroup_flush_stats(). Reclaim and refault code paths are modified to do non-atomic flushing in separate later patches -- so mem_cgroup_flush_stats_ratelimited() will eventually become non-atomic. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt --- include/linux/memcontrol.h | 5 ++++ mm/memcontrol.c | 58 ++++++++++++++++++++++++++++++++------ mm/vmscan.c | 2 +- 3 files changed, 55 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ac3f3b3a45e2..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,6 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); +void mem_cgroup_flush_stats_atomic(void); void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, @@ -1535,6 +1536,10 @@ static inline void mem_cgroup_flush_stats(void) { } +static inline void mem_cgroup_flush_stats_atomic(void) +{ +} + static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 64ff33e02c96..57e8cbf701f3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static bool mem_cgroup_pre_stats_flush(void) { /* * We always flush the entire tree, so concurrent flushers can just @@ -642,24 +642,57 @@ static void __mem_cgroup_flush_stats(void) * from memcg flushers (e.g. reclaim, refault, etc). */ if (atomic_xchg(&stats_flush_ongoing, 1)) - return; + return false; WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + return true; +} + +static void mem_cgroup_post_stats_flush(void) +{ atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } -void mem_cgroup_flush_stats(void) +static bool mem_cgroup_should_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + return atomic_read(&stats_flush_threshold) > num_online_cpus(); +} + +/* atomic functions, safe to call from any context */ +static void __mem_cgroup_flush_stats_atomic(void) +{ + if (mem_cgroup_pre_stats_flush()) { + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + mem_cgroup_post_stats_flush(); + } +} + +void mem_cgroup_flush_stats_atomic(void) +{ + if (mem_cgroup_should_flush_stats()) + __mem_cgroup_flush_stats_atomic(); } void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); +} + +/* non-atomic functions, only safe from sleepable contexts */ +static void __mem_cgroup_flush_stats(void) +{ + if (mem_cgroup_pre_stats_flush()) { + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + mem_cgroup_post_stats_flush(); + } +} + +void mem_cgroup_flush_stats(void) +{ + if (mem_cgroup_should_flush_stats()) + __mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -3684,9 +3717,12 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) * done from irq context; use stale stats in this case. * Arguably, usage threshold events are not reliable on the root * memcg anyway since its usage is ill-defined. + * + * Additionally, other call paths through memcg_check_events() + * disable irqs, so make sure we are flushing stats atomically. */ if (in_task()) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4609,7 +4645,11 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. + */ + mem_cgroup_flush_stats_atomic(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..a9511ccb936f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); /* * Determine the scan balance between anon and file LRUs. From patchwork Tue Mar 28 06:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190550 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63795C761A6 for ; Tue, 28 Mar 2023 06:17:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232692AbjC1GRq (ORCPT ); Tue, 28 Mar 2023 02:17:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232573AbjC1GRX (ORCPT ); Tue, 28 Mar 2023 02:17:23 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 272363AAB for ; Mon, 27 Mar 2023 23:16:55 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id h4-20020a170902f54400b001a1f5f00f3fso7039788plf.2 for ; Mon, 27 Mar 2023 23:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=qIxeFfAv7LZhUaMY9U0TvTtrAPX4wRqNbnIeY0Yrn/RnFRGntBTlu0hfuIAsMfAy2n 1P28EpdkOB3njh6q599A6aRPcbQ+/6Ngp2xMzs6rBLHlr6vd8s4xgEvIn8RMUD6Mze8g o6DwJQ//Mohpbdf8YD2EiOxbMyBh8HdrF+/Yzlad7E0D+Fg0j+P6NKtF5T845D/HLRrp rP0WF5ztka030C2opL/oGdjy+y9vRWrHiXRiQXH/e0QgpKRtMDaGNueTHyNdIDcH2xFM Q3P8IwnLdzShQrK/qG/mnGtQghgE6XGNQRvgt1djyelN/qv5GiJdyRld4l0gXlsYj6Rl orEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=Kzdv5SiN4xHHxeZOKzC/32LQ2S2lGFf19H7RvkIARmSvHevTnBzJ3N0ur/xUGlc+ht +o/xvsEF+XAxVHi5pTxVwSA6FIIIfoDV0w0Wp/Z/SWZCvlchxMDLgPdmk7+3liLFk0SO h1tX4kV1UeCWHB/qKAPhhx7G2uuSE48Pj8kk6xX00N2PiNUNyo+/DfA8vKXEZ7mduT7H ru8Rla+DJuKZc6HWriCO2s43QWLhQLqfmKNpvvw4xzCgDsvSUVuixVz6wUuguDXBE3GR 6dVTfj4jtsfAlekbxQ0y4EJORQ8NcJD4JcrINiOQuSF2ZBpX4/q4uWDMEiQyhSbeaGkN qGPQ== X-Gm-Message-State: AAQBX9fXS3tgKN0q9ZJcUWaqQjUDiLWkdRjYYVR1+089JAWy0o7FXT1u zSMkr+iEHKazkDvrVe5LiI8qgD99/sEk1AqJ X-Google-Smtp-Source: AKy350aBuKbfjhOAuCJYm2iwk4HKu8S/jfcCuw8uP63TY3FEEFp0rTq/+r6BOwoVOwDyCp4xkVINZeVu5jFTXH9F X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:115a:0:b0:50b:dfd4:b56f with SMTP id 26-20020a63115a000000b0050bdfd4b56fmr3747909pgr.5.1679984214584; Mon, 27 Mar 2023 23:16:54 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:36 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-8-yosryahmed@google.com> Subject: [PATCH v1 7/9] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org In workingset_refault(), we call mem_cgroup_flush_stats_ratelimited() to flush stats within an RCU read section and with sleeping disallowed. Move the call to mem_cgroup_flush_stats_ratelimited() above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_ratelimited(), just make it call the non-atomic mem_cgroup_flush_stats(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 12 ++++++------ mm/workingset.c | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 57e8cbf701f3..0c0e74188e90 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -674,12 +674,6 @@ void mem_cgroup_flush_stats_atomic(void) __mem_cgroup_flush_stats_atomic(); } -void mem_cgroup_flush_stats_ratelimited(void) -{ - if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); -} - /* non-atomic functions, only safe from sleepable contexts */ static void __mem_cgroup_flush_stats(void) { @@ -695,6 +689,12 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } +void mem_cgroup_flush_stats_ratelimited(void) +{ + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) + mem_cgroup_flush_stats(); +} + static void flush_memcg_stats_dwork(struct work_struct *w) { __mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index af862c6738c3..7d7ecc46521c 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,8 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +463,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 06:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190551 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B18EC76196 for ; Tue, 28 Mar 2023 06:17:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232711AbjC1GRz (ORCPT ); Tue, 28 Mar 2023 02:17:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232598AbjC1GRZ (ORCPT ); Tue, 28 Mar 2023 02:17:25 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DAD822136 for ; Mon, 27 Mar 2023 23:16:56 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id q30-20020a17090a17a100b0023d376ac2c5so2971894pja.5 for ; Mon, 27 Mar 2023 23:16:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984216; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=x7IFy8THM3GBUAgd4d/GEKRMIpmDkqCRA5AxR/3TFCM=; b=BGZjnL6U27C3nu3nfjJxe/VV851RC/VP1GsLD606AgPd6FshXe6WNQk0I659Owa2wU gVJoC9Crt60CfuYM+ql+MgbEx44waD/ankuLiQjCR+6oc5mr8Ex9gHFcY/MTbuN2MESv WG+o2vmHU2skqPZDZWhEEvumwbMUFH8XB/2P7lCWIxY9j69P+XyQb7YmlC5lJf3dl83N R/+G9MzJdyv2aInyc3KBZTsCKswAeBzT3OW0sicqNfQnZRl6LhPxfTDtAjqxejZWrmX4 kekpZffHaXAsR+R6/xnXuXgj0ngR8GdA98eO9lk+Gv9Mx4rtmu3ZFp7UhRacTBWLo1ks cDtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984216; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=x7IFy8THM3GBUAgd4d/GEKRMIpmDkqCRA5AxR/3TFCM=; b=x0KpXDcPb7aed14pcpVgn+r2UFPj5rdLul9hcEFLHKSDTIf/NklxsVDN3EblRXzhqM NbQ+8TR0BYJ8Fqg9bQT2t83HzIxLIXn2JFQmZcrqMNmr+KWplLmgzFyGOB3N6Wt673bm Q8ZGzmmNiHclSx6C09m2FUOOG+bcB7aCPzM+n0XAvFTtzEydMAqRBDY9p3zsgjIqLxbJ CQRADSK/CmoJbUmIbZ8gqlmk588hEHSiE0HrdVGS4D5fOMyLX0PbbaDonxKIAjfAreT3 gZoEBOKPE2wYrmOdOPtLLtzwDZ1Ij4Xg6h/b+hUNDgo833m/8z+v5ALzs0ECQ2AfRFy3 Rqxw== X-Gm-Message-State: AAQBX9dSYxhprs7GtibeYX9Vrb+YbaQazOTFfJ91ApzoVlOCat6SWXm4 JSA7wkVRCy9r5D4nnSojJHerHDEXZ6ERLPSw X-Google-Smtp-Source: AKy350ZXKLWX99WV75Zn5bhPto8WUfX5S0Ha1cMIMK8Z1sFTdLjrJB9JMxzEwgWixGxokt8xM3tEMiwbJzx2bThN X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:c49:0:b0:502:fd71:d58c with SMTP id 9-20020a630c49000000b00502fd71d58cmr3919025pgm.9.1679984216134; Mon, 27 Mar 2023 23:16:56 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:37 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-9-yosryahmed@google.com> Subject: [PATCH v1 8/9] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Memory reclaim is a sleepable context. Allow sleeping when flushing memcg stats to avoid unnecessarily performing a lot of work without sleeping. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9511ccb936f..9c1c5e8b24b8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); /* * Determine the scan balance between anon and file LRUs. From patchwork Tue Mar 28 06:16:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190552 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BED13C77B61 for ; Tue, 28 Mar 2023 06:18:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232752AbjC1GSB (ORCPT ); Tue, 28 Mar 2023 02:18:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232642AbjC1GRj (ORCPT ); Tue, 28 Mar 2023 02:17:39 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 766593595 for ; Mon, 27 Mar 2023 23:16:59 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id w135-20020a62828d000000b0062c4eb40ddeso3879511pfd.3 for ; Mon, 27 Mar 2023 23:16:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984218; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Vy8wz9BoJKucmcMBXvIGPOuITeThi///HtllaZbgtCo=; b=G3C3U8FV5xO4vsplclhU9+4PkBH686vIIIrQFFquICu8//oGn5xL/Tmt4VNVE1HnMt hORJK4mB81itHhGs1+plTWPpHn6PXb4UvAXp2In5M38sRtIYXUydb86gW9UCcwU8NUMK AKSJULzbSmJ/Zh8xR8z20CA4EuuFhupmK6yUb7XM0dWX+orGZyC4roBLz+bFSVl/SrGz jDFcau8INkey+e5jkuXnym7vXagjhy1slym0NeO+iUEtBeJGIjBlnt29eYQhaZCdCsb9 TItPIolnMlslNIl39r9v4yrb91hLywelng+1GUeMlXmGbk+u0V1YHrp34uG87/cMCuf4 DKLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984218; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Vy8wz9BoJKucmcMBXvIGPOuITeThi///HtllaZbgtCo=; b=hiJuO2NgiQWcOxgQTCfzE8zc2zU0Frr8a0Xmg58I/SvPegjntQW+gcc8ehdNBNrqFz q/5K9VC94Zud70WrD1PJQBI8YmVTnDQ4m1RTbvYyl4StaTUa4jq5BR5MG2D+es3xzgms nIIECBHZvV+iFZQQgXOIHwbScRClGLleKmSWUei74PskyAIAbvchB1BFHOjiSVqBmEYK b28eRHLdz4tSZlREO+3i0MRAHAJSu4N1J0UpUqgayqJFaHSrB7UL4EZCcdlBEeWVa632 Xyje09sejRiThigPl7W7rczhZ5MBFucTDoyNi1x1gz3RI6iSUBBP2+7KQ78sP+AaWXce 8ifw== X-Gm-Message-State: AAQBX9eMpQdl6WuBQnc3/ve/1tVQ/tdFzDjNJtlvgzEk5B6wIYFXV22U cu6QZjag5d/CZlNiwe/2wgNM0e8ePzRr4XgS X-Google-Smtp-Source: AKy350aVn7i51pwbUpBHdDCfopSzB0y7+YNrEoZI1+8jV5mnjDSATrQRcelVSU3ikfIyno7DP54WO5q/0DmrmRgv X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:1781:b0:593:fcfb:208b with SMTP id s1-20020a056a00178100b00593fcfb208bmr7364360pfg.3.1679984218020; Mon, 27 Mar 2023 23:16:58 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:38 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-10-yosryahmed@google.com> Subject: [PATCH v1 9/9] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0c0e74188e90..828e670e721a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));