From patchwork Thu Mar 30 19:17:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 017EBC77B60 for ; Thu, 30 Mar 2023 19:18:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 471D9280003; Thu, 30 Mar 2023 15:18:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FB9B280001; Thu, 30 Mar 2023 15:18:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D8DB280003; Thu, 30 Mar 2023 15:18:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EDE25280001 for ; Thu, 30 Mar 2023 15:18:17 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C08EE81169 for ; Thu, 30 Mar 2023 19:18:17 +0000 (UTC) X-FDA: 80626525434.18.D16CC02 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf21.hostedemail.com (Postfix) with ESMTP id EB98D1C000F for ; Thu, 30 Mar 2023 19:18:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EVT1tk0h; spf=pass (imf21.hostedemail.com: domain of 3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=lHSrkDEdcMBMgcu67V9SoOCMYJ+6mYdlALV4J/leHN+wj+JIQof38QzAAkPwUK3u0VgWkc BGwzIowoO9wxXxGIxTxTpXu2cji5KRle67o8rv1ykwpeUzBD4IUo7+4Cg5Wcn7fqjt0CVg JbTkzo8AD4BUQXrjmNfkf8q3id+W0+4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EVT1tk0h; spf=pass (imf21.hostedemail.com: domain of 3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203896; a=rsa-sha256; cv=none; b=SbU3yDIEQ+AL4WsGUCi6x2e4Vf+MrAadrXEs2XbsLUpC9VpmkhMD9Br25cSMzIy4JsZuPz wrPMYWpm+/0ooFZorrcDelxwONEPa7/6on/yb/fSmXAGYAfU2hKqfJAkjRR30rqM4L1BKM 89VXEDBHt/x9o2jQbD8NDkOCbOiu3+4= Received: by mail-pf1-f202.google.com with SMTP id i192-20020a6287c9000000b0062a43acb7faso9111008pfe.8 for ; Thu, 30 Mar 2023 12:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=EVT1tk0hlY3K/RTYvcQHMKPpwQPVXOnpdSgQM68BKQtZmfmON2APgqvMOvMMFyRoss Dmd2H4VUZwWGIfs6kzYLS/yY0LBdZ4oH1VkvSRWDGcSOJqUREz+4yAA6vO0jGdRbUllx xh0AWez/MaDxogJM7rwIORKSy2FNGdepATzkqraYhiO3RCB8HnWcqe3zSlq35IxkzxtC FV1PJ6neoxu760xZAlTGk8nkupYvKz5UcVjEtp57Ggrbl9Lds+WScYdzag5BG/1gW5Yv DUw8gcccaLuVJ49JXssrnmfFrYddyvv++tF0t2hffuUaTqszDGsJ7ZLQoChqmjJGkrqP YfTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=zSMsbuVz9xiyetHcS2v+belY5+KGQALCCrBwGOnxHOld2Jst3VogPGjd4Zf5UYI6DM a7jWf/4V/rhnQj6AFtdHOqhZVuUhO532E3+wD1bwDXcVL6KsDagVo9zLKYjKhzoNsXZT 8kYBaGAVVJm1gHkjtGWvVl4seedXf3VBxgbKtdQhMVJlPg81bSXcWyv/NyVTcKlP4Cu9 v26ry2Fkx5CuapMtF01dTtIp0RNUbmOb8JgMYx1t5DMHHYhJXh99vy5yHpgsU/0mX30S 5VI5N9iQ5fuD+5nUnsFN0JB6tiQb5ihiQnbRT2bIGxy1Fry2HR53+yMOG/cHgRolwG2H HgSg== X-Gm-Message-State: AAQBX9dk/lCkiAQU1JvRKS/ySg2ZsqUmVnYcD9BYR93uQTqnhmWNeut4 ikVdlKTt9Wbuxx4NseN3FIB2AK98nb4e/t7h X-Google-Smtp-Source: AKy350YjU2Y2MgqNkqgZoIOPOW0+HOYv1MqQ0pV2FUPZVEGY798zCaQjIBhDtJyF0YZxiEsyw9QKlZMU8q/qm34J X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:6b44:b0:1a2:6e4d:7832 with SMTP id g4-20020a1709026b4400b001a26e4d7832mr2680127plt.1.1680203894775; Thu, 30 Mar 2023 12:18:14 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:59 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-7-yosryahmed@google.com> Subject: [PATCH v3 6/8] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspamd-Queue-Id: EB98D1C000F X-Stat-Signature: txuccs7586baj5r3d3kkisxtcnhcknzj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680203895-230380 X-HE-Meta: U2FsdGVkX18BWEgioZiUpfvp0h5PVD9rYWw139EdqGfTrr2UptFOAey0FSRM8ytOMyGjsj2e+0UBc8X3OU9HayMYNYJ7M0NEAet1PzWVJK2nuWw07eKG0EoiREh0g8PRbu1qlLltyoR0Zoi0IXSsT+dm9Xm8//3C4lyFMRwjE1xyXVg3BtVFvBE4I7gmEpCwtncZhbLoQT4X/hT9D6jqn1xd7YA2dk4Rj+HYprqIQ4hU3yzoQHQqMhfpcpohH22v7ja8iyohSn6j71jMni0efv2XRljEh99igMIIHWgD8sbyEgc34hMS2YePVbIECcBDScvL1L9hhvWETMrEckMTYxR9knhBNlLOFBJ0ARJyDu9U5iZyrUQ82Ce03KWRfctzW/lQ+kocyA3bVUzYzzMM1FoTDWo1ZSRjMPHvA2Ks8AhgyOJ5cD0ToVKh1Yvy1m49Ohcw2uPqfpK1Mwq/n7Xy1XO5NXKma1FBcId3ZEHsn7sXyGVnBfTJ9H2/uWGYAYrvIMaif6bW9cT404/VoC+cEyuz4P/79kX6Tr0hdgiXCp8ohl0H4YMtDtVdipS6zPsAuswI+Euy+X+9EDiRNE/FTDXZvlojKEq6ZaxsMMLWl5OaDdISeefHl/bovP8tCYOwGB3t2I3aJDhLPmwZoL6vUVxq62mT9tR+x2SdCUZ9qAD2jvw0qVLDuJxDvNfYWF1neB1oPS8risX6wsLPtYS7lrDirsGouQZdQT7O6fznXhdsFAFGIJWrnxvH4kAzA4Ra19tF8hTCKkYlg0jRa6HLW5OAlSvOKPhN1bMto4q3liJpq6TL5uhHPVlXTBBJR3F7IBlZeJdW1yKmtmIqD0Xw6pLvyWL3gEcYLpMsijqr38Zax/iAqOKN58R/javANqt0IwL75+qGj6HDahKoiJzlLTcHyLc0wmiVUJdsyqIUuzR7XFK3NQlqLF3gbvyw4VM/xV8hBBaqTJyP/5QXbHk tbA1IpEF bgwynuX8jGDTKBAyVSR1Y9QPhjmm7i+19q9/10Qbye4qvvqLgo6T1P/Qy0OyDCwU8bOjTK2NpMro7spgpldVjg8HgEiX1AESwBLtQs4MZzeZOZHqMgm0aF0H3OXznaoUZtY4TeXcaZM18OQApRd2aR+U3Lo6/zFJex4Sj6IVYTGTCMpo4SM5gfN3PkwCTl3LFWry4KuQtK8KQ7KHtFwJb1fzckS5wq255nPexEDoXVci1EHzjlTugmerx/nuGwWgOdZO/ysYTLJ3WrRFE6s3I8hLoTs0FRdkXcjekQDUPY8o80PdLZeOFxXfxUmLQeIwY8gYccn2wPTq7uc23v0foraI7vp04vZflgyj0QBECfbNPcnPvpCZ5SQUKPYPJdkF4agOZex0f7lBd5EtHPbahk4zSgw9EALVMQ64u0J7U9Hh7aA6YvAOHJolFGXX87/z0tPhSTCw8iikDyVPEgzg2IK5ldFDYQlPwC5kkO/KLGY/8cYcLf2FkSGpD/dz/Ge4nRNlBZPh129kY4UdB+UOhmT+4BXRbk33IUiwLWQa3VctG5m8rSQ1WdPeWzfha79t/osZtwdouRfLvZa6WqgyULFu6kd7hDcc6p+U41H/lcPPob4ZchAcif5WNzvcNv5XkgGHFzTHfucrVzmZFjxpisuKNVJsyNivQVZqsLQZ3YsPyMN91I2IJg0M8n3y6DTrDT1hNFG5yRAXHYZYKsS28Dfdl5bIfLX9/4QtRdP1hVfkyyPuIYmPRxDBqHHFYTsJFK2O6I2DzkaS4uVfHOqvde1/NnLUv6z77W0k8jKjD/naTJLs4cdw7704htOOKBuW+Q7Sr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In workingset_refault(), we call mem_cgroup_flush_stats_atomic_ratelimited() to read accurate stats within an RCU read section and with sleeping disallowed. Move the call above the RCU read section to make it non-atomic. Flushing is an expensive operation that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically where possible. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_atomic_ratelimited(), just make it non-atomic, and rename it to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 4 ++-- mm/workingset.c | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b424ba3ebd09..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, void mem_cgroup_flush_stats(void); void mem_cgroup_flush_stats_atomic(void); -void mem_cgroup_flush_stats_atomic_ratelimited(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1540,7 +1540,7 @@ static inline void mem_cgroup_flush_stats_atomic(void) { } -static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a2ce3aa10d94..361c0bbf7283 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -673,10 +673,10 @@ void mem_cgroup_flush_stats_atomic(void) do_flush_stats(true); } -void mem_cgroup_flush_stats_atomic_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) diff --git a/mm/workingset.c b/mm/workingset.c index dab0c362b9e3..3025beee9b34 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,9 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +464,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if