From patchwork Tue Mar 28 22:16:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5034DC76196 for ; Tue, 28 Mar 2023 22:17:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30AA86B0080; Tue, 28 Mar 2023 18:17:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21EF96B0081; Tue, 28 Mar 2023 18:17:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 097BA6B0082; Tue, 28 Mar 2023 18:17:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E47F76B0080 for ; Tue, 28 Mar 2023 18:17:05 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C82C740B3A for ; Tue, 28 Mar 2023 22:17:05 +0000 (UTC) X-FDA: 80619718410.27.4B7105C Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 140564000C for ; Tue, 28 Mar 2023 22:17:03 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=j2h7VlEd; spf=pass (imf11.hostedemail.com: domain of 3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a7F6fk9fZnPdw9ddOVIzAjBdiTpqd88vdaggqxhF/qs=; b=HBVoOEL3Vv+dVdM+XgmwNukui/JBRuImZuSf5wXocBdicvCMSXEQdJ6KBNtLtECYxir2FE ogrhblvwfsYG7ozuid8DfVo4EgQFeV1WDV2SMD4X+uGQweZyERD+bgYMoQ2gQONa+GeXBA +xIrPFS7RcHmGC7zxUGv5rgGL/MLYx4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=j2h7VlEd; spf=pass (imf11.hostedemail.com: domain of 3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041824; a=rsa-sha256; cv=none; b=RTktQ4zUW9CQo7sAF+SnBfVtGC0qezLSMm0pAHtPzMvu5ZjXGshTHC48+5YNgCWFaNdEs0 PclHbt85aMdU0pLJaelY5w81wx1BVABvsy4qvcBpRQUYmkOheGet/MI7pT8+EDm609VqEF bxSanzzfGfvILEMzGtLQ2hMi5AFf5GM= Received: by mail-pl1-f202.google.com with SMTP id k3-20020a170902ce0300b0019ca6e66303so8351661plg.18 for ; Tue, 28 Mar 2023 15:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=a7F6fk9fZnPdw9ddOVIzAjBdiTpqd88vdaggqxhF/qs=; b=j2h7VlEdeGIL4dqtCRH6krZRXCzlm8zdD/VKFrhl2Ix5hroKwD9u6hjC+9l9HOi6UV H1JPSNu1LdgWvdxHE9hg+SM93x1Jht2L9pd8vgLQEuBZlRpZngZUgkiaZwaZa4gCkUC+ zgspBsM6/B8TFiwy4IbbIfRW24OLGYSFku2vitXovcEwbuAN1td3zyOf9AyMfligZ5Lp BhTKdBAPhJ+a7QI4Cj7LZxYyn4A2yXYMpirZryPEzjspDaCYUkIzUrMU1Kd6ErRoShgT XFMZOBTc6edK5ZIe1IhscJC8wUQ57Xb89+9zAOCcDb3JVfb8T7SfV7MPXlrp4JguLYCz aCRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=a7F6fk9fZnPdw9ddOVIzAjBdiTpqd88vdaggqxhF/qs=; b=SQVgVKhQkv9+WDCj1aHG1IZ1s3FSqYQvZV6ZevhaqjsrDh9qhh31yHuoSpK249CxXw fuNtHsYBz16b+5zeOcf9WfwlmSxX5dG4apdZ6Ql7W4xJLGwMLNWK4H2T5m131Z4PMnI0 zHx2gHnuCQujoxXg7G2DQuZa3/agipnHQGIxI3nyo9a85yVaJzyzza94E4HP/ZJv5qOu LAqHNZrUk/pBNZEikX6NGnkmGaZabuDDuf7taQcJ9ij81Q+YKIYY5NsSclmVqFHX1RZ9 EKoXVFXBXKADAx1wRnMeKNrRjsmAZG/zvaTfXl6RNmmapZJH0+88ldBAKH1Bb+l+kUbQ nRlg== X-Gm-Message-State: AAQBX9eXu0Jo8PSpi8+QzYkk1A+kpoKJ/VdbJ8o5prUR36FO9QB29IZu 5AlcJqsufivXtM9P/r1dKRzHF5zNs7z+oTG5 X-Google-Smtp-Source: AKy350YX9lhtezGewKcmJEZ6+PWl0qdkFOVRrH2qPmewBQAi0Jj2O6028tHTh6zpbuIXp3Yq/V52VHvowCQfk3sU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:ec90:b0:1a0:763d:6c2a with SMTP id x16-20020a170902ec9000b001a0763d6c2amr6633613plg.10.1680041823533; Tue, 28 Mar 2023 15:17:03 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:42 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-8-yosryahmed@google.com> Subject: [PATCH v2 7/9] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 140564000C X-Stat-Signature: rt6y6zyj95sunfgkirizmfxqxck4pir4 X-HE-Tag: 1680041823-269731 X-HE-Meta: U2FsdGVkX1/5ZaYk5qBOFr+5ecwgEGw0InDzox3Ae4bEJ2JtDjUugGUhEIM4ODSF9YBtBfphyNP2ZYqTaQat2q29tG7dw7x1qU1ekMDAXC5zDOwaRynm1wcGMmn13s4EWtCiqRmUkCe7RBzMJ7g8UKDrdB8Zur+uzdyWEgxplngfL7RNi9NAlSepa3N+38Ihzk9U8/t+QgdAy1fSo6Ci2yuODaH7dutvLVzgQsG/i6G00YhHM5oJMFBl5ll7J4X+3EmCWFHNnvfdhaIbyE3gVjQSy5Cys5npO3KR703h7SEYJV6Egbk5Y0TVj9vhbwPA/A+y+KLFCFWl8XSIllNDB1II+fP0i2EJ7/LSgZ+ACWH11wOIxQ5Pap1NnUyPWX1rMRnQLKt7BWAhx8ZaZP/5n2pL8Hz8Q6dhJdoPHIxU0uPgGzLOWpXJK1cCrhtiJk2yswmOaErxUsoxZj3jkQ40354F9mUGiTMv18njqPIWqTAWUazQ4snshUGPo1hzz6IOpZpvthC7fiL75+ovYu0GA1A9PYFxpDrq+08ZA3ujri0zGI0iiV36TfVaz1cL+qX18T8b4EyofmvIzZKX67izryq8UkfkhBYN4wQRILZoR6i6a6rOJUGv9lwEySSpOmGHhgqg1JIOxrpBbVb2HUrbP7d6lJEe/LvpZwDxjIIRbPBr+Czk/J8KG4PDuLGqBRyPZJhw8/684DeznV41D7rZH5jfPXOxkmgDL/GJHBIweBfYvLgv1FRMK2Y9C0f4DvRmNcj2F5sURHxkTmosx4/nKO461y1+3KYQJVFEODFDl27G6Ht/EYMkrROwUncGnadWjmsEo1sTbhFIK65NEgQQhUIXrGK3/RJ1+sAi7umEuh9jEZJIweUu/CVoGdvhCHGR5tMGm8+MNhnMKDmhd1H3hZwAhfYr4oXPGvsdp+DZvFEUXejg6Ait30Vmw4VYpAoHkaoPDkx/frA1pby9T+9 aEBlMbWL 8FsyIXIY7fzIc5QMWdGKBLdm7Jvf6vFd642Z+GIgfCdIsk4BlUA5+FlXDPFKkWPnJrdotVsHTParGEDPIs+iVpAXHIEDUKMq6rs+2p1T/bnk9Y0/NK55fpGwvfw0rCaEF8Q9mkzaIZkOL9GY8+py/HOZDNU2+/Nu5ifFDQ27DoatyWcbdXzLjWxl2RqV674z4y0rzVy+tfc0PYLcZ4ZKTK2xbfGyYF0hDyijTpIL9SjnwCfWVI8zd6hmrZnbDTlXyz82KKGOMRNwE+9k2kjDE8MbPO7ETqsh2ZnBA5+RdRBCd0iOsQly8lvRh2JfIT1AsBvpJEsUya1pbHDWwogJVcv47sGkNNvGZYaBELfhQbZe4aI+8h3rxXNkc3s04Cgz506TTxVPEUERQao7KGMLCoxsoc4/SUMUamoMIV6Rj29taLaOmHtwDo//BiD6HjyWiTYDae/xxPRJk/xO2uKZ1/L0v3vvCceKC/+KoecN7fXSauYavKjHlWnhkAcwh60aahoTRYEgtErbulosJ670U23a8rKtQundXaqhU+SEgVnGSQxsvrM7XSzq063t+kyR5/+M0ph67vRsSDa4EH/3QBU9no0FB0qZb7E7Ftalb0kbiIVHKywf6iZCu4PCahlk1gF391y6y0CQsg1gNp4DGrzvKh1XwVhuPi3BWsxBcgJdnDI7ziUGIdc6401vSE/PnzawPt51AWokcZ5XCmVW0C5psvTCxRUixwiXoEGp8/sp76JNJQS5D22JUym2UBhNstIx+kAqh3Zw8eAM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In workingset_refault(), we call mem_cgroup_flush_stats_atomic_ratelimited() to flush stats within an RCU read section and with sleeping disallowed. Move the call above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_atomic_ratelimited(), just make it non-atomic, and rename it to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 4 ++-- mm/workingset.c | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b424ba3ebd09..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, void mem_cgroup_flush_stats(void); void mem_cgroup_flush_stats_atomic(void); -void mem_cgroup_flush_stats_atomic_ratelimited(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1540,7 +1540,7 @@ static inline void mem_cgroup_flush_stats_atomic(void) { } -static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a2ce3aa10d94..361c0bbf7283 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -673,10 +673,10 @@ void mem_cgroup_flush_stats_atomic(void) do_flush_stats(true); } -void mem_cgroup_flush_stats_atomic_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) diff --git a/mm/workingset.c b/mm/workingset.c index dab0c362b9e3..3025beee9b34 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,9 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +464,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if