From patchwork Tue Mar 28 06:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B2C4C76195 for ; Tue, 28 Mar 2023 06:16:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3142280004; Tue, 28 Mar 2023 02:16:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B44E3280001; Tue, 28 Mar 2023 02:16:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 948A9280004; Tue, 28 Mar 2023 02:16:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 86EF5280001 for ; Tue, 28 Mar 2023 02:16:57 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4F34D1A0126 for ; Tue, 28 Mar 2023 06:16:57 +0000 (UTC) X-FDA: 80617298874.19.734A78D Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 8CF29180017 for ; Tue, 28 Mar 2023 06:16:55 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qIxeFfAv; spf=pass (imf16.hostedemail.com: domain of 3VoYiZAoKCBUJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3VoYiZAoKCBUJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679984215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=M2m2HPLwMoGiea4FrF09YTfHf6+iku1imTM1aEJvbVci/LKkivQp3Q8GdCi1uBs+Qx+Rlo Bxgmeg5S3bS+g0nVtQzdS84YVC+SPOAY0hQ+q1fHFEP7dOUa+mW7LPF4WaJgkLzVV8PYAz GGWOuFAghbuLjYO/pQX5jlRZAMTgKws= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qIxeFfAv; spf=pass (imf16.hostedemail.com: domain of 3VoYiZAoKCBUJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3VoYiZAoKCBUJ9DCJv27zy19916z.x97638FI-775Gvx5.9C1@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679984215; a=rsa-sha256; cv=none; b=OvuaZy3fK+bQymHthh0yJ8yFoyNrPDzyASrc8xKEKuR8V2SannjioLNpLyi3WNPmEXpjiQ s+1UjYmLAD+CokSwK1BLX0Vy2x7zkFHd/jYmyRENC4k0+uawFh0EFOMgxlK3epeiku7uLQ x2QSF2B8lpc56/Y/97qrxU8ZIuw3YdA= Received: by mail-pl1-f202.google.com with SMTP id w14-20020a170902e88e00b001a238a5946cso4524582plg.11 for ; Mon, 27 Mar 2023 23:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=qIxeFfAv7LZhUaMY9U0TvTtrAPX4wRqNbnIeY0Yrn/RnFRGntBTlu0hfuIAsMfAy2n 1P28EpdkOB3njh6q599A6aRPcbQ+/6Ngp2xMzs6rBLHlr6vd8s4xgEvIn8RMUD6Mze8g o6DwJQ//Mohpbdf8YD2EiOxbMyBh8HdrF+/Yzlad7E0D+Fg0j+P6NKtF5T845D/HLRrp rP0WF5ztka030C2opL/oGdjy+y9vRWrHiXRiQXH/e0QgpKRtMDaGNueTHyNdIDcH2xFM Q3P8IwnLdzShQrK/qG/mnGtQghgE6XGNQRvgt1djyelN/qv5GiJdyRld4l0gXlsYj6Rl orEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=PUXgYn7FfgUFYHnFl71tO7borZmFe6lelqpmlE+Cahj/RKNipwdE+itovaYpVLAGEB +V0sGtjcqGvCIZH/0jo+34qq44PKnWvIitVg6ilLlTrXXq8AmCGWGUq7gSqktYcysoxU j4ujazA6P61VrRKxkyjsBmpeumgL6G7KwGFGDrQ8o8n9LZ9X8tfavNjfvIxymSwTQIC7 UcL9qZQ8ZsMlGpXqeaM5pG10/lKqXQKorA9azhOsSsQZHNT1XyNpCjzbAG093i2tAb9r nfKJ2II71/+egr6YhEFilzfqzakuMp7IV3T084pFr2YTPJjXp2iMepvxMuxR9txL9qgA J+Jg== X-Gm-Message-State: AAQBX9dFyHCGnxmtPqcG7Z3p4014a1mHTwXCAQzSaV4+GOc/qs/2hK3A jJ+GW783UkYxEPm5tF0Fv40p3NetmMcbr+C2 X-Google-Smtp-Source: AKy350aBuKbfjhOAuCJYm2iwk4HKu8S/jfcCuw8uP63TY3FEEFp0rTq/+r6BOwoVOwDyCp4xkVINZeVu5jFTXH9F X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:115a:0:b0:50b:dfd4:b56f with SMTP id 26-20020a63115a000000b0050bdfd4b56fmr3747909pgr.5.1679984214584; Mon, 27 Mar 2023 23:16:54 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:36 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-8-yosryahmed@google.com> Subject: [PATCH v1 7/9] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 8CF29180017 X-Stat-Signature: pk1neh9e7gosku5z6b3hjnoc7u4j1z5k X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679984215-625486 X-HE-Meta: U2FsdGVkX19RmZfo9QkG0KTzR/WbUrx14Z1fEpNro0DoFP/N2M/It5jxW/DJytRpM+OvDhG/neMIABpcpNysVOzBRbMA+hL0GU/Oqxv2962q5Ql5cbCfIYyE0Ey+q9fqRQTksQ1wRhSvvT2Fbrq55A1EQt/yqmD3zmwiVaJxHUpISTWkWD0gbC4/iUnDqrPCmY5V43Uy0ZqjPt25EGMgSVa5L1HIbSXj4wO9/snTdoUA+SCQ2+82n1pryDtcW7mcm/xJP0VaDzkZSlfUiwdF4LMqhXYZnh+7XFsprvmwEfR5jqNPmOKxTvyf3TR07a5FoXHcIjmVVuGd4fcLxgrjyc/4sAYbLzCEuLi59ihNdTTC+qcob105UsG+ANcnLlFw3U2ao8FcN4cCM7vL7JDId5mPWHcjAoyolmfeSAVfk4eKFfsGdd7ynwgjhLEmODVqAxRRpzh6D6TI2NmqvuanbbDbfD4YWTv3RYcaYZLTase4k157x98ko4VX5pU6PLJyabQe3TgyH3GmE8Amb0tiOVm8VE87Ws+PXIjuP6tkqq209x01JYf7LvwGTAAjwotlWMFSvA2EfLn5W6YaQ8dqc4YcI2/qVhEd87oObBBH8RNlOglUaOX8zZq9j/fRuDc+/aoKFwrg6Ye+676Wt+XWnFb1JDDWgbo7g/XqGM/neqzJWUbUP9It9XRX1HGMYurSpdVbN7Xb2vpXIQkB8+Nkx+i2F/11YpyjDFEoMW5NAgKHYfmzE11EYvmqCgG35PMo4VarttVSeHUtZ1BN8FZ9sED0QPX+gKhO500EagUbQznmYHarDtADjFtch9ALgkTlNWW6i2X5BqTmMVjQHQjbeFOqDfK2cmt1OVzaiCm9PS7SjF0sOoJu2O466F9M7D2IBGQaUg81UXPHR36vxBJeWtSVdx8baWL1Klxr8QiVZAe8sTaHx5OjXQF8Q7387+YYzGcrO0d82o5mcpYRAhP JKW8oTyi Zlw8ovXobsHIOMDjsTPEJkK9RF5QOIEOV5z+jPIDMfmXCEDvBGzWeIVeZVRFN5HaT7u69qFhzE3FqS1NV9KFSgRbkk8BcWFS0keISAi0A5/3pINfjtO8n+VdJcyfjcFYrPfW8twhAIFzlQ4H5RadHMUTxk5zPxPNwwe2IXqSehIZbjkzI+UiVpsE0UaAyXl8WXG4T32ZvmYeLjqeawhEqP9nNZZKkYiiii3pl+sWflT1MeLP4zxWJRoQ9fGqrz2Xf56imsDLF/FtjGxuvrORpeaLqySfNusDugJF+NZq2fSeMLcxcXEIgvrzPzZW35HMNKqhZYX9cdplDFQ+MaZajHT78/UGqC7LWvtnNvfsBGIwrfc5uD0owYAbFct6Tm0UwVLPGz88j+pXxbg+ClWUvA8s9/elSULln5bV4IMq5X8lZtsyyDMkBTKZKM4gg3k6a46BIzZAP5PbDDxgcgloxXu8n+agnzUE6XxIgCEfuEuUqaNVopTIgcIoh620yKZDY68BiypWAfcPNNpmuBtJ16asVB0/v7UNQ2RUkfS8PwyvSeyLiInoreZjI6LNrudoWOZQpaIcA55cEf+joseNIhfOYSPcOpGh4zQMcoMIQGIbOBGxUSzyhn6GLmHpzGIhovaoZU3o7kEEWm7nVggquO+USpRrqimxh1ni4jnRUACqpgCPjLM0+rA0zLmdm8e/ayl8h6Ic0kIu+7es= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In workingset_refault(), we call mem_cgroup_flush_stats_ratelimited() to flush stats within an RCU read section and with sleeping disallowed. Move the call to mem_cgroup_flush_stats_ratelimited() above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_ratelimited(), just make it call the non-atomic mem_cgroup_flush_stats(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 12 ++++++------ mm/workingset.c | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 57e8cbf701f3..0c0e74188e90 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -674,12 +674,6 @@ void mem_cgroup_flush_stats_atomic(void) __mem_cgroup_flush_stats_atomic(); } -void mem_cgroup_flush_stats_ratelimited(void) -{ - if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); -} - /* non-atomic functions, only safe from sleepable contexts */ static void __mem_cgroup_flush_stats(void) { @@ -695,6 +689,12 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } +void mem_cgroup_flush_stats_ratelimited(void) +{ + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) + mem_cgroup_flush_stats(); +} + static void flush_memcg_stats_dwork(struct work_struct *w) { __mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index af862c6738c3..7d7ecc46521c 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,8 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +463,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if