From patchwork Thu Sep 21 08:10:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13393761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D739E706E7 for ; Thu, 21 Sep 2023 08:11:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB2CD6B01F7; Thu, 21 Sep 2023 04:11:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C63DC6B01F9; Thu, 21 Sep 2023 04:11:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A66E06B01FB; Thu, 21 Sep 2023 04:11:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 923596B01F7 for ; Thu, 21 Sep 2023 04:11:11 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6BA3E16109A for ; Thu, 21 Sep 2023 08:11:11 +0000 (UTC) X-FDA: 81259884342.05.D17E841 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf09.hostedemail.com (Postfix) with ESMTP id A9CB1140020 for ; Thu, 21 Sep 2023 08:11:09 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BVZJrS9n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3nPoLZQoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3nPoLZQoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695283869; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ap3WQf9RL9Y6q/AtYDIGLhJT98YcDdn8KWtOOFcbuqo=; b=hex0mehM8w2boyv+XCrlD9ZvOTRMnaW5/mmdDmoLLG9riQwcHkxL0k5jZjRlSS8ojjZJ2F nTCPwHBQ5AlKsGi5HcU+ZG15X/Kqfjj7PEpiFKyCvadizJ3HzXEYdh0xWbU62eW6Q2vjnm p65+Tww0gqQgEjAT2gUpugLbSvA1XWc= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BVZJrS9n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3nPoLZQoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3nPoLZQoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695283869; a=rsa-sha256; cv=none; b=f817tM4BPUdNzXRUOYHmqMGKrSB51BRFaiPjUCDAyTWYsN5y7iyUymN9ysLWzah3rwPZ5J FfJsGYBAevsmW6Sr0N6Z8tBgNoqfDdbpa9ZFSuhS+wcN9FvPeelyIe8oqDh3aY7JQUuipV kdpdFYDrJ1CHf2nOf4NBVUv4nEvrLHE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-59c15767524so10553707b3.0 for ; Thu, 21 Sep 2023 01:11:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695283869; x=1695888669; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ap3WQf9RL9Y6q/AtYDIGLhJT98YcDdn8KWtOOFcbuqo=; b=BVZJrS9nORw8w3QsuSO9GIrrSPJ9hC5vnEes6M8V5VbZoz1HNIcSKrdmJ/41SrspnY F8dGlEME9e/DUVASxPTxVy6mZkNTK27LRg6W20fu+3oI1eqb4K9NbrnHwjr3JUc39bb2 LO5VsY/YRBPWqXgyMaNRMAQDXXVLp3a7muK9gFZ3hFEh9eI89nvhufZoi5m2fqEN/hfF AkPRg2efUjhBDoipFZIBFqpIQuktcsHxhTwN9AvqwG424lk1/qxyuE9EgRUxbKkXnAto 16WdK4+EJpJFGiHQ72yv3Hk6r2oV3Jr98ZKaqoAJhi5evtdlbsGnOzipYm0Mgd53hcOj 1x4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695283869; x=1695888669; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ap3WQf9RL9Y6q/AtYDIGLhJT98YcDdn8KWtOOFcbuqo=; b=etRR6kFxdwZalrcAQESd13eLkff6LTB1G4CH/rpNaavMsiwdEq2WAsfjN4TodvtMkI vzw3VyuZ8OMDiGg1aIbh+gzUh1rb7QJ5h7Jma+h8tK/BCHcwVdCHxdzWq7r5SH9FLm3b rcuIMgIYeU9JJgeC4P01v5LbakI5vf/blHXlci41RrIlLBx8UQpLwpGtSY/SKZgi4WJB XUZSlqe6nhazw7UYrgFuOhqFWG432sWMFchOnhYNH24ZRHmA3MMkIHKj+g8Juop5kXRQ +z0yf8xnX8/82J6DUvtU+BE1fqFrp6DDOA1xnZyXsgZAdZ4jvJSz9Bm3AfTw1o3JB9Je TGDw== X-Gm-Message-State: AOJu0YxKPWwycEdNvmG9Da9e+P1iKA5m5b7IqLgT6t+RCYgGb4o+vTyd x27DfWI8LA2EH/xa8PxeW+mxjKnpPbJoSTcn X-Google-Smtp-Source: AGHT+IEctDDQF0RHmq3JRn62J6oQCd8whI4eyWsA+THF8MJIpkdXC4/+HB3bqA3JcJDKn3Dhk9UKpKL30JXlgF3v X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a25:504d:0:b0:d62:7f3f:621d with SMTP id e74-20020a25504d000000b00d627f3f621dmr75596ybb.11.1695283868835; Thu, 21 Sep 2023 01:11:08 -0700 (PDT) Date: Thu, 21 Sep 2023 08:10:56 +0000 In-Reply-To: <20230921081057.3440885-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230921081057.3440885-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.459.ge4e396fd5e-goog Message-ID: <20230921081057.3440885-5-yosryahmed@google.com> Subject: [PATCH 4/5] mm: workingset: move the stats flush into workingset_test_recent() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A9CB1140020 X-Stat-Signature: 5rbhutqsf3wyi3nt398n5i5a8wawr8a1 X-Rspam-User: X-HE-Tag: 1695283869-623625 X-HE-Meta: U2FsdGVkX1/Zfn+MWNDOMyGbDyBAhCDD3uZTziXbkzBMR0JDL2beTdbnFYml9FdSdnToL3XxeDh5hkX5X7cXycZ0t6SgnLKthgTaHNztyxoQQ93p1/JYWyijHc/gPYtfYSfWSJcdfFdangEd7GFfWX//264JGYiCzO6iaDdVJIfTD8gZTa6CIDbby9WIGEyAQ0qO/X6dSXsL+0oAI9dlkjX2i5A7SUXAnY83fr26dyA3xMSEhy9rWOTdRWdH3KEO/gcooHJu87yUjqiOYoOsLDYPGGa5S1iSXPRmkgeIjAHlj5ygfzde0Hhe8GJnoSJY+3Unm/bs8CtrYqExvDEKZhJX8fFf2wfmVLsaBGN0QT37I4SIRiCfLIDZKEw4zwiKaZ0+9uZdqJUpdETJL/RdHbhuvTrJAwmoncNF5iCChYjFtczVQth83pWTUzCXco5KVPAQ92KnBURt0MhwQgVzdtE+f1N3rMGdz0aVeAnIRE1XmC4fRa0vxezbeMVB9QP8Uuq0BNcJt1A+R2ZTcnnCc9H1kBCnTXYVUxUyj9Chm6xVULCD1SR85m9VbAzALg9hBRXlMSA2lIrvKVSdCGfRNFprHB8qjtHPlQgUMN4WlNmht+5RFdBf5AsI0Ex2wiUu216iiirvw0FsrdAVH7KwcKlmqrgvlwnlccHgo0hRL/UKOcmCvUwvkYAS1JfPgAizHg+lNMZYEOAiaHZ0i9NoKdyyfnHTYeuJNC+EK5WRiQegEqPpCapNlkzxKaY3m+dgg0a7ghS7IgG5QlbdxlRAKZc2neGOhohvEz0dHvUIJ5qimLnxbVhDHi5qOj+eclJmyfxEpiwAgrV36/S0e43khoclkY0+2CZwxsgKGmuc4BiIQ4EjsV5IBZLkNZd+seNT2Rp16uEJoaYUDZrOx9IyqtcX8pWZS2YmgwC7W58dSuCOHRRCYOHnWehiLm3vM6846b8o5h6hu8aHdBc8WG8 PfEwlG4F g4PhBniLwMAHg6uZOPAYxLnvW4dL68yRX6uFCkQStv8NbDFwV+P8Zk+Ri1hidQTvjQc/BKuY2PndKhjEmTiCsD53ycqYtavVOtdbJIB3h90JFKmQGARNmND5aQ310VZr98iSA6mdOGdjxJDv6BGQM4wcIXKS1EDTP74lOc2YP8dp3AI5QA3+H7fMsDrF5n08RwqPFCW4HXtfR0U/Trj/JZKJ9cAFurw7i5LNPR7G+Y5hiI34uM6bL1efhqYeSETNeofnZfrhXh9aStGelG01yItZQWvxdi3NGhnZayGvbr57d0KZ7m31vkzef3CeN6pwQbpvZY00mDKm3DqsJCxItvRThlM/NO64MuJVYml8r4OUYfq5Ve9AxHJBYyUqGxhPWvaq3NAXUDI37WTghNr/qErnocCXYSarrW2ePG5XmG5pR7JTw386epzVdwH0ozKyonm/VG1LOAOlBPqc8wsC9KMcoweSfzWONOguv4onrK2UFBb4bgrHmdJ9XpLvkH2ADlD77+NJ6wR4MjiiKEXzUEAg5bJ1pcITO6fAIopKMMWsniYNye/9hGQPpaZZ3UBjyzB+FCTTbq+v+fGHaJK/6krwRUA8Dp7dv3N51FRztcTu/L4PMXkXveKinKIJmV1yZcyJCENDFbRrecQBMNBu8gMdOQIhP+v7/BJy44LI6veAVYcMUjWmgKDQfAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.004925, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/workingset.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index b192e44a0e7c..79b338996088 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -425,8 +425,15 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, workingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent = lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, + workingset); + rcu_read_unlock(); + return recent; + } unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -451,6 +458,12 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) if (!mem_cgroup_disabled() && !eviction_memcg) return false; + css_get(&eviction_memcg->css); + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); + eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +506,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) } } + mem_cgroup_put(eviction_memcg); return refault_distance <= workingset_size; } @@ -519,19 +533,16 @@ void workingset_refault(struct folio *folio, void *shadow) return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); @@ -540,7 +551,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +567,6 @@ void workingset_refault(struct folio *folio, void *shadow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /**