From patchwork Sun Feb 19 07:33:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13145802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14DDAC636CC for ; Sun, 19 Feb 2023 07:33:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71FCF6B0073; Sun, 19 Feb 2023 02:33:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CFF46B0074; Sun, 19 Feb 2023 02:33:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 572C56B0075; Sun, 19 Feb 2023 02:33:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 470436B0073 for ; Sun, 19 Feb 2023 02:33:35 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 01FE6A1419 for ; Sun, 19 Feb 2023 07:33:34 +0000 (UTC) X-FDA: 80483226390.22.3C7DC23 Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) by imf19.hostedemail.com (Postfix) with ESMTP id 327C41A000B for ; Sun, 19 Feb 2023 07:33:32 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=M942xidG; spf=pass (imf19.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.160.48 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676792013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=foTbT4QwdeCG8p/+pG7+6hYxA8iHmzjKmefWFVB0gjc=; b=ewfRoQd8EPC+Z5ADTr6ZQC9E+ErD9gx4Bb7gGhX/Z2MT1iRAslI7G25VPhZjWoZDTLwuKr QVsH9rIGFdXROjgzcSaETeCn8Q5dpAr4o3YpHb2Sb5X6cOLIp+qaZZQ+lvVURmLFxSQL2N fMqUY8ascvc6ttLAB4KEl/YtUDwScs4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=M942xidG; spf=pass (imf19.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.160.48 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676792013; a=rsa-sha256; cv=none; b=4bFKbnNMvnj9sI8GIsTEvSR5RL0QVk3smJqPw3PhTPIhjavlkp3lKS7TWpsz3QrWzdldD0 aMOMosYnZKclJDNul0NQfWgUvPRl0DJwiiSuTzRGNkE7AP+2zGtX7DfO1lJdXX2wT1r+ov A1Ipac/z65pWNVdtPfoJRn6U2iBqqiA= Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-171d9ff4af8so130555fac.2 for ; Sat, 18 Feb 2023 23:33:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=foTbT4QwdeCG8p/+pG7+6hYxA8iHmzjKmefWFVB0gjc=; b=M942xidGgMKr/OauZOPmRqRj/n2NxWP9yZV+FKV0A0eh+VjgQBEhFdESqBfA0GlCuc lRqkkkX8S8t6DWDnX/Bt5qBtZglDeLAwAiO5VrB9tzIsYBi7MmITfMxahbeV9o20Ui// 21WZtokjKghT2CQKgpfQ8cxaj9PRTsbnj6o0m7YaxXlDmx3DDOtfJEdaDfKtgxy5r1uO SrS3Rofj0+iN93Q9Jwts+h98i5F0HPQVpOnn2pYXG5oiuIHypLxc1wGgvbot9Hnmuoid /YX+UV0H+dP4Mj+E9GHdI8k8hEQmmZglej7TjZHdQ1FrDRPjl6eHgcKt2Ad0i0D7pU7l rX1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=foTbT4QwdeCG8p/+pG7+6hYxA8iHmzjKmefWFVB0gjc=; b=dyZfVvBkiG7ccb9pE94PiuPgrxwej9qP5Aq1brfnodfLEgIOq3b/35mcqiObFdgHMn FLJGwXcEgDhdDTOXmyitzNktRrhLXjPoFqj7GbPz15GRswVXFP7k3APgAnSXtIK6nEbI QV4tLTveUI5g7ZVqn+bRs9oi1dKRckQdqzBVTe0jVm5JQKL2J/LRtypS8ctg0HY3dwiY s5K5aQ6OqddJxeAKDs/eNUQOJJH2E4h6CIMoe6jjDKENFQOosjOq/ZscOpvMS2zMEoQV u5CuxDYQB0D9xWFqkWzl2QMrOXqbphYOT2K6l8fowC0NDIvN/IQbe2TkfhsloO8X6zvN PiCw== X-Gm-Message-State: AO0yUKXd2DMHH3TjXLqC/UrnqHIe3p3N8yhWg0V8vQoTddvBBWeLWtkF UddPbA6Aw9a1g25sH8A3f/E= X-Google-Smtp-Source: AK7set+guib6pOxFgMZlsXMOeXidKZHwLMDNZTHAHLW2ZwyXzCVFLoEpTY2q9QtWgwU+f66U8u7PPQ== X-Received: by 2002:a05:6870:c111:b0:16a:7b27:71d3 with SMTP id f17-20020a056870c11100b0016a7b2771d3mr6234295oad.7.1676792012296; Sat, 18 Feb 2023 23:33:32 -0800 (PST) Received: from localhost (227.sub-174-197-65.myvzw.com. [174.197.65.227]) by smtp.gmail.com with ESMTPSA id u6-20020a056870440600b0016e49af5815sm3425295oah.51.2023.02.18.23.33.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 18 Feb 2023 23:33:32 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, bfoster@redhat.com, willy@infradead.org, arnd@arndb.de, linux-api@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v10 1/3] workingset: refactor LRU refault to expose refault recency check Date: Sat, 18 Feb 2023 23:33:16 -0800 Message-Id: <20230219073318.366189-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230219073318.366189-1-nphamcs@gmail.com> References: <20230219073318.366189-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 327C41A000B X-Stat-Signature: 4by3atwpr6ez5sb79e38bpe5rrcskgxu X-Rspam-User: X-HE-Tag: 1676792012-780328 X-HE-Meta: U2FsdGVkX19Hj8vd3QYSIw+9f/QaZdMeF+fAVZD82qGJ9fsSImlq8TOn+Vt8XvVQMI/tHPWKXc1z2Bidp6Htcsv6XY5j1Sc4UEuR9TNbt08Vc84mRFmb4y7l3N5T2ZlVBof3WzpKIbGEe//oHmIgGZa0df2zsbREH3DzwbzHSdZ5LHDT1h/QFat/JIsW/SIryczt4VIrByGK1+5PofLwWyJluftyy8pchimlB+RpmH4E5pneSlJAvHCYa2OpwOZPjCGaKBp2sASHumOrEZc9GEUKXu5dFBeXauKkbhkrrMJcRUUjsBrycEehTSzucpLEiyn+n2Iv9iKPKsrVKYa6XxXGttoqUdoSprVM1MO+FaW8yQwpSOwJHBMbC+U4cjqsbymXc2UmvWJmoe8zQ4auD8aAqL89WOh0wllFxU8/8C4wVJ+8QurxU3IUqPrwv2s4TbBvyjW7aaj7YZ97XXQsMm8Llelc0ZsoLsVyvigfAlD66kRKL66dXrVrI1O/Zlbj3DHUSwwt4hONcXTUbfRhx92W+hA7yfiYyg8HpZjbl7Hpl3PVIKzBdy4w9Rv5eDJPdHIa1yDuz/XnyTfCdCH2u1ufZc6QEux9ERN5CGsPC0PUMa8wfnfl21dNg/Dh/Qk7OZ+//AXTyWxveFrCyaQ1RkKdlP4xa8DSyNopM4zIMFY/9DrZwkHAdS/JbCE0/Q0s3826KREh3DATkgigYCOA8A2xnMN2h67G8OLsdY6KKDo4WrGmjJYmeCeyn35ujoHvxNIKLj0cXk9arlmYvLr5ZibsjVFik32LfnwLC64LOkGNNCn2l9XmDTdWxpjmLOO69GvkybyJxGa/BUXgceyG0hJdEx5zUdiJDi4Sm87gJaKt9gOQXtriqsqBjiftKfprjXAaXyaaZg2rejxBXpq30hi4ZwHTrZHFC8RmZbP+miol5B0SEawKHwesb1v2rLLoJ/pTwyFFmHYnlXu96fN xyJoR5Bi PBItVU6JuOSzw8fk6VMY64Jfc72sdCOENSk0BrtZMkJZpU/koyDJrrYA+65X4mbSpfN5r+HZvtQ1AzEgn57sdDw8+OPy6sxhaSUJbb4XIDS2cwAJaRPp3npQWK8NP3EKHJmFyW5gMGkksLOKs/plIdN9x6sEg/F/NT7FO9QRPDisU+HV2WmbbEpbiDYPEU/WwodJTvh/K9VT+qb7ut8P5zY31P9C6kWIwRxDKUC32SBswk5gSFf4EUVdM21jnnNRSJA7ixYnxeL9ge9YGJHNxk+m6FOhqJS/AwDAb/HuCHudj5FFOhqJTiN+RsmkzAfwISzJwMbaczDjGZWUO9Q2AxzJvezAY5SowUQPvQcp1/gaVIC5nBBArfkcDPG/4CKz0UPOPEKOKbjn8VJYjSsGEJPkVDgDyCR9HUvXLCkFJNo2ltNpy567Ci1kHGAMh8d0wWKNspi0PzZ2pzYg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000825, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for computing recently evicted pages in cachestat, refactor workingset_refault and lru_gen_refault to expose a helper function that would test if an evicted page is recently evicted. Signed-off-by: Nhat Pham --- include/linux/swap.h | 1 + mm/workingset.c | 142 +++++++++++++++++++++++++++++-------------- 2 files changed, 97 insertions(+), 46 deletions(-) -- 2.39.1 diff --git a/include/linux/swap.h b/include/linux/swap.h index a18cf4b7c724..dae6f6f955eb 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -361,6 +361,7 @@ static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) } /* linux/mm/workingset.c */ +bool workingset_test_recent(void *shadow, bool file, bool *workingset); void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); void workingset_refault(struct folio *folio, void *shadow); diff --git a/mm/workingset.c b/mm/workingset.c index 79585d55c45d..5f1d20a0a2fd 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -244,6 +244,30 @@ static void *lru_gen_eviction(struct folio *folio) return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); } +/* + * Test if the folio is recently evicted. + * + * As a side effect, also populates the references with + * values unpacked from the shadow of the evicted folio. + */ +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + struct mem_cgroup *eviction_memcg; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + unsigned long min_seq; + + unpack_shadow(shadow, memcgid, pgdat, token, workingset); + eviction_memcg = mem_cgroup_from_id(*memcgid); + + lruvec = mem_cgroup_lruvec(eviction_memcg, *pgdat); + lrugen = &lruvec->lrugen; + + min_seq = READ_ONCE(lrugen->min_seq[file]); + return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); +} + static void lru_gen_refault(struct folio *folio, void *shadow) { int hist, tier, refs; @@ -258,23 +282,22 @@ static void lru_gen_refault(struct folio *folio, void *shadow) int type = folio_is_file_lru(folio); int delta = folio_nr_pages(folio); - unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset); - - if (pgdat != folio_pgdat(folio)) - return; - rcu_read_lock(); + if (!lru_gen_test_recent(shadow, type, &memcg_id, &pgdat, &token, + &workingset)) + goto unlock; + memcg = folio_memcg_rcu(folio); if (memcg_id != mem_cgroup_id(memcg)) goto unlock; + if (pgdat != folio_pgdat(folio)) + return; + lruvec = mem_cgroup_lruvec(memcg, pgdat); lrugen = &lruvec->lrugen; - min_seq = READ_ONCE(lrugen->min_seq[type]); - if ((token >> LRU_REFS_WIDTH) != (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH))) - goto unlock; hist = lru_hist_from_seq(min_seq); /* see the comment in folio_lru_refs() */ @@ -306,6 +329,12 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + return false; +} + static void lru_gen_refault(struct folio *folio, void *shadow) { } @@ -373,40 +402,31 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) folio_test_workingset(folio)); } -/** - * workingset_refault - Evaluate the refault of a previously evicted folio. - * @folio: The freshly allocated replacement folio. - * @shadow: Shadow entry of the evicted folio. +/* + * Test if the folio is recently evicted by checking if + * refault distance of shadow exceeds workingset size. * - * Calculates and evaluates the refault distance of the previously - * evicted folio in the context of the node and the memcg whose memory - * pressure caused the eviction. + * As a side effect, populate workingset with the value + * unpacked from shadow. */ -void workingset_refault(struct folio *folio, void *shadow) +bool workingset_test_recent(void *shadow, bool file, bool *workingset) { - bool file = folio_is_file_lru(folio); struct mem_cgroup *eviction_memcg; struct lruvec *eviction_lruvec; unsigned long refault_distance; unsigned long workingset_size; - struct pglist_data *pgdat; - struct mem_cgroup *memcg; - unsigned long eviction; - struct lruvec *lruvec; unsigned long refault; - bool workingset; int memcgid; - long nr; + struct pglist_data *pgdat; + unsigned long eviction; - if (lru_gen_enabled()) { - lru_gen_refault(folio, shadow); - return; - } + if (lru_gen_enabled()) + return lru_gen_test_recent(shadow, file, &memcgid, &pgdat, &eviction, + workingset); - unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; - rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might * have been deleted since the folio's eviction. @@ -425,7 +445,8 @@ void workingset_refault(struct folio *folio, void *shadow) */ eviction_memcg = mem_cgroup_from_id(memcgid); if (!mem_cgroup_disabled() && !eviction_memcg) - goto out; + return false; + eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -447,21 +468,6 @@ void workingset_refault(struct folio *folio, void *shadow) */ refault_distance = (refault - eviction) & EVICTION_MASK; - /* - * The activation decision for this folio is made at the level - * where the eviction occurred, as that is where the LRU order - * during folio reclaim is being determined. - * - * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. - */ - nr = folio_nr_pages(folio); - memcg = folio_memcg(folio); - pgdat = folio_pgdat(folio); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - - mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); /* * Compare the distance to the existing workingset size. We @@ -483,7 +489,51 @@ void workingset_refault(struct folio *folio, void *shadow) NR_INACTIVE_ANON); } } - if (refault_distance > workingset_size) + + return refault_distance <= workingset_size; +} + +/** + * workingset_refault - Evaluate the refault of a previously evicted folio. + * @folio: The freshly allocated replacement folio. + * @shadow: Shadow entry of the evicted folio. + * + * Calculates and evaluates the refault distance of the previously + * evicted folio in the context of the node and the memcg whose memory + * pressure caused the eviction. + */ +void workingset_refault(struct folio *folio, void *shadow) +{ + bool file = folio_is_file_lru(folio); + struct pglist_data *pgdat; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + bool workingset; + long nr; + + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + + rcu_read_lock(); + + /* + * The activation decision for this folio is made at the level + * where the eviction occurred, as that is where the LRU order + * during folio reclaim is being determined. + * + * However, the cgroup that will own the folio is the one that + * is actually experiencing the refault event. + */ + nr = folio_nr_pages(folio); + memcg = folio_memcg(folio); + pgdat = folio_pgdat(folio); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); + + if (!workingset_test_recent(shadow, file, &workingset)) goto out; folio_set_active(folio);