From patchwork Fri Apr 21 23:14:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13220791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2062BC77B78 for ; Fri, 21 Apr 2023 23:14:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 914AE6B0072; Fri, 21 Apr 2023 19:14:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 89D546B0074; Fri, 21 Apr 2023 19:14:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A1206B0075; Fri, 21 Apr 2023 19:14:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5C1EC6B0072 for ; Fri, 21 Apr 2023 19:14:27 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 175C8140980 for ; Fri, 21 Apr 2023 23:14:27 +0000 (UTC) X-FDA: 80706954174.10.E371FB8 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf06.hostedemail.com (Postfix) with ESMTP id 44C9F180007 for ; Fri, 21 Apr 2023 23:14:25 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=qpzapAUX; spf=pass (imf06.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682118865; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Leee/43AWJ8Q1BedRn9GrIwj8H1dQXKt4vvVqdpQ7zA=; b=HRoi3Ji5yKDvOGdVWu5oYx4PaoK7rA6fknOeygyp/WAVETaiQnk4bCikoi2LwDI2w2zIiV xYust69gDBQMiO30EB5xmKt4od0mGFI7jy+mvqz2GsOlazOtHWd25J2Mr95ekfke2glaR9 2JcMjzvK6tOPRHhvfca0a4VfQ3mEuBI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=qpzapAUX; spf=pass (imf06.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682118865; a=rsa-sha256; cv=none; b=rN+g+XKPpqzYNNBtUOiHT6ooMRCFdffTankICWBSef0hl/6ADwhT3Rg0qIerkzUoit1T0h KDsdIZ3kTG9JIBp5QLQYrzkVv/+ZAoxm3sH6dPqTKkq1NkLpH5HLLLB8uDZ1MPCh8tC00p B8SMJpQAPsjsvr+Sfa71e7ii585M8EY= Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-24756a12ba0so1870731a91.1 for ; Fri, 21 Apr 2023 16:14:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682118864; x=1684710864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Leee/43AWJ8Q1BedRn9GrIwj8H1dQXKt4vvVqdpQ7zA=; b=qpzapAUXJSG/VSkZAcIof66MBA6g3b5FkbFLwfeFp6lMkj0KeYA1L2DPWCUCaX55b5 58lTt9YmvZsiyymOOCAk1pfZ/4vUUtyyDnCugENVZXo4Hts19czpCzfJJhQdTQVATqpc F3h4ciihVzkjEXCDGwMyDIHET8jZyOYQ2605i0A57uWLfHS3FTFz3IIfVn5SxGUhpz9F 8HkwegZeAySWkFzBKLskrElCF4Ea09TTATHVpywkd2zF1/+QTMgxTwbsFAuHj2m29EHR KhLKgIH/IRj6bN7bwRXiWqDcfq4/mfKnK7XnACVL4VYU706AaMLstyWVcDIw9A53QPrk +27Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682118864; x=1684710864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Leee/43AWJ8Q1BedRn9GrIwj8H1dQXKt4vvVqdpQ7zA=; b=g7eAYz+o9iejEXHi19lxaj/sKpfcWjf4mNnSnrfu2aAf0AePPvUSpZ1R0H8z/eF0Pe kW+HJfIdN3sDGY01cMKO31cQ9VvZOu0Dg/3RCsMTmAkEKpjOlDT+BJFF/1IJPAyjyHYW N7YnIGEY9gjbXdzNrOd7G5y0Q9gzM54OuqmhZcvnfZ4owYKjewRObZ/UU7qPpTRmF+D8 H42zntDxBfQb5oIYaMF5oA3q5uUT6j+fUAFmJs+Y1CBSbTNM6dgZu26j8P2YHTI75FF8 NXNPM2dqLXPHigbamwDUBb9Ec6wamEUIcxAMLVMh5cuPzVuIh5M0Rk8hLhDWtIwEbDLg fbaw== X-Gm-Message-State: AAQBX9cTD/+iXfWPSaTMMddnHbptP4E/80jy/4Z/H9vkzEYosBhDPwjU eLyvaMJE8QQhFyp+9dQyExE= X-Google-Smtp-Source: AKy350bcFaY1hRgAoGPvRk70m6ChPhjgG1GESyjlVu0QqKyDnYX0AUVVRTS5wsIqL6PrFbsOH7i/4A== X-Received: by 2002:a17:90a:fb95:b0:248:839e:551 with SMTP id cp21-20020a17090afb9500b00248839e0551mr6469738pjb.35.1682118864039; Fri, 21 Apr 2023 16:14:24 -0700 (PDT) Received: from localhost (fwdproxy-prn-004.fbsv.net. [2a03:2880:ff:4::face:b00c]) by smtp.gmail.com with ESMTPSA id y8-20020a17090a784800b002476ee46dbfsm4808420pjl.57.2023.04.21.16.14.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Apr 2023 16:14:23 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, bfoster@redhat.com, willy@infradead.org, linux-api@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v12 1/3] workingset: refactor LRU refault to expose refault recency check Date: Fri, 21 Apr 2023 16:14:19 -0700 Message-Id: <20230421231421.2401346-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230421231421.2401346-1-nphamcs@gmail.com> References: <20230421231421.2401346-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 44C9F180007 X-Stat-Signature: 4j7o3hcr7c4kskyo3qb9n4w4hsg59jzn X-Rspam-User: X-HE-Tag: 1682118865-949174 X-HE-Meta: U2FsdGVkX1+/AOIVih4j9J6PQFIS28BoTjHuNwLrRG2QPuIPDwM5jAbL145Y7z8cPRejTVb049VZWfhadNmZs0q7c5KoUY14ZOVQI383XedcO0BcH4wBD9ipJJrN7GmKpadsNeyxHuwSTi7BypKbgvUpDyJnL939gyNohlayom3JcLak8asabEIqgnFH9P4WyicdA6qDzHFuPUInSwVleVs0rEkxomudCJmDqB4EV4B4QKpk+HfocTyo/kt4qpQd7ltB6PuIQMKouBrovdXrqZdsxRUjJUp3X5EsG5AgMrq/P9HP5lJi8E6ooj0izHuyyOmCHequrailpqsiLFfHmyLYKlJQb74W5jDBfGdrOb3CTG9tg6MVxa/v4qc3SjFjpS2lf11FATE13dSyedZxSE5oy9GJjDv0cv2YtEeIBKXKUVQPCeIgfk1opZtWN3fnDuLQFMvjm/e67ApkZPESts554KM7s+TO8j8r3PxhTzPw0DJzhbtTBQVwUtjex5C2O/q58lv3brlkzNaoeFIaie088kzxoZ0DyZIuHxX0brsDjdWI7FK8zLfL+4BSmj8Ge75SN1IKwxpQmi8zGjIS25dk53lDLzGovHk+ghbSfYH1mmOQUCaiLgWSHuqSC8GOPO0yg5wy1bmOrBTc59Wh5VFPrUZ3poBn1euh4qs5pWfObcCQIcYTBHJH8syeWihEnFWQtbI6YukkdqbLNc+qQlQcWT/X5755ITBjWTdWO5ctAJiisrA2syvXxQeBInAcaKIOuiq9V7ZFYLRMG25q+K1kw5L3Y1S4eZLVD5mogg3BATsoK8Omd/gKkv97LHyEjVAJ1dkJCgK2LRiHZlbzDtOyp3lna7o07inVPFX0/HfvlWNb6OeMA8CkKKAwM8pIjf2IC3gP3xPum1ogdkzJPBdM++KUGnLZqsUJIxQWovhjDohtNkIMvVYmlBY8HiSOg7fTQIxRSNuJoTUHowP m9gId5gy BBGBmxuA9lbQSy8fopy6WaBSRX2G25QhE2v/QDnFGEUjWNl+5xGUL48ju3ufcyuyO06EKJUGxkCvnbxd92aVLzu4bGnAHJLQ6Ff25VMaqxV4SIwXpmdcUnyJlD56voRVGuJWF3Z13sG07i+n1iEAbLYTNqCqB2LmlG1yqjspNA4HQYCYkjVbzgqfF+yUbjPFag0GPo3MbpJ7abIGJBteR06Qy0+AmR8XASWFsyReELC87bzIOhL1DeeDeH1bTbgfAgY/xIu7A62wxIfztc1C92UqCQWWj+ogc+enY6aQkmdrbMumfy5iGi9ISiCIf66PS315l2wlscNIeDvPnxGkJsa7b91BnjXFXRWiNQ438ASwaZ7DlFfFk4mxFFgUca3gcTB/Alg1g3wqaQlnAnJQkHYqC+Cy9WHhuRJm1jycXeoZXVJA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for computing recently evicted pages in cachestat, refactor workingset_refault and lru_gen_refault to expose a helper function that would test if an evicted page is recently evicted. Signed-off-by: Nhat Pham --- include/linux/swap.h | 1 + mm/workingset.c | 150 +++++++++++++++++++++++++++++-------------- 2 files changed, 103 insertions(+), 48 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 7f7d5b9ddf7e..417d965d46d0 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -368,6 +368,7 @@ static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) } /* linux/mm/workingset.c */ +bool workingset_test_recent(void *shadow, bool file, bool *workingset); void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); void workingset_refault(struct folio *folio, void *shadow); diff --git a/mm/workingset.c b/mm/workingset.c index 817758951886..d81f9dafc9f1 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -255,6 +255,29 @@ static void *lru_gen_eviction(struct folio *folio) return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); } +/* + * Tests if the shadow entry is for a folio that was recently evicted. + * Fills in @memcgid, @pglist_data, @token, @workingset with the values + * unpacked from shadow. + */ +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + struct mem_cgroup *eviction_memcg; + struct lruvec *lruvec; + struct lru_gen_folio *lrugen; + unsigned long min_seq; + + unpack_shadow(shadow, memcgid, pgdat, token, workingset); + eviction_memcg = mem_cgroup_from_id(*memcgid); + + lruvec = mem_cgroup_lruvec(eviction_memcg, *pgdat); + lrugen = &lruvec->lrugen; + + min_seq = READ_ONCE(lrugen->min_seq[file]); + return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); +} + static void lru_gen_refault(struct folio *folio, void *shadow) { int hist, tier, refs; @@ -269,23 +292,22 @@ static void lru_gen_refault(struct folio *folio, void *shadow) int type = folio_is_file_lru(folio); int delta = folio_nr_pages(folio); - unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset); - - if (pgdat != folio_pgdat(folio)) - return; - rcu_read_lock(); + if (!lru_gen_test_recent(shadow, type, &memcg_id, &pgdat, &token, + &workingset)) + goto unlock; + memcg = folio_memcg_rcu(folio); if (memcg_id != mem_cgroup_id(memcg)) goto unlock; + if (pgdat != folio_pgdat(folio)) + return; + lruvec = mem_cgroup_lruvec(memcg, pgdat); lrugen = &lruvec->lrugen; - min_seq = READ_ONCE(lrugen->min_seq[type]); - if ((token >> LRU_REFS_WIDTH) != (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH))) - goto unlock; hist = lru_hist_from_seq(min_seq); /* see the comment in folio_lru_refs() */ @@ -317,6 +339,12 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + return false; +} + static void lru_gen_refault(struct folio *folio, void *shadow) { } @@ -385,42 +413,34 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) } /** - * workingset_refault - Evaluate the refault of a previously evicted folio. - * @folio: The freshly allocated replacement folio. - * @shadow: Shadow entry of the evicted folio. - * - * Calculates and evaluates the refault distance of the previously - * evicted folio in the context of the node and the memcg whose memory - * pressure caused the eviction. + * workingset_test_recent - tests if the shadow entry is for a folio that was + * recently evicted. Also fills in @workingset with the value unpacked from + * shadow. + * @shadow: the shadow entry to be tested. + * @file: whether the corresponding folio is from the file lru. + * @workingset: where the workingset value unpacked from shadow should + * be stored. + * + * Return: true if the shadow is for a recently evicted folio; false otherwise. */ -void workingset_refault(struct folio *folio, void *shadow) +bool workingset_test_recent(void *shadow, bool file, bool *workingset) { - bool file = folio_is_file_lru(folio); struct mem_cgroup *eviction_memcg; struct lruvec *eviction_lruvec; unsigned long refault_distance; unsigned long workingset_size; - struct pglist_data *pgdat; - struct mem_cgroup *memcg; - unsigned long eviction; - struct lruvec *lruvec; unsigned long refault; - bool workingset; int memcgid; - long nr; + struct pglist_data *pgdat; + unsigned long eviction; - if (lru_gen_enabled()) { - lru_gen_refault(folio, shadow); - return; - } + if (lru_gen_enabled()) + return lru_gen_test_recent(shadow, file, &memcgid, &pgdat, &eviction, + workingset); - unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might * have been deleted since the folio's eviction. @@ -439,7 +459,8 @@ void workingset_refault(struct folio *folio, void *shadow) */ eviction_memcg = mem_cgroup_from_id(memcgid); if (!mem_cgroup_disabled() && !eviction_memcg) - goto out; + return false; + eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -461,20 +482,6 @@ void workingset_refault(struct folio *folio, void *shadow) */ refault_distance = (refault - eviction) & EVICTION_MASK; - /* - * The activation decision for this folio is made at the level - * where the eviction occurred, as that is where the LRU order - * during folio reclaim is being determined. - * - * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. - */ - nr = folio_nr_pages(folio); - memcg = folio_memcg(folio); - pgdat = folio_pgdat(folio); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - - mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if @@ -495,7 +502,54 @@ void workingset_refault(struct folio *folio, void *shadow) NR_INACTIVE_ANON); } } - if (refault_distance > workingset_size) + + return refault_distance <= workingset_size; +} + +/** + * workingset_refault - Evaluate the refault of a previously evicted folio. + * @folio: The freshly allocated replacement folio. + * @shadow: Shadow entry of the evicted folio. + * + * Calculates and evaluates the refault distance of the previously + * evicted folio in the context of the node and the memcg whose memory + * pressure caused the eviction. + */ +void workingset_refault(struct folio *folio, void *shadow) +{ + bool file = folio_is_file_lru(folio); + struct pglist_data *pgdat; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + bool workingset; + long nr; + + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + + rcu_read_lock(); + + /* + * The activation decision for this folio is made at the level + * where the eviction occurred, as that is where the LRU order + * during folio reclaim is being determined. + * + * However, the cgroup that will own the folio is the one that + * is actually experiencing the refault event. + */ + nr = folio_nr_pages(folio); + memcg = folio_memcg(folio); + pgdat = folio_pgdat(folio); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); + + if (!workingset_test_recent(shadow, file, &workingset)) goto out; folio_set_active(folio);