From patchwork Mon Dec 2 03:28:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13889939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F414D49798 for ; Mon, 2 Dec 2024 03:28:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4A026B009B; Sun, 1 Dec 2024 22:28:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D12D6B009C; Sun, 1 Dec 2024 22:28:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84A886B009D; Sun, 1 Dec 2024 22:28:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6081D6B009B for ; Sun, 1 Dec 2024 22:28:44 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 27280C1973 for ; Mon, 2 Dec 2024 03:28:44 +0000 (UTC) X-FDA: 82848586464.07.6314D18 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf03.hostedemail.com (Postfix) with ESMTP id 312262000B for ; Mon, 2 Dec 2024 03:28:36 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IBfduiEe; spf=pass (imf03.hostedemail.com: domain of 3aSlNZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3aSlNZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733110111; a=rsa-sha256; cv=none; b=wEuB3hpAKB8UvId0SeBqd/ToQObk6UV/yy1bUIfJ8Do5bcpzI+aKtXEfVe2iZ+4GUHKiu2 yboEs/XzW4HJClKXqSQCJH5W4G0rVPoZ7kGA969etO6+SucmtjW2LwhAAOwWSlul5zW1/6 HtAIzXdjMnZCZHde6AjkyjFaYalOCAo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IBfduiEe; spf=pass (imf03.hostedemail.com: domain of 3aSlNZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3aSlNZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733110111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=HBGOc/3+RwPb3Zm90r5xELRTFseZX0gK7sCFZpDi6J4ucLD4a3Be+0NwzPHokdxX3MXHDG DdG2sguwoqHASZnfh1jtRj6YiGR4KOmqG+q+lRZeMuRHmq3K9gr+wj+Jlp6Q96pm87UU+h KCM1EYuaENJRwUCQ9EjzanmLmBEjwn0= Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-725164ccd4dso3045532b3a.1 for ; Sun, 01 Dec 2024 19:28:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733110121; x=1733714921; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=IBfduiEenVDI36d/CIlCyWFt+KCEPJs1lLOviVV75CgDIgwFtdnFj3NxpJsDfAMAMZ VathYJ6VrpG44lz3JzDZiyVTaAF8BDf50i7MebScF71nS3Smjf1yr6KATRZqSRCGpLYU 2eOPErsp0PVSthGakWfnihRBxxhglaSx3qQqmVBryWW6L0eCztcAGCVdNUAemIxyqLgI qbLYfNw0ez8HNN8yQ1dpJwsGYJEuHy5l+lEbpqEA5LwMvICRdGNi701vSCz2WoMCcHIH FDqg7JUalhWfUyf0rGF9JtWNciw9upYepjeuQuRTzphuTka6N/ZvzcNRW/iOhgRD9dka BamA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733110121; x=1733714921; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=Y39ytAorJxMG4hhiZnJtpP1MINmcNKdsIDMHPm74GCDatNPCN7bevmDw0lbvhQit9c CUaD56LbFblh2ye57F1VKjSLThkRVBSxxVcCswrtaCYoPHEGo+O71Ju1z/DpF32INHui b4FLjfyAVZaeJgRxIqBR5ioB5UHXnm762SZKYgNK+wtPAqZzNvY6cTMuMu6LAUDE6pfv 4oRg/ue0udgUf3f4rJKadBgxYqigmGXL9Ay0cZQFDjXVigMzvNQfV8vypjESc7ti1m2q tvKPc4Pk9y8+krv6N8BBs4Lj6JO+Z94do9W/zFaBzqxQADLpvGt2LCee667og+lMVTH6 c+Ow== X-Gm-Message-State: AOJu0YzMWGL9wilNlozFJ2rueYVP6okWCY/PUeCgDNCygqmGKtwUkva7 QTLffoxvaN79/3z9WA1N9THFPbPIjREyaDnM5pn2gAfDlO3HLhCJzWOhAiCP5O5b9ThPRwWe7/o MQg== X-Google-Smtp-Source: AGHT+IGV/8vBTUsUxmc+joe52tDW4U0gAzmSevdVnbajr1Q1NEr3Inb6b68KRcu10pksEIJ78suAm86dnjs= X-Received: from plgp12.prod.google.com ([2002:a17:902:ebcc:b0:212:3f35:e4bd]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2452:b0:215:609f:3f9e with SMTP id d9443c01a7336-215609f4515mr110182305ad.40.1733110121122; Sun, 01 Dec 2024 19:28:41 -0800 (PST) Date: Sun, 1 Dec 2024 20:28:22 -0700 In-Reply-To: <20241202032823.2741019-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241202032823.2741019-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241202032823.2741019-6-yuzhao@google.com> Subject: [PATCH mm-unstable v1 5/6] mm/mglru: rework refault detection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kairui Song , Kalesh Singh X-Rspamd-Queue-Id: 312262000B X-Stat-Signature: 3xaodedf7par89xatcey8nm9yeuxtp7f X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733110116-523369 X-HE-Meta: U2FsdGVkX1/J28lnKksU8oXNzMKQM9gsJmdhi2ti6aRrFwBwg+bXkzvPH28Kex3ru04lvtbSmEgSxGteTJ6KDz6638/D3CUnsZKYjPiUrtPMGP7wrGi6vLCtNHStdMOXLRa4O/U0mxeXZ9IkQv6faTs281QBGKZD5fMHFhUUYY2DLn8xxe/bcnFiVCCuFHTI0tac4cnWxjxmC0FHz2r/kt20S5zLs/5PFkei+ooekLEJ7JqFPkdOr6vcNhc76Tgz1BUqOWkImfvO/MrEDFNqXbrqT96MGdOktd4d3lDDvOnliVfjPX/JBC8I7styGX06sU8gVsyPeG9PPLV/P9jJYS4E9R0aKanyxQVlLkbYl3GqaQZAtuTi2/Fe2aX7amTkwByW5r3uRfVX+1M4zLKZO4fwE/2VtaeH9exK+SwsQ/uf/FRNpkN2KVBT/LFVELmTMPrXvlfMiUJ0NaLEX7mT8iWJ2U7vBO0sxHX9zadlE6VDdQSuDr23jEa9I9DA/fAh8fBbJFYTCQSmtPX0mR4W1FzzJfVvT7Dzu+Z9yx7LOWaUtFfg3UzVYBWkWPbXVwST4tST6SlR3aq+7m//ceGVcVQvbqV0GE1L4tk2oYQI2vr7auTDRHlV0rE5merpoNFjvy0oyYvt9h0mOW8jv8djJNETsoc4P7UwyJVv/mFA73T+0sKOVLrYBBeQI6/EtfWmxTXdV7nQyWzieczZCtomQfdBWXUfNgBVw3y2Zxovc8qmg+BcQsmZxCmBXUto1bG2A5g6ZpfVVnMCdFAsoOD/my4rtQyI+wdbKpCRGNXXouVtcNXq4NY0WgYlIrcENP286sxutI6IZ7X0avrmp4d/GGfg+bUloo4ZiiWnWnWAnrC7F0mQUFC6naW3DByRUkMmnYMd4QLgynSmH+LPbN+7r/eAczLockL1jLZbmOTFJCOLGrKULav4cnCdZ8Bwwkz/62+9k323+qBjrwWH1eh QiTlgYxM wrYtASypMcrpaqQ2MXsw9Np3aUlbCyDzg+NDI8XKyT80SyXCgOsV4s6Sb5zfJOnlcOgyzEDRV8Mp0/ZuAkExrh9jLkEFLdU2LUhU/yGs1zbgrvyK/SLVebnao7U62mdRCJTSsEtN/qB1NSUtFh3zljGMwbPltg7u5EvgWIA9gyypvV0YQdGJDcwubTn20MmSJa11f4Rx0K30nqm0tr42CaXcnjc2kqGIJ94bSfBh0EQf9CykJ+IQEtItrFLt+8z+72kE0i4OJNWmlJKTpV85EZQA1uF8HJylDKUjEzHdneBeVDHq98AF2LZ4qzCtLzpmQcf4SefkhwZ3TktfidBohXImJAsi+BHs6J8bK+wzCwrrqw8kGeeatQl5FMiOkEE9h+suiFSfmLFWmbWXn65DTeN5X/VWcqRjo9QlkpeUqpzYSG0V3FH1YAY/+DuJlD3XAO0V1A2Bt6TlAv+50jOnUC0FJlbmyg3P3B5SZ7YvE36uxbkioTiV5xQie7hlnUf/OTWWHZOI848He81u5BJU30XyX8QKGvSL/HhM8oqLyAFxAk+W3JRgYemBp5AeOC+0UQI5DPECaFyMniiL7tVBMj0rtR6fLbmVD7XTS09pQSx5CFdM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000778, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework workingset protection as well so that the comparison of refaults between anon and file is always on an equal footing. Specifically, make lru_gen_test_recent() return true for refaults happening within the distance of MAX_NR_GENS. For example, if min_seq of a type is max_seq-MIN_NR_GENS, refaults from min_seq-1, i.e., max_seq-MIN_NR_GENS-1, are also considered recent, since the distance max_seq-(max_seq-MIN_NR_GENS-1), i.e., MIN_NR_GENS+1 is less than MAX_NR_GENS. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: Kairui Song Closes: https://lore.kernel.org/CAOUHufahuWcKf5f1Sg3emnqX+cODuR=2TQo7T4Gr-QYLujn4RA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/workingset.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index ad181d1b8cf1..2c310c29f51e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -260,11 +260,11 @@ static void *lru_gen_eviction(struct folio *folio) * Tests if the shadow entry is for a folio that was recently evicted. * Fills in @lruvec, @token, @workingset with the values unpacked from shadow. */ -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { int memcg_id; - unsigned long min_seq; + unsigned long max_seq; struct mem_cgroup *memcg; struct pglist_data *pgdat; @@ -273,8 +273,10 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, memcg = mem_cgroup_from_id(memcg_id); *lruvec = mem_cgroup_lruvec(memcg, pgdat); - min_seq = READ_ONCE((*lruvec)->lrugen.min_seq[file]); - return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); + max_seq = READ_ONCE((*lruvec)->lrugen.max_seq); + max_seq &= EVICTION_MASK >> LRU_REFS_WIDTH; + + return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS; } static void lru_gen_refault(struct folio *folio, void *shadow) @@ -290,7 +292,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) rcu_read_lock(); - recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset); + recent = lru_gen_test_recent(shadow, &lruvec, &token, &workingset); if (lruvec != folio_lruvec(folio)) goto unlock; @@ -331,7 +333,7 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { return false; @@ -432,8 +434,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool recent; rcu_read_lock(); - recent = lru_gen_test_recent(shadow, file, &eviction_lruvec, - &eviction, workingset); + recent = lru_gen_test_recent(shadow, &eviction_lruvec, &eviction, workingset); rcu_read_unlock(); return recent; }