From patchwork Thu Oct 17 18:15:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Xu X-Patchwork-Id: 13840573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 634B6D3C526 for ; Thu, 17 Oct 2024 18:15:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D89026B0088; Thu, 17 Oct 2024 14:15:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D12886B0089; Thu, 17 Oct 2024 14:15:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8C906B008A; Thu, 17 Oct 2024 14:15:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 968F46B0088 for ; Thu, 17 Oct 2024 14:15:42 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 990E1A08B8 for ; Thu, 17 Oct 2024 18:15:21 +0000 (UTC) X-FDA: 82683896862.13.33E46E5 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 23EC68000A for ; Thu, 17 Oct 2024 18:15:20 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=INMg+kmJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3SlQRZwcKCHcrZdspbXbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--weixugc.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3SlQRZwcKCHcrZdspbXbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--weixugc.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729188892; a=rsa-sha256; cv=none; b=Ltk5ABZFkDd92ByzYAvi/WbbczPCKMrfAqiArHz1zb/tWWxblytYSDlL3Z9zytctD6dAqw 0/s5U7IjOFqQmNPlMlut3lwi3IN/P/1l4cstAFqyrcJdKSyQvdwML/Tn3OhT63Tk3Germd Fw3KESpKTRrZbz5cUUCz9JYuR9+y6KQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=INMg+kmJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3SlQRZwcKCHcrZdspbXbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--weixugc.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3SlQRZwcKCHcrZdspbXbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--weixugc.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729188892; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=HxA9E1IHlt239nhDTqDd4dmPZVRZp9vSfPKedRWwzZg=; b=i/gXi/sZdSCWxMwjAX+g8ZqfeCn8ZTJICvDnCsJbJrY+AYqZFNlr/bTa71x8r02obCp3P+ 8FvABorp0pjZWVbJZvf7PnQvYxcUnWewX4wfGj4OiEAKdFIdJRwxPdQ1m3vrPQBswNEZ6r 2hfhdE1AqmVeb6xXG7mVua6IRZFb9mM= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-20cb3d9f5eeso12057105ad.1 for ; Thu, 17 Oct 2024 11:15:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729188939; x=1729793739; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=HxA9E1IHlt239nhDTqDd4dmPZVRZp9vSfPKedRWwzZg=; b=INMg+kmJ4nL5knz+fBWf9+2pdAFLaPPL4iKrV0qq3yZGcV8jiQKl19/0rPPcBwAuef TBWo5KRYq7NBn9gIOEXvOXvU2okLpbbKWHQRjx+PaS4tMy84T1sDkjJEfzldLON3HpfY 9q0cGL6zeuagR+v0iQS/cdgK40M8sSmpCgZ74r6izFBXstLqGQuhnHNINoU053wuhx3N Q3ofxlVLslKn6HGpdynXTxrN3Wxt5yiDVX6/RIcOD77jTnyDc1l048Tpm+bqva7CISks nENl4NNMmyPmxG41HdpFZDY8cmXH7A3MtFj4/xpMF7uFJe3i94wgklkdlHFKc0anCw0Y hYGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729188939; x=1729793739; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HxA9E1IHlt239nhDTqDd4dmPZVRZp9vSfPKedRWwzZg=; b=OLcM1p3tg8QJSBvtftC7Yd+CQGMB9uSuxXriahRamvHtnbJOsQKuvvM8UNwO4FC8rJ QL+oXwWidEQV7lhzgCQmB6iM+BFRJNFZGr74xgq5VMsVu7/Art9pJyMqzSAbhFBGC3XZ oVI9bdIf7sEoo9dMXtR0keo/1o3EI0Y14WKJqVkHWyJZK5Ga1hYjM7Ea8s/rlJW3s+cF i3rxbxg4NPo66LsYEwHN+4rv0E0bIogguq6+LwhZycUlCbHaufyTHnICK0B1E1tlyOJe U1okxgrxiRn3n89W4K6Knd0VKhLKrCtrNr9JgDlAKz1qvbe/XeJMbq/DFPWVP3BrgRxM X96g== X-Forwarded-Encrypted: i=1; AJvYcCUnXZVUQWYbD5zLvXwn7lvfvrDpGKd+ZGcNxxp95vtvazMvCfNRaJYse0NxUARcU9f4G2dp1K4yxw==@kvack.org X-Gm-Message-State: AOJu0Yw7fQNClkd8FcLvZ31satdYozo8NTRpRzgtG2TkLuGLgGrMRdST 5uYZ952cezy92ExwF18aE/WBemtEzW7BNhTwN2YPRg40vN5UwdWmDd1JwmRGuSAfMfkvtSHcz9R gsLFUjg== X-Google-Smtp-Source: AGHT+IEM5Vb8Wnjqgfj0YMkL/+m9MvF/5frBqIhXLjjHbwiFYCqRQAulc0R8W0vdNIg7J3UV6981NeDrJSjv X-Received: from monterey.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:641d]) (user=weixugc job=sendgmr) by 2002:a17:903:4287:b0:20c:6764:6681 with SMTP id d9443c01a7336-20d4720842bmr44275ad.2.1729188938643; Thu, 17 Oct 2024 11:15:38 -0700 (PDT) Date: Thu, 17 Oct 2024 18:15:28 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241017181528.3358821-1-weixugc@google.com> Subject: [PATCH v2] mm/mglru: reset page lru tier bits when activating From: Wei Xu To: Yu Zhao Cc: Andrew Morton , Axel Rasmussen , Brian Geffon , Jan Alexander Steffens , Suleiman Souhlal , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Wei Xu X-Rspam-User: X-Rspamd-Queue-Id: 23EC68000A X-Rspamd-Server: rspam01 X-Stat-Signature: 3mobjbgkbjcdf5xj651ya5hcnoy8j1hd X-HE-Tag: 1729188920-738617 X-HE-Meta: U2FsdGVkX18KgbnzHhBzOPYYvZKLdcXMG8rkYpq3wUbbUXgQa4+RWT08iRaVgLUnNVCZftb1eZMVvUPGd43vLrPnDTGY/2PpYx9nsyRLlb7v9TR4c0jLFRMJp4XgPgoa1B0AfmsgyQSWbspyey6KTV2T0hCsm8CBOuA147U4W557CNbU43YSl629Oz8H6z0VzpKBVWrtgIa+VHZ6pJhIaegga5HRtnaU+MoNcpFozCR+yB0MQqVpaOa6l8q3aP0A3f34SGyaUHt1NhE3hWT0BIlOV+vJ+pzCE0tasvNXAfVRB/THrdx3fO49rJ+hle4BpTK7wRCUgqZMzUF5Xm7V0dczb+zGyCpmsROBEYf+2W1/cgw4S+wN3MbLU6+294VQwwrkUJ7M5LaVLNosRIMB9s83hlcB4FUxU6/NCOOX65ALJEyuIrXWca146Odfk+fO/MfIeY/2Fvde1N+5B1C4ICuRMX4EzNcHDIE/CviEirnHWJbS/pT/2j6R/P9rBiQlnhpEviwP9vMXuegSP9GElYUb8jK7votnWNJ7lB+5hKbBxhkgZbbx8BSP93Hi5eBhxs1t0TzUADdoCVYRncPEUOF1h/WcyMdOa2CpbIFLYC1/vaVz7jxkSJAax/NLDWZKdYvnbQzno9SuUt6i2lUXspUyvkrqz7qMWaYRNdABspaDWVhhgiYGLiOJZ7ZvX9LRg1XVc+wyvYFiIR7ImQ6iuz38Dug6aL3iZNfVgrI+h/osHuELGaATodX4wnF5Xjnq0sqHYKKVgw/bGqA6jE3z7DYznKICdklnVzH2+GV5qsm1gY8cgak4+KSt8V8lvKzs1YamtVJj69H+w+YkVf0zIIRAHccpPRZIh4+AVi6oGObmSKQ8bKxaZWrlfEKCpfLnWtv7MwvbDbOUzVQkpTKumInOudj/JxfALKZrHPa46faDUu2hE+HZbcUBzasQsnH6JCy3bqcFbusYh1Dm5Ks mAFNcV4T h31yBR0MM13r9HtP485lxDhhCRvAZPUvWboNKprJTwN2VAUHC/8WCVpty7Oiw4tb/zXdLd6iIN3nz2smIB7O6RO8Ikz/LQjj1JBkuWRi3a0fWbDf5O+vU9QAKVTgJ1q2s1L82kk3AOWwRb58XE3qqA6wwwjwaIE3ymlsfOW8s+Z1mcHgSN/AX9CE2LwEWkz/BCPlrgbIKziL/YCSfKiw4Un9odyBbYjqmowi+rC+zJ26jQGr1IDesC8o/LjAgfdYpdHHPTwjlHjLjojHj13QysvbpaRt+RjHfG3bDawoimk4Mu1HXceiyblWvMahpTCU5dmnPfKwSq4ARhScQpzOGnmQYTT6N90dvYMx5S3ylBq5IDCgA9C5de540hceJawJ0t5IHP9C8MLh/a1Ib9/OUG/C0IenNt1vFTeNE4Svk9KO4kwDUQt1pp2f0Vsv9BXno+0FXtSSX08+zGGOCw8HFZfEYHTq/RYd49Ax9AlDsTY/rMP362IA3D3y/HCeJWOUk1xkDjfACztoylMmoDpc1pxyL8u1Ym5f/qjm6Ujnp3jaGxzM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a folio is activated, lru_gen_add_folio() moves the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). This inconsistency can affect how pages are aged via folio_mark_accessed() (e.g. fd accesses), though no user visible impact related to this has been detected yet. Note that lru_gen_add_folio() cannot clear PG_workingset if the activation is due to workingset refault, otherwise PSI accounting will be skipped. So fix lru_gen_add_folio() to clear the lru tier bits other than PG_workingset when activating a folio, and also clear all the lru tier bits when a folio is activated via folio_activate() in lru_gen_look_around(). Fixes: 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap") Signed-off-by: Wei Xu --- include/linux/mm_inline.h | 15 ++++++++++++++- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 8 ++++---- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 6f801c7b36e2..355cf46a01a6 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -155,6 +155,11 @@ static inline int folio_lru_refs(struct folio *folio) return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + workingset; } +static inline void folio_clear_lru_refs(struct folio *folio) +{ + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); +} + static inline int folio_lru_gen(struct folio *folio) { unsigned long flags = READ_ONCE(folio->flags); @@ -222,6 +227,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, { unsigned long seq; unsigned long flags; + unsigned long mask; int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); @@ -257,7 +263,14 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, gen = lru_gen_from_seq(seq); flags = (gen + 1UL) << LRU_GEN_PGOFF; /* see the comment on MIN_NR_GENS about PG_active */ - set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags); + mask = LRU_GEN_MASK; + /* + * Don't clear PG_workingset here because it can affect PSI accounting + * if the activation is due to workingset refault. + */ + if (folio_test_active(folio)) + mask |= LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_active); + set_mask_bits(&folio->flags, mask, flags); lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 17506e4a2835..96dea31fb211 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -403,6 +403,8 @@ enum { NR_LRU_GEN_CAPS }; +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) + #define MIN_LRU_BATCH BITS_PER_LONG #define MAX_LRU_BATCH (MIN_LRU_BATCH * 64) diff --git a/mm/vmscan.c b/mm/vmscan.c index c12f78b042f3..2d0486189804 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2602,8 +2602,6 @@ static bool should_clear_pmd_young(void) * shorthand helpers ******************************************************************************/ -#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) - #define DEFINE_MAX_SEQ(lruvec) \ unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq) @@ -4138,8 +4136,10 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) old_gen = folio_lru_gen(folio); if (old_gen < 0) folio_set_referenced(folio); - else if (old_gen != new_gen) + else if (old_gen != new_gen) { + folio_clear_lru_refs(folio); folio_activate(folio); + } } arch_leave_lazy_mmu_mode(); @@ -4370,7 +4370,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca /* see the comment on MAX_NR_TIERS */ if (!folio_test_referenced(folio)) - set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + folio_clear_lru_refs(folio); /* for shrink_folio_list() */ folio_clear_reclaim(folio);