From patchwork Mon Oct 14 22:12:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Xu X-Patchwork-Id: 13835529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAF54D18156 for ; Mon, 14 Oct 2024 22:12:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7809C6B008C; Mon, 14 Oct 2024 18:12:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7309E6B0092; Mon, 14 Oct 2024 18:12:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61F106B0093; Mon, 14 Oct 2024 18:12:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4235F6B008C for ; Mon, 14 Oct 2024 18:12:40 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7979712132D for ; Mon, 14 Oct 2024 22:12:32 +0000 (UTC) X-FDA: 82673607828.06.F1B765C Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf06.hostedemail.com (Postfix) with ESMTP id 421F8180011 for ; Mon, 14 Oct 2024 22:12:33 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gUGRYTF0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3VJcNZwcKCPcvdhwtfbfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--weixugc.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3VJcNZwcKCPcvdhwtfbfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--weixugc.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728943853; a=rsa-sha256; cv=none; b=BYamrJkKZWhSQOox8NZ2vqsG0FQflpuUwREaAlT0cx4DaIWtv/nSbikguDNKWmGy6pvSQh lCMOJBMl4YFMCFFZ/3x29Vu9VYR59cMvXVA8vGupdzVGGkbbdZVlZexQTrvpJKmBgiOAJ6 BSYVpi9uCIbJq70NwTAL9fz1D+jj2Bk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gUGRYTF0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3VJcNZwcKCPcvdhwtfbfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--weixugc.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3VJcNZwcKCPcvdhwtfbfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--weixugc.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728943853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=VLFlB8ArhD8sBmaD8/9E7s8jR7UTSwheYDE3N7Ewx8Q=; b=EGqNGpSTAPSCuPyZd0OhSSgk1LpBB+RqUA4ELU5N0D+/sEi0RnCSm/CXZ61RyLpmKu8bX8 PteoEA/3+RuVGZ6HI+7mVDaC4btW/Ol7K8XWqZymYi8bB1S5iwSn2cbMeWwaUHJnVgd+5J 005CajVGwKAqnyXZ5MoMOvWZvmR/Djg= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7ea0069a8b0so4136501a12.0 for ; Mon, 14 Oct 2024 15:12:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728943957; x=1729548757; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=VLFlB8ArhD8sBmaD8/9E7s8jR7UTSwheYDE3N7Ewx8Q=; b=gUGRYTF0lY8LGEGnaZBEEO83530Pyxsr6f9QTK7OyosE/g/SD7aegbLF05k46SWLF8 tsPy92XsrqEclcVl6crsXowOv2aoV7CypahfYE/NVvh9NGlN/AI9XPIgpA5bu09ih5cR XKhhyhk+nXgLO38I2adn3LPhgFcL9ePjjWhA6Zb10OsMk1WIBMvffeL5/ZMv19xwxVfr 4WI2XNa7fgxT3HFLqZcMov3Tu4Ti+CNp8KLOGQqoBrldikxwEkcAYkF7ONXjuS7vRXIA 1VDkq+updnvdCJCWtp2/Zslaz7L7bET60DAsRVnYkm6FZTSwv1Yr4gXAnf9/ia7uAIeP 8ZUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728943957; x=1729548757; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=VLFlB8ArhD8sBmaD8/9E7s8jR7UTSwheYDE3N7Ewx8Q=; b=W44LISjwrIoRlLGWlZkQLTj74OuETFYleeu+qlQXNUKwbSJAwTOuPzCLX3Q9wcQ1Lg +61OYmmf7uQJWO3tQWdNWLhXtiQUcBmlPd4SfyIsl/gkZ+ddcJDfA8JzQOuahxKUbYaA tvRf1yFVMir2CdCwETTVYUlqI500X0ZPbbWqTSfdXi0wkNSHOvOhd5L9x5P32ZJK8YTG R+G4tafuMlogbbV8N2mut0C++bO9HzVnOAplgzATxRYC3u6IDRvi4ib4v1aIqk8hzvpw 9oB1xQwMv93ZTZz14q/8/+x0xoMKYcTo95AtkvY0blA8iFIe6jo+ir9MW8Xq9MFr+dvb XxuA== X-Forwarded-Encrypted: i=1; AJvYcCXP3HVKi3xDj65eh0mE68If7JAxnnD3hf9ESZc/l3JMYYoggFazaZTOsAh8ZqWSD1wI8fq4cAs8DA==@kvack.org X-Gm-Message-State: AOJu0YzF8GUR4LLlEEkyEX9R6JNnZZj9U2Rhwe9Etmw4c4SjCznesO2h lRTT6xAJ7P8EyGjUL3WYxA1T/RjSvASUiJzz1qYtFMGe1ChUbHSB5UZhnv8XLaXrktMscDjdhND c0O85tQ== X-Google-Smtp-Source: AGHT+IEZbp7h/ndmghwF1NV+bCjPhzb/M48R/4ukJlrDWG9j0yNm9iKhDQ51QtS4buCnNidxEv+y6NGvapg5 X-Received: from monterey.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:641d]) (user=weixugc job=sendgmr) by 2002:a17:903:181:b0:20c:d303:cae6 with SMTP id d9443c01a7336-20cd303cdffmr228005ad.5.1728943956134; Mon, 14 Oct 2024 15:12:36 -0700 (PDT) Date: Mon, 14 Oct 2024 22:12:31 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014221231.832959-1-weixugc@google.com> Subject: [PATCH 2/2] mm/mglru: reset page lru tier bits when activating From: Wei Xu To: Yu Zhao Cc: Brian Geffon , Jan Alexander Steffens , Andrew Morton , Suleiman Souhlal , Axel Rasmussen , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Wei Xu X-Rspamd-Queue-Id: 421F8180011 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: pam43o8yj4a5oo4h81m4wjsgz4qewqwp X-HE-Tag: 1728943953-882911 X-HE-Meta: U2FsdGVkX19BJ0jSnK/c2/eQq0E4B4SBMWFpbR5OgDPpvuQUzXTuESEJI565PGOEYbO90wPZwoW1pcZpsBqVj9GH6cF3/xOVDjwur6hn7wFXW/25uTdG+C8LFX7FfpMiGhlmGaGqrM550Vtcw87IxTe2a0rV0X8tqiXWnKW4rH1SPBZZuo5nHHIdi6SwC/Zc+F7gydIt4716TGU16/6PK0JDxOReBXQswuA4WwyBfRavJnNP91Tb1VIMyY4jd9NWMDb6+bfgTsZDmvlp+w577Xs9O/eGUoibC1ewHadZ5IlsEwbLkYelpfm7/j9+J0vqqVAEfZ04hsktHe60YpiHKSplQ1ML1H0MfB+oO75s/UFZJsPRubyzLZ3b5hrctaYqqyQ9Z48tuM2qpj8USZ3Ic6zeRoTcyP3waOLymJuBFt4YdRUhmnw30SfcLVQTwoMZcySH9d6JPnOIQ4EyDvJa5l+FpHMOMra20r6EvveTloAfzDVsD3QdM2SyTkSWLdnNU2NUHTGIdet/jT9tN0FAE+c51/sHNYvzLvEFVVlTa8B2xlqdj6jTDwKPSQzYt72TI8WEUz0AnC0zgm+FGY2vy4C9KDe8mVMjwcYFu8tTpekVbGK+8M2eJuYfrIhoCah7/IiEkVxVaXTyuMwi61Rde10o7dMtVIFD/m57jsq72y6/5Vr7i/iyoAUlql9bUnUzc6fYkjR7iADxfL37DrPxv2QFgZT9R+A8rfEfGCFF+irQC1igkrmw+tKnwSzMozTDUhj4Kr0GgCf+tOellmbHT/C4t8FX8VDbjkR/Ms907YSB9Oj2btSs0+nq7V0GoWSiiCqjiW3zF4VdO1uinV+QSCqiKDg95sBU+eMCTZvcOECWAh5Rd0eWR+wUD9SrMheDDvkApgXb/jq3bDfEDL8U/ACXQjMhNTd+JgpjD+0OA5wR+qL2sipXeG+N21DFFC2YP6sGFxAkBG/nvxjUEQm 2cuZ0+c9 M5URs3d+1WtrEcBlS+C49OHxTTE+HOsw7FtgLLOXq7ox+qoc38guLONNaf11t8+WbWWIR3JocMDP3hV4DAokAZd7N3BIhGRf/PR5BqhRYLjJiyT0WVd5/iZEy+21UgCwKL26YHqvKY5CVQia6DrZFMFnYh93ViWHbpo9/R3A18vnpLdvVlIqmmctAyOTvoQO4bn30m8QGkOpMB0jJwGWplBe15Kxwt2qvCbhYwHpkgUR9mgPT4ApWP7co0m4/mJP0WN3M2zaTN5W68H1/RJA6JHm/GvWnRrwrTpMklogmlEaNz6UeolcBk5kEfZnoUtaDt4a60i9licXFvbBruD58wQShfdidi6AXOBvY4yJOcSQbqGyNsX0+T0/fUyxPYNeJAVvkue/3hNsGbEdEH/1ROhnCGrBZZyvhAKjxgvhC0Zkkj7ojAnigY9E8KxYL6D9pu1JkM2UmVMk54uqMkfRmbsbGuEDWoZFv84ujnCWHTn9NNC6pDkMBAymLmRCTtN1xX7Tl7Gs3HjXz+xL1YMrhbp9Kq+p2W2wWOkgcTDeFTa+6OeE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000112, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_activate() calls lru_gen_add_folio() to move the folio to the youngest generation. But unlike folio_update_gen()/folio_inc_gen(), lru_gen_add_folio() doesn't reset the folio lru tier bits (LRU_REFS_MASK | LRU_REFS_FLAGS). Fix this inconsistency in lru_gen_add_folio() when activating a folio. Fixes: 018ee47f1489 ("mm: multi-gen LRU: exploit locality in rmap") Signed-off-by: Wei Xu --- include/linux/mm_inline.h | 5 ++++- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 2 -- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 6f801c7b36e2..87580e8363ef 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -222,6 +222,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, { unsigned long seq; unsigned long flags; + unsigned long mask; int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); @@ -257,7 +258,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, gen = lru_gen_from_seq(seq); flags = (gen + 1UL) << LRU_GEN_PGOFF; /* see the comment on MIN_NR_GENS about PG_active */ - set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags); + mask = LRU_GEN_MASK | BIT(PG_active); + mask |= folio_test_active(folio) ? (LRU_REFS_MASK | LRU_REFS_FLAGS) : 0; + set_mask_bits(&folio->flags, mask, flags); lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 17506e4a2835..96dea31fb211 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -403,6 +403,8 @@ enum { NR_LRU_GEN_CAPS }; +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) + #define MIN_LRU_BATCH BITS_PER_LONG #define MAX_LRU_BATCH (MIN_LRU_BATCH * 64) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9d1e1c4e383d..907262ebaef8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2601,8 +2601,6 @@ static bool should_clear_pmd_young(void) * shorthand helpers ******************************************************************************/ -#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) - #define DEFINE_MAX_SEQ(lruvec) \ unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq)