From patchwork Sat Dec 7 22:15:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13898426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4568E7717F for ; Sat, 7 Dec 2024 22:15:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F43D6B036A; Sat, 7 Dec 2024 17:15:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A4676B036C; Sat, 7 Dec 2024 17:15:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 234646B036D; Sat, 7 Dec 2024 17:15:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F28866B036A for ; Sat, 7 Dec 2024 17:15:42 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 77502802B6 for ; Sat, 7 Dec 2024 22:15:42 +0000 (UTC) X-FDA: 82869570756.29.14B1170 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf22.hostedemail.com (Postfix) with ESMTP id A221FC0010 for ; Sat, 7 Dec 2024 22:15:20 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nOzZE1Oe; spf=pass (imf22.hostedemail.com: domain of 3C8lUZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3C8lUZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733609726; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=mmfeBdA2V5raFNBIyiafdRV7E93kfMU6iN1ntUhCSKPD1A1F3BgXvt7ESh4aAOHd0IELiQ reU1yGBqjapFeJrJMO3SPR77lT5VhmSl1pbHbnG6Hc2mlrDridXnlvq3NNMPFXn1CPZyka X06Nu4rFTZDKMS8TQI3CsBuy6zAml3k= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nOzZE1Oe; spf=pass (imf22.hostedemail.com: domain of 3C8lUZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3C8lUZwYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733609726; a=rsa-sha256; cv=none; b=qjw4UaasPJeyG0qVRFxTD5Gcb3iXEEGRsR8uu8SExNggYdYrSI9xZc9YSP/65lU409ybP4 8eDIQjVBJTtfKEh9fg7UpuBVeEFBX1z/JIx2OuymAiZOnZ2qbKM1JuOAqmPiegogJMvUcT Azmoeg7/jycHVHM4Y0MceSwuIpu912g= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ee6b027a90so5027152a91.1 for ; Sat, 07 Dec 2024 14:15:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733609739; x=1734214539; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=nOzZE1Oe/Us/vhLafHntA9ArD8CDnRotNoYUx2H67jT9Klf3Ef+6WW1jIrzlsA+zNv HeG+bN7LqPyoXcJeuG8enDfZhFQFAGy4uRO1vChcXjBT2e31jE6LNjPD7LDViC7d3XuZ +OkdwyU8Q5ddPIVlnKc3aT7nCtYCaV3/1hLWYwHe9Rj5LINZBCNikzO2NT4TkniNvwJX PzKdZKaPSvlkjJaZqmSEnOMmKxsS7migtVT07nIfOSkar6eFde3C+aRAQR36Fh0JS4QB ku3hJOAVuG6rxo17j7kmJlbkw1DXJIlQ/nddyYEuu5VH/MvTyxlyOLTDcj9G37gVW4Sz +b7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733609739; x=1734214539; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=WbR8IB1l9EUHF/78r43Zm89vrxHcmcfPxx3XLvFSaCMXZ7JC4L0DPM4j86MKA2IX9I ySlBSrXCKQW33eDZgfnntBvvk+SBUH2ilfM59NW3kk4A7afuGqKUarFkJrAaMqRk/v5i gvzfLDpaJ15PkiKx+RPt1N/AYF7rjqrnkIUQ8PERAGoGkCUTJjFU2EVNEEn2Dv3MuAwU rHM02g74iJfswEu8/GFn21V0Kew9iExXzWidXEu4nrt69yNVZAyKGYtk7yvg61EOOkEx WdqF2KpadvvLmvD9g4EXucEe8N+nQBMlqsE81+6gYSIfCTnfy0ekh4GP/yyqSHUbnjxT 51Hg== X-Gm-Message-State: AOJu0Yz1rJyJ0o82JD0IXfO7tlN8NsUinuScukAoMe5wpNWJa2+9lbuh t5Jmx2/gWJkEFZbIfO2z6lZPO28LwRgQAHM9i+QmotvU/fZVE4nO5aVQ+C7ZO/K4VvnfIJGpZVu Ipw== X-Google-Smtp-Source: AGHT+IGLJjHyW+Ykmm2+aL9lWknHp1gQ6xAIWeFiqeIX5Kn/EtyOEYWZv/R88rJ8RijCnornkXxmr9Is4Ew= X-Received: from pjboi8.prod.google.com ([2002:a17:90b:3a08:b0:2ea:4a74:ac2]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2d05:b0:2ee:7c02:8f08 with SMTP id 98e67ed59e1d1-2ef6ab29d43mr12407603a91.37.1733609739414; Sat, 07 Dec 2024 14:15:39 -0800 (PST) Date: Sat, 7 Dec 2024 15:15:21 -0700 In-Reply-To: <20241207221522.2250311-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241207221522.2250311-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241207221522.2250311-6-yuzhao@google.com> Subject: [PATCH mm-unstable v3 5/6] mm/mglru: rework refault detection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kairui Song , Kalesh Singh X-Rspamd-Queue-Id: A221FC0010 X-Rspamd-Server: rspam12 X-Stat-Signature: iydzedepnh74jxmbu6q9g65an97a1hnr X-Rspam-User: X-HE-Tag: 1733609720-386006 X-HE-Meta: U2FsdGVkX19Z6oYhfbwg0ppJBYaWJVCFZMmKKux508bhxdhY5G7cdj/Ptek0v100WON1THl2fRYTtciA0rv9rBVHlRMnL/EHF0p03qrBzt0oSu1i9YyX8taq3gEQuo1vDK8PUj5Bs6vlp7SqF0pb+9O1F2DGQ5o2yqu2fbgoFhzDnxiag4Rd6VYOSeMIfRAC7g1Q/T1FwfNPC+JNH9JIodtzoOOFNfov+NuhVauUMN4v0DVC6anVLhvt/eH3EdVoipWeJeI6pqUICCtrna5Yle0bPZG62K5JVx2kRG4XfFQXsVe9yp91+MiaZZdPGpuuicCKVMqniTBwTnmjncL8Wvs/6nA4ZopKYXl7Irn3eRqe8gCdhc8slgd0lJIYdAyh3CL2PUzZvrfAQ4wIaJe9F+15dq1iQagVD4OQKoG3BTHKgfx+r0yS3RmuC98xzFAVObXn7dXRmxNPaL0uIyiMJLQiro8hhyS6Tq+eEhnxFK2D9YHkITZ5SFnhTUzK/wNpKYILEfJ5xBl4jvuT9fz5YJsvM5/vQhdIRPIXoZwNHiyTlFSu6uc8n7aCq8W2eedQTuaiLvTqrMA5ZsGJ3dkArCxE1tMgxOkFg0TOUBczlQgbalESEiDuGvWzfFGmTdrbl2NJL7uDxIl69MWJETUjrpdTRO55/vqDzJ5aBb+IHBQY7pW7Zr+EFtkFFyspvPmeWUATnzvY1QWFd/nIB9X+gDnvGo5EEoecI8H8uFm8Q+MQCVO3NE40HQdRsI8zXoyOPkDP5/w263kca592O5+VaNO0tu4pgUzv4FQr210S/NTZlYUnjHwMi9lGgP91p2UUYt7AyHU0qvpFKKpdC5YIPvJ5K6fdmpLgysjPAkioDA8Z0F0wqfo9Xp59AnTNScXt13xELuB3W1GvD8HEmFUV2BbWBdzVe6etyAHNqgGSSNa4950slfYdejQPR4HuWsJ+S6cJl/FU1/EoyuW1ftu MGoj9wxw qnmELZMD0DVrKXAxztLLSCChGhZrZTbdBESZy/OGRu/bB1Xl8nn4BJfigIC/0UPWg1A7vPztKpgJ6szcYD6ylI3yHsyKsth0wH49CE3jYR/+D3sN3Kton+l0aazrXmPCmeK8D4fscdtcFYnTaEREt+QEOGkzOsToUojgH4dyBVBSGsQ8C1o0UYLChkxWZzaqlkkaP1F0etcyX/xQ0xyzZ0XmhSJ1oiDH36Vapd7mGQyDTUbAE1K6I1Zg8PZy1Hi/EPUS6PuXwK1eZyGwBb06qsryJxXkAdquEUCwNTMIx2fE8yDSeJ64EE8eW8826TMiON3mmXlLaVVSuNQhx9O7mPWCgcfUHJPiH6g3gEG4LNu0Ic+iZGatAaKLmMLoLRIQEZ4ALcVoUOLhfWItGr7M57rOBiChhZfHmBiT2F6mgs+XaOD4YVGIxFFZ5NCOEFcYWtG8SCvwTsk4SywKDR06165uISr/Y3yiBBB6MYtPSX04xgyvyMUc1LFRaoGqTwscSTfIxD6dlUTSey6+bPYDiP8oDQYBFYh0nLNX2yLKepzUDs3AS3XJ8tq3qxeK4rM0yihzZbDT5s5pneF5kgryBXvS9X3OqmiKZBgE+fyuiJ9bnkks= X-Bogosity: Ham, tests=bogofilter, spamicity=0.010037, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework workingset protection as well so that the comparison of refaults between anon and file is always on an equal footing. Specifically, make lru_gen_test_recent() return true for refaults happening within the distance of MAX_NR_GENS. For example, if min_seq of a type is max_seq-MIN_NR_GENS, refaults from min_seq-1, i.e., max_seq-MIN_NR_GENS-1, are also considered recent, since the distance max_seq-(max_seq-MIN_NR_GENS-1), i.e., MIN_NR_GENS+1 is less than MAX_NR_GENS. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: Kairui Song Closes: https://lore.kernel.org/CAOUHufahuWcKf5f1Sg3emnqX+cODuR=2TQo7T4Gr-QYLujn4RA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/workingset.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index ad181d1b8cf1..2c310c29f51e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -260,11 +260,11 @@ static void *lru_gen_eviction(struct folio *folio) * Tests if the shadow entry is for a folio that was recently evicted. * Fills in @lruvec, @token, @workingset with the values unpacked from shadow. */ -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { int memcg_id; - unsigned long min_seq; + unsigned long max_seq; struct mem_cgroup *memcg; struct pglist_data *pgdat; @@ -273,8 +273,10 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, memcg = mem_cgroup_from_id(memcg_id); *lruvec = mem_cgroup_lruvec(memcg, pgdat); - min_seq = READ_ONCE((*lruvec)->lrugen.min_seq[file]); - return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); + max_seq = READ_ONCE((*lruvec)->lrugen.max_seq); + max_seq &= EVICTION_MASK >> LRU_REFS_WIDTH; + + return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS; } static void lru_gen_refault(struct folio *folio, void *shadow) @@ -290,7 +292,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) rcu_read_lock(); - recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset); + recent = lru_gen_test_recent(shadow, &lruvec, &token, &workingset); if (lruvec != folio_lruvec(folio)) goto unlock; @@ -331,7 +333,7 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { return false; @@ -432,8 +434,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool recent; rcu_read_lock(); - recent = lru_gen_test_recent(shadow, file, &eviction_lruvec, - &eviction, workingset); + recent = lru_gen_test_recent(shadow, &eviction_lruvec, &eviction, workingset); rcu_read_unlock(); return recent; }