From patchwork Fri Dec 6 00:31:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13896159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29395E77170 for ; Fri, 6 Dec 2024 00:31:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC3726B0133; Thu, 5 Dec 2024 19:31:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A72246B0134; Thu, 5 Dec 2024 19:31:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C7426B0136; Thu, 5 Dec 2024 19:31:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6375B6B0133 for ; Thu, 5 Dec 2024 19:31:48 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 19970161903 for ; Fri, 6 Dec 2024 00:31:48 +0000 (UTC) X-FDA: 82862655120.17.60A316E Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 645F280013 for ; Fri, 6 Dec 2024 00:31:15 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cabHbOqK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 38UVSZwYKCAg627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=38UVSZwYKCAg627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733445098; a=rsa-sha256; cv=none; b=ZpogRO3W1fCyOh31iJ/fH6PJ1bXe6VTQdQ50Ra6bYyyUNj1K/OJNIsILa1YavFeHYgldia mFPQ03akA7BS7llzPxT579ucJqR1jR+ybwMHkxlJIPc3/ScohF4+PVYBk7H/Nav6MGCu3r yzL3lvubKXyCY/01RXqp/yFaDQXGPN4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cabHbOqK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 38UVSZwYKCAg627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=38UVSZwYKCAg627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733445098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=BygYtUkHeSK9/GK60pwaZCsiZyA9ZZaxTp8hjgo/vg8v6+h//OjiQ1Qw2X5T3WS0HDI7i1 xAeKtVmHgqB28Uuiofu5UOgbZ5X/93mf6KPywnvNcKSDbx60YwsT7Tt8dQtEAtDvSNi8K2 +wxPdkXu2u0ihfrmRDXIlyve3vQCVWU= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-215b30585b8so16142355ad.0 for ; Thu, 05 Dec 2024 16:31:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733445105; x=1734049905; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=cabHbOqKSKWi0qfx/2mbXmKIIYABpQtLno/4xMG1kuYdH9mfWcBZpEGxXv49I5AZ0v 6R9r/OHy+d5y7ms4YQkPad0mdQn86m1/9OjDqhf70F4FLpAhVHjBO9QOmHS3djrydAlp MEi47Fy5nin4yhnX8x7oOkLjO9oGOmC2HhWhyhSVabzPpAebcRZMg/5VQ/9tl9iUWunh MS1og+POvSr8nzDERuPPwxqC9dROar96KlDYE6MYOQMymPmKOpzqNP35B7BVWF+k0o69 +pvvwJovNWvQ/K5aXhqfqSyFgigo8ev17/+crGs8eoCO4biaWztAzJQ1NCzWzsDV8Wy3 F73A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733445105; x=1734049905; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KW2cgcwHB51Y5IYZaisd+Bp0HxAyG3VEQ4AUms4DlWs=; b=R1YrwtOeIeSDQqtsdrEspqchL47Cudy1ORMGKBUALY5SYBxKxUferB6cQBcLyL78sE 7gOuiWCZ4clWAjohsnBXcCnIhWF1ntpyRIoOHJX7ZyhP7agwvULTA1boPAfS3tfVIESs gSs9a9YSAtzGfjrT+NDgA+gEWHgGJQydjaLluMkG1l1RNW8EhjvsL6D5R8nNc8QNQngh wZtjy3YVJHqAx1wxFd/wjQzYqgGTElT+mlzbAkbTrFVgnzQ5BoqH+EibChUkHOmYVoT1 SmisEjHJU3BmeHyzY1eE0mfYuWI/Ydbx52jMK8H7NZxiJPD6ZpYGIh3OF1WyIj/lvFnk SUJg== X-Gm-Message-State: AOJu0YxjERFEWA4f7LBUZVHlMoiWUHup/iDiGBqb+OvNpU5rlt+TGPcT vzHFVLaB4+HhMRfQh5YNXZAMmvUxRtWSzU2Ykxeree0U1zjB4ALOiDwhlSl9TRM2uWd9fTmxUeL WGQ== X-Google-Smtp-Source: AGHT+IGcBw64P6PC43lwi6CEzaODCf5NQzsvKmv+pkev0Vd7uHGWUIM2qXHUyE9c6qDAYcuhyiX00sGOta8= X-Received: from pllb6.prod.google.com ([2002:a17:902:e946:b0:215:83cc:b135]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:24d:b0:216:1367:7e3d with SMTP id d9443c01a7336-21614d67fc3mr14295335ad.31.1733445105156; Thu, 05 Dec 2024 16:31:45 -0800 (PST) Date: Thu, 5 Dec 2024 17:31:25 -0700 In-Reply-To: <20241206003126.1338283-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241206003126.1338283-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206003126.1338283-6-yuzhao@google.com> Subject: [PATCH mm-unstable v2 5/6] mm/mglru: rework refault detection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kairui Song , Kalesh Singh X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 645F280013 X-Stat-Signature: jqcwuuyrxknqzaepxdbceyqmfbrj5fmo X-HE-Tag: 1733445075-663449 X-HE-Meta: U2FsdGVkX1/L46T9/6Z57DV2TmICR4i0Vs+e4KKWGC9dpEU7f1i5yU6fZT0GlF4Yic9lXaR/gGrvib/XVoXVASli0dAEutyu5j7YHf42644ip8+ygJjjesMk305qGfnMxInQv+wcGDuPBlBmDxLdVLU9aWX9u2GTtjdfRcLeVUUAriJ3mVs/7jISWFdcXU2D1hkHClmRmpz5ehvCm332cs6E8L455NWvSxIPSMPAshyIVc6M8rCzrt1/MQv9/Z8ZETsjfBkTBWXsbr2AtbUHHowdPdony1qBMG7G4zeLgRqYHFFNafJ0G399NN/kTm6yoyHU/vv1qSXwJVTwlt7wEksuP5DJK3AWxlAEAoEmz/Adw43Ao8jS3xkUQLDpZ83xOYnHv98tjkCS0oiQA7CNExZ41feo/jeE9KlRBH/lvW/wX+iKSP8kTb+1isNH2vEqJ8+MSTqereWMOJKSyFJUgDn8SRDC+tx7sXHDlw95fBVudBT8Y9ANNsHCpQ7HJBo02t2/aFtpJiJag35Q+KtemwiV6oemZ9yRRyhjDVtQ4JKuUhoUzvyTisRzIbfUQln7ZLDtnnpk/Tq9x8HRE4cLQmVUfQH9kmbWTSMMxhl3Rw6f45VfyCgvaI8C06v9OjscjM+4touc/U/kAD9yJfGIYrdxOzY4Hfcryw3Qoqg0dTLezMqudVdZu8oGOQb5qr9C9d0G2QeXhgEF0Z+tkKOYHlZVQFZdnTj+kLLncJT2QyB6AlJpBo6FLDlE4z/WpCHmYtuZOcCXhoIubKFoT91Dr5J8tzQbFc5l1Jq6Tenn4jZv3lwosyr++SrwhnZda2xkKNYnsyKS+w/AYFiWwWrlxSYzdXQKTsNnw5+BQSRddOBE5hwBWYW4vtamT2Z1CFraPfjdOKlOzGNCslIkwhqnr/PWLKfZOrlIkOJpAQPwcElbAXu8AadQKJcQKiy9vgUchfpor5qdS3ddpb2aRUr myfIgMc/ pOb4zZnXgJ39hcvz8c5fLV67qE1zGiL2f52U4YowR56yXC/Zny7UEszwf/LcrvN0iX75KpCIqwGdx3i72OvwcON6A6Su7qhkazhXIqoPqPUodqprTYyGTbrmK1ogpic4VDFO3iz3PKCYq6aVvr9N5+5DlKeWocM01F3vRN30QQLd3ht96zz0HRIHoCb8E2oVP1aWnwDQcJxmfe3RyFn8xpDLLWcVdVgFAwPcdWVVIlVhIL3N697F6SaKAVDWVblBmr85T8GqPxYhtyGlTp4JFBnUdnfuoobiyMQDyUF6nVUp5YDJY+6VgNRK9cAMe3i/Zc+HS9scodHFwaWfZ/kDMeEtbcBaQR78EYICeRvjAyTPwRUkpT059xiBNDiSQM7IKMl6OxaLiZ55YLqY4qGxzeHBC11loUPjvah6f+kLg1jK+a/LrHOgy0Xtkm/C8FtVeHIlWpE1UCzWSucIa0rqYcP7RvOt02gfsbjN6jiGfGCIeaxzx3H03SpvHQUz4IRoIS8prhU9UVE7Tb7S/P7c+GexA1lDWMGiKSnIfDIohg/RzIqcPNlTH4VEU7dkLfX5DDRmnCaY4CNu2VTWAgT5QpuPG4xvm3okv2dJmSlTizzwIPE0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000520, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework workingset protection as well so that the comparison of refaults between anon and file is always on an equal footing. Specifically, make lru_gen_test_recent() return true for refaults happening within the distance of MAX_NR_GENS. For example, if min_seq of a type is max_seq-MIN_NR_GENS, refaults from min_seq-1, i.e., max_seq-MIN_NR_GENS-1, are also considered recent, since the distance max_seq-(max_seq-MIN_NR_GENS-1), i.e., MIN_NR_GENS+1 is less than MAX_NR_GENS. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: Kairui Song Closes: https://lore.kernel.org/CAOUHufahuWcKf5f1Sg3emnqX+cODuR=2TQo7T4Gr-QYLujn4RA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/workingset.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index ad181d1b8cf1..2c310c29f51e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -260,11 +260,11 @@ static void *lru_gen_eviction(struct folio *folio) * Tests if the shadow entry is for a folio that was recently evicted. * Fills in @lruvec, @token, @workingset with the values unpacked from shadow. */ -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { int memcg_id; - unsigned long min_seq; + unsigned long max_seq; struct mem_cgroup *memcg; struct pglist_data *pgdat; @@ -273,8 +273,10 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, memcg = mem_cgroup_from_id(memcg_id); *lruvec = mem_cgroup_lruvec(memcg, pgdat); - min_seq = READ_ONCE((*lruvec)->lrugen.min_seq[file]); - return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); + max_seq = READ_ONCE((*lruvec)->lrugen.max_seq); + max_seq &= EVICTION_MASK >> LRU_REFS_WIDTH; + + return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS; } static void lru_gen_refault(struct folio *folio, void *shadow) @@ -290,7 +292,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) rcu_read_lock(); - recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset); + recent = lru_gen_test_recent(shadow, &lruvec, &token, &workingset); if (lruvec != folio_lruvec(folio)) goto unlock; @@ -331,7 +333,7 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { return false; @@ -432,8 +434,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool recent; rcu_read_lock(); - recent = lru_gen_test_recent(shadow, file, &eviction_lruvec, - &eviction, workingset); + recent = lru_gen_test_recent(shadow, &eviction_lruvec, &eviction, workingset); rcu_read_unlock(); return recent; }