From patchwork Mon Dec 2 03:28:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13889936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65E65D4979B for ; Mon, 2 Dec 2024 03:28:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1CD86B008A; Sun, 1 Dec 2024 22:28:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BCCF66B008C; Sun, 1 Dec 2024 22:28:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABD906B0092; Sun, 1 Dec 2024 22:28:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 861AD6B008A for ; Sun, 1 Dec 2024 22:28:36 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4432B1C7BB8 for ; Mon, 2 Dec 2024 03:28:36 +0000 (UTC) X-FDA: 82848586086.01.3F770E3 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf09.hostedemail.com (Postfix) with ESMTP id B0103140003 for ; Mon, 2 Dec 2024 03:28:26 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pgKd6v57; spf=pass (imf09.hostedemail.com: domain of 3YSlNZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3YSlNZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733110106; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MW8P2vBtR0tiHVuNXF698UnUQTzNbjkVYUdDrWgvFuo=; b=AY35d5Ky3qe7PMB9JP6BBORgsGEj3Ma+dWlSFps7Aa/EozBTmmyWQyF7viLzWxOsMP0ziz AWVL+EmvaMre1XzlSZrhBb14N8M3qD6wsFGUEhlqriIJQCgq6bujkgRXMtQns2NNSD52At 2sVtUVXWoAqEx05iRzqhf3Z7ce2bkOk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pgKd6v57; spf=pass (imf09.hostedemail.com: domain of 3YSlNZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3YSlNZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733110106; a=rsa-sha256; cv=none; b=R/xQhSgFApontObUEYaC27hWcr6ecmAGvq93pSfiCEhoDxLUJhQgUXWrNaz+/cDWd1mqlM 749UzkL6Kr5omYbuK3H47Ma0Oxowoh9rxNigZd/jVR/B78SQt5tvngFKPA/SaQqLkzCFrL ETJVuUN3OXYrtj/9iwgTDya2v0nmz3Q= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7f712829f05so2803709a12.0 for ; Sun, 01 Dec 2024 19:28:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733110113; x=1733714913; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MW8P2vBtR0tiHVuNXF698UnUQTzNbjkVYUdDrWgvFuo=; b=pgKd6v579OhzgvwEBfO6oH4HlMBlDtHuzQQWUbUeFL7AihQHu9teJmU7fG9EWeFuNf E7rwunsH8rPKIs6a800ZQ0TU+1s6CL/PVd5sJ2ALxVkuUR5RE5O5NNx9czfsfiGQoIzY rzF+2NSn0FNvVm0mq1tyPZEJVDUbKXlC9jJIKgeo3/qDtBxgjz5YAZcB9sty18YFd1tV KesddcV3ILgy8DTyAJYUu45SoCxG62SckDmf1MC5tlnxYvT+G+Pp4nf9+hTKZgZLX4so B+sDHCc+O/ZqXtqEUlkNc0Zkr1vheWpyD+RXfXlDq3P4sNHTSXdutJRI5oglnOpsZZwY yM2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733110113; x=1733714913; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MW8P2vBtR0tiHVuNXF698UnUQTzNbjkVYUdDrWgvFuo=; b=Fmd7cjRND2Vvr6NY2kfp2exlwkuHqnpyI58MRlVtZIrD9cgFQL7f4TxqKEPbv+qcxG 6URkDfKwYlfDzMob+cNCaoIHBZwFuObcziJtSozO+6MRaBegLAXLvht9cYMXSDcAxVK4 rbn4TNXjedLi2zFoazpPZJwclgDyasDq1grhe2jAC2S4MuuPgd+7npoBXLuREIndhm4V lVNn9OHLge2lqmvJHBxNxMUJeBaambvQRNmPfm4A7LEOAlXGEOtXeciybc1e2Y8LpSSp K8c/34evaioy/S4YSOiAEPy4wh+OtyTS891zVYqMVF842TBlG8EYEWxOE9XSGkL5naoS YNIQ== X-Gm-Message-State: AOJu0YxRBopqiKLV9PZUfzxJc5VS6wFB0Z91SXbneuiE8DmJC7NyAxfI HbvyoUXym+Yf3427t32BWKWEFLNEIsodMVwkJGf3l6wnFpcZ9/hlQyRwbT19RiaElAA5d2OwCfC /ug== X-Google-Smtp-Source: AGHT+IFcUZRmwT3M3+dp5We5TbG1PhbrCpT/ilv6sNt1Wnzg29EAa63YEsnLXpC87arqRV9kgWQHWdyeDzU= X-Received: from pgct22.prod.google.com ([2002:a05:6a02:5296:b0:7fb:e344:7ca0]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6d9e:b0:1db:915b:ab11 with SMTP id adf61e73a8af0-1e0e0b0f11cmr32948241637.24.1733110113337; Sun, 01 Dec 2024 19:28:33 -0800 (PST) Date: Sun, 1 Dec 2024 20:28:19 -0700 In-Reply-To: <20241202032823.2741019-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241202032823.2741019-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241202032823.2741019-3-yuzhao@google.com> Subject: [PATCH mm-unstable v1 2/6] mm/mglru: optimize deactivation From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Bharata B Rao , Kalesh Singh X-Rspamd-Server: rspam05 X-Stat-Signature: 88a7xraqgnqmw8gf4yzucg49ykuzz8go X-Rspamd-Queue-Id: B0103140003 X-Rspam-User: X-HE-Tag: 1733110106-25641 X-HE-Meta: U2FsdGVkX18Qe/R4AO+yyeRJInfksDE7ajvURQLy3zjCDGAYvGnW9yb1bQqlfD7QAnvIlrUaGfceND/vVeIcN/RASK6HiPtWS4q/PeRw786uypKy0qpTw42gzSzo6J8YkNwr0WsQ4opa7dl+2gPSw4WhevsEGkrKvcr3XhQRUMZwr5ipcUxpi6E3fEFa+Hei4AFkFk+A8+cuZtueKvBm5oa4I47w5JqJ+mkfFFzmD/00g9iRxcyun8MymEEAURfXAf42XnLJQqLEogFPCKAR+e875rHsqjwyAlQrGF/PKwcp8FJNcSqMMrwaz5hcpW6mZtvUzeQtJrOtzfZoOrq/Z3t+pYiB1h7Lo3Qqf23Zx7ogHdfVX1v8Z7g6JdBgNmN5BIdHVgvvjZvHUSBuTLooKZczhWrruwU5Cl7ZN2EHwRpKc4eXspYF9p7JFQZewaPpZ9ZWDMX8BzqTyHMQoyH/C7fnz6GzWARUBn386za1RLqk1XcEpT4KIVSar7kpydphT9dUvsUURlr1ol95glOO/1GtFEWjJiphOf6EI6KERypNC2LkJXGG/8AXm3vG8oQfxhgs0yN7cxH74UyFKn/4u063QzPes0ZoUOEjBTdjK6wtcSvZAVQqHKYPtsbK26oe4q8/jsYSxxUlNX2QL6Jr0oHeZpsb7w2odaJ+C+mSD7eYl9UheNy1Wy+Gi9KYujw4qbNyRjGINneb08c5cc2p1ljVzxcUn3AyeofVAoZYJj0OuUIUN2c2pMCV3fDa/NNF1mTLXWi1tHYG4x6qAFMkphvRldgiBLjQVUlUW9nBu+eFAEUWTC2KR0cPyLNBxeEHoR+EcPVvmzmCFI1tZEUnxWJeUeRZzqdkapKZLc+4cBiCi40IBVkde4lPHS1JlTL06FEFB3OCVft+JZ6gIxBYSaeF141ICSzOlPKd0PyJvHKoLV9YpT3UFMy9Fq1SEkRaCDWXWbAdh0jShPxCStw KU1LtkZ3 Dp1tWCG+hWHQA2Xjzk2+5QZJ6OG6+7U689LE7DiYGjrKQGD5GP++d5YCBY91EWTI9tyIvs9XM7T8UMH98+tWYcIuotab2eKw9ZfNIqb5gDAE2MjCi+5me5TXxEfzrfVRxO3sLmOEJCCL4RGVS6genxm+r28hIcnHF2yDn29XUGTnAgOyCOVg4M6J2YGStQYXIDVU3mzmVpDuu7SQjjR6RYxwwK2pLXBabkVwUj8BJ+TzlCoAC++OTDNUWpxc5Ji9WnX2nhc9aUBNXN0zFukX9X7efOgVlxEQ036HHVfbEiGKPHlRU1iqv8r4F7OATNMgkvL8naIWKGFPS554ywNs85NH26UV7ZiZ/H2KYK6sbkZbGMpmjNMmZvwczLhQ3Um47aQFxGOTPm4osShA98MJjMbm1gnDXIK74IGEdpXhsmh4QfA7OMndiquAx+AUSKeXvKcId7LL9f2QwP0G5xCvh4E6gdpEGwFulhIy8IlTPFaRzo00NrRoJRd3j8WE8xg2YB+TNGBRYlTY6vLDUmGV6Hw+7bMU4RDDuOD3gwu+aCsmUSuAx5hv2H8HjRaY/Q9ANxIJaz70kggFTUTic2HuKyRASGnqs4UpE00ZxXhXtws/+Ci0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not shuffle a folio in the deactivation paths if it is already in the oldest generation. This reduces the LRU lock contention. Before this patch, the contention is reproducible by FIO, e.g., fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \ -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync \ -bs=4k -numjobs=400 -runtime=25000 --time_based \ -group_reporting -name=mglru 98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl After this patch, deactivate_file_folio() bails out early without taking the LRU lock. A side effect is that a folio can be left at the head of the oldest generation, rather than the tail. If reclaim happens at the same time, it cannot reclaim this folio immediately. Since there is no known correlation between truncation and reclaim, this side effect is considered insignificant. Reported-by: Bharata B Rao Closes: https://lore.kernel.org/CAOUHufawNerxqLm7L9Yywp3HJFiYVrYO26ePUb1jH-qxNGWzyA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/swap.c | 44 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3a01acfd5a89..15a94be8b0af 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -379,7 +379,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } #ifdef CONFIG_LRU_GEN -static void folio_inc_refs(struct folio *folio) + +static void lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); @@ -406,10 +407,30 @@ static void folio_inc_refs(struct folio *folio) new_flags |= old_flags & ~LRU_REFS_MASK; } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } -#else -static void folio_inc_refs(struct folio *folio) + +static bool lru_gen_clear_refs(struct folio *folio) { + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + struct lru_gen_folio *lrugen = &folio_lruvec(folio)->lrugen; + + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + + /* whether can do without shuffling under the LRU lock */ + return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); } + +#else /* !CONFIG_LRU_GEN */ + +static void lru_gen_inc_refs(struct folio *folio) +{ +} + +static bool lru_gen_clear_refs(struct folio *folio) +{ + return false; +} + #endif /* CONFIG_LRU_GEN */ /** @@ -428,7 +449,7 @@ static void folio_inc_refs(struct folio *folio) void folio_mark_accessed(struct folio *folio) { if (lru_gen_enabled()) { - folio_inc_refs(folio); + lru_gen_inc_refs(folio); return; } @@ -524,7 +545,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) */ static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio) { - bool active = folio_test_active(folio); + bool active = folio_test_active(folio) || lru_gen_enabled(); long nr_pages = folio_nr_pages(folio); if (folio_test_unevictable(folio)) @@ -589,7 +610,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) lruvec_del_folio(lruvec, folio); folio_clear_active(folio); - folio_clear_referenced(folio); + if (lru_gen_enabled()) + lru_gen_clear_refs(folio); + else + folio_clear_referenced(folio); /* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal @@ -657,6 +681,9 @@ void deactivate_file_folio(struct folio *folio) if (folio_test_unevictable(folio)) return; + if (lru_gen_enabled() && lru_gen_clear_refs(folio)) + return; + folio_batch_add_and_move(folio, lru_deactivate_file, true); } @@ -670,7 +697,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled())) + if (folio_test_unevictable(folio)) + return; + + if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio)) return; folio_batch_add_and_move(folio, lru_deactivate, true);