From patchwork Fri Dec 6 00:31:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13896156 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3CEAE7716E for ; Fri, 6 Dec 2024 00:31:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4814E6B0128; Thu, 5 Dec 2024 19:31:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4314F6B012A; Thu, 5 Dec 2024 19:31:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25C616B012B; Thu, 5 Dec 2024 19:31:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 01B886B0128 for ; Thu, 5 Dec 2024 19:31:39 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A6AF9AF0E1 for ; Fri, 6 Dec 2024 00:31:39 +0000 (UTC) X-FDA: 82862655750.28.2F79DCE Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf01.hostedemail.com (Postfix) with ESMTP id 96F9D4000A for ; Fri, 6 Dec 2024 00:31:25 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="PPrJdc/H"; spf=pass (imf01.hostedemail.com: domain of 36EVSZwYKCP03z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=36EVSZwYKCP03z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733445082; a=rsa-sha256; cv=none; b=lxP6lf+rZz6r0JZlkXl5Igm8Rr3yGJpj6UExdVNKb/U+X36gmXFamB9anlFKka7ElgIfJJ uPmNghn60LhsLUByvBPWRjaHqOGLcVv5e4+3URwuracrKAiZR/8iyDcyNnjZIju/1ySSss 99CG0R6nBCRKO1I4ea6IE69UQ+qq0QA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="PPrJdc/H"; spf=pass (imf01.hostedemail.com: domain of 36EVSZwYKCP03z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=36EVSZwYKCP03z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733445082; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MW8P2vBtR0tiHVuNXF698UnUQTzNbjkVYUdDrWgvFuo=; b=FSfXNGoNB58U5x31ry6HSgAq6gJv/XOGGhZEXhICHBZsqbLI5weYTGhflEYMzKq6QeKnnK 25lTJd7gNft7ENFZUhPdKra8FSdinVXe/++GPy4eMDpndmBG488bpzL/67Ax8peXNa3VSX /G9Ab3pnyRLeml9kf9xfB8keW37bNns= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-29e5f045c93so2186190fac.0 for ; Thu, 05 Dec 2024 16:31:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733445097; x=1734049897; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MW8P2vBtR0tiHVuNXF698UnUQTzNbjkVYUdDrWgvFuo=; b=PPrJdc/HS6oA782MnsGZHTGSUvkVCEBAKjXPbXkwzQoqmNWpD0/q1o3ZmZvv5sJIKB 9Qu5zF200QV7ox1QAhkx4Ph/wJaSiK+HczXOvcjNozkykvWI2PeAyK5q765PUciDL/3c oiirn/JPNif1L2hpJXF37DI9+Uc5tdNokZ5QF5ACG1ELXcGXcaqTI7/Tu7Fc9noiUttn n98ge+TxqmlIvCmQS8bj7JJDMWcph884C8DkIUXV+ReauMMkxIVv5jQcnKw3a5aNcGmP FzXC3f/fMp+WDOWh0N2Pf2Gy9RzM+H+Lijoc0Z9DTnrHIfaIZVriE0u41whrPFl74HkX Hw9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733445097; x=1734049897; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MW8P2vBtR0tiHVuNXF698UnUQTzNbjkVYUdDrWgvFuo=; b=MGuGFvZHluE3ssEmIgEYC4lCsRLiqLxx/oxH/ZDU/RZBrT+8Pn4P6rZ1M3Ba2AV3CC 1d0m9TmrABW+dYE2uTzwQ8L6qd+In3DLOH6UzRuh6kvVNadmMX93h811nBNGigtN8MFM lL2qwxTCpZE1Jul9cdITg9IWODYajBDv5t32cgSkzZva3lg7KD6+NYwHPLRMwwOo4Urs WKyAWzN6tP+2M0qIFoCNbJd6LvuAfq77e76BZ818m4pRN6X5vH4TxYF/83RkMtswYV7U vc02o0tVgrV2l3qbojU/JYsI+2DoL6bZe0Z0mXrAVvlYsOF6VgvhYFYyOshv/GX9QC4d zGBw== X-Gm-Message-State: AOJu0YxDwcdLCJEwOG1VD+SNkm3WJJPXp3NFYZy0wJn3ZbIkgUzXv8lO eBcPWLBIgGf2moSQxvyo3DT6/gy61jWUhRX8ge+8QGMNlDcPOrzZS6kdWVzx4KLmha4bXHHwKS/ uNQ== X-Google-Smtp-Source: AGHT+IHG56U6ff/2ISJCA5gXcXDrcTLQkj6CAfollUPpPgK2o7iXSrjbLgXvQK+fFy4kFbchimZeACpqNOE= X-Received: from oabra18.prod.google.com ([2002:a05:6871:6112:b0:29e:9a9e:5041]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:d90c:b0:29e:4b63:d7 with SMTP id 586e51a60fabf-29f731a2894mr1012088fac.1.1733445096951; Thu, 05 Dec 2024 16:31:36 -0800 (PST) Date: Thu, 5 Dec 2024 17:31:22 -0700 In-Reply-To: <20241206003126.1338283-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241206003126.1338283-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206003126.1338283-3-yuzhao@google.com> Subject: [PATCH mm-unstable v2 2/6] mm/mglru: optimize deactivation From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Bharata B Rao , Kalesh Singh X-Rspamd-Queue-Id: 96F9D4000A X-Stat-Signature: kh4y9cjzeayqmsadigrfnbeznts31s3f X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733445085-392889 X-HE-Meta: U2FsdGVkX19n2m+nOT9kmfMS6RnhWElydRAO8ux3dIgBhLv8QkwvmeD3ABHqO4PajyknRU7l/t7BGSqxCL3rN0tqxYLipiUQWpyP88vJBrwWmUUbWBNaaINOaLCYsx8/+n7Iu91DMxEYXaGTijrjIwTuQRqXmDucD0P1SWGQrRlr9YhjBPp+MqFUXERsVGoJRffuH6F1GjX12nevVDoTX+9oQOE7EOTPS5ZOA2Toub/0feAZEZpKhH3TYqGl74M+AGGNsCeBFyhq6Scxhuzo1EiWIEOBwsVaGwUX2MxmOOaiH9CF9YiKJ8Qvmst28AmVNi2YTkaWrnZpLWZJs+iU+f7nc6gxye5AosqUlVhXH0n9EhlzULBXrz5ON5J7Jj1NR2pZKQqNWb8mdBAZoVNG9RdgswBg+xkxdjyhRMA9PLBdoXZaddetDByekR1MgvJmA4UzMmPje1dvv0kMkCkC0whcAWpZZPji/hzA4/eFIBc8L6AVZn5ZLGDIwrL1EF5iCAcaJZQZLVvSSQBs6yOnfJcARSOBq8cyKZSyUiejHCsO2aJiWSDSpKa5WxK64fKi/418a+ITukRDtgIMU9ojNef6smYx69weSY8yzq7gBXJuFU/mhfFlwV4hYkEro8mC87/XNlguS/jnvr7m210o2eiCkjhs9LbqDFLwGbTM2qp5/vEbM1cFGX0x/Din6bvHw6s3US3hi0uELjO5XMONLqrhn6NsHgM6Anu30BgHlFfbcdr22/dLV7p1Uq2Xix6NAyIOwLlQP5daKuY+AauhK0z/tZXOk9+Vrdtl8q5dBjhEVSrHePqsQVZywfWuHxB170dUnHYLW7c1OB8WzrTjUhPhOKcwO0asvtwEIZlDaCnZh3Em59lPJCNmJsp6/qiLTKki8JBt+RyDjbgHN4L4tuYqUu5c8xC+nLU1bCiE5CHWbczpiEUCuAMC78jjj68HEfBV5xuayuhnY2Ie1EO 0PZTcod+ YjypHfXrv6xonGr/xy1SWZL+ovIkYKNIaIzzIGVh1wwAktsMLlicXkhUbymXuLPPDw0VRkjI6ddLdQ1PHTGDGi/Z0XGgBLJHNrqCgkUL62xfNfxf0Dlj71uYilwFtO3CEXZU7Qce2XN3quvffrOapJnC11Vai6ewrsYL8guch6sJcPk/jSL2CQIQgUd+Aid6kqagzfQpHCkLo1Bw2BPMqcAUWNcApXfoUVZknagUg2HxvyevIAa58CxWDO30/d3SCr+l3OGV3iOt0uOkGyNu1qGEPvrvmPf5hil8ZDAtrjXwyMSrhz3/Nj6Onp9krFdwh5La0XN+jiBDt0Lh9raTgv9i0B2OVqKKo3s9iVL0QYQuhOkHiweBAVCoTXMj6AX+GB8M9gyNUFN/6N1h/5XcUXRX8k4YcB3HrE+DpD8AVjCDiHcrTz+ErWxumnXIMizz20UYR2vn0/gWXQcQ1vUh14tHLrX61LVGABn7e10dw3vgvsjwgAEzt8WWVT8Qd74hif0orku1uCI6B/1/MiALNZAZQKUVDONbPtlFLfUqIFVsBG38xuavvcoHLT5k1+3V7XEPqMtZCJcYqmfY7JhqkDB758Swqh4WBzmw0HAdpWX2cFI4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.004072, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not shuffle a folio in the deactivation paths if it is already in the oldest generation. This reduces the LRU lock contention. Before this patch, the contention is reproducible by FIO, e.g., fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \ -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync \ -bs=4k -numjobs=400 -runtime=25000 --time_based \ -group_reporting -name=mglru 98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl After this patch, deactivate_file_folio() bails out early without taking the LRU lock. A side effect is that a folio can be left at the head of the oldest generation, rather than the tail. If reclaim happens at the same time, it cannot reclaim this folio immediately. Since there is no known correlation between truncation and reclaim, this side effect is considered insignificant. Reported-by: Bharata B Rao Closes: https://lore.kernel.org/CAOUHufawNerxqLm7L9Yywp3HJFiYVrYO26ePUb1jH-qxNGWzyA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/swap.c | 44 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3a01acfd5a89..15a94be8b0af 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -379,7 +379,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } #ifdef CONFIG_LRU_GEN -static void folio_inc_refs(struct folio *folio) + +static void lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); @@ -406,10 +407,30 @@ static void folio_inc_refs(struct folio *folio) new_flags |= old_flags & ~LRU_REFS_MASK; } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } -#else -static void folio_inc_refs(struct folio *folio) + +static bool lru_gen_clear_refs(struct folio *folio) { + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + struct lru_gen_folio *lrugen = &folio_lruvec(folio)->lrugen; + + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + + /* whether can do without shuffling under the LRU lock */ + return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); } + +#else /* !CONFIG_LRU_GEN */ + +static void lru_gen_inc_refs(struct folio *folio) +{ +} + +static bool lru_gen_clear_refs(struct folio *folio) +{ + return false; +} + #endif /* CONFIG_LRU_GEN */ /** @@ -428,7 +449,7 @@ static void folio_inc_refs(struct folio *folio) void folio_mark_accessed(struct folio *folio) { if (lru_gen_enabled()) { - folio_inc_refs(folio); + lru_gen_inc_refs(folio); return; } @@ -524,7 +545,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) */ static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio) { - bool active = folio_test_active(folio); + bool active = folio_test_active(folio) || lru_gen_enabled(); long nr_pages = folio_nr_pages(folio); if (folio_test_unevictable(folio)) @@ -589,7 +610,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) lruvec_del_folio(lruvec, folio); folio_clear_active(folio); - folio_clear_referenced(folio); + if (lru_gen_enabled()) + lru_gen_clear_refs(folio); + else + folio_clear_referenced(folio); /* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal @@ -657,6 +681,9 @@ void deactivate_file_folio(struct folio *folio) if (folio_test_unevictable(folio)) return; + if (lru_gen_enabled() && lru_gen_clear_refs(folio)) + return; + folio_batch_add_and_move(folio, lru_deactivate_file, true); } @@ -670,7 +697,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled())) + if (folio_test_unevictable(folio)) + return; + + if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio)) return; folio_batch_add_and_move(folio, lru_deactivate, true);