From patchwork Sat Dec 7 22:15:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13898423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E187CE7717D for ; Sat, 7 Dec 2024 22:15:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 394AB6B0364; Sat, 7 Dec 2024 17:15:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 343588D0015; Sat, 7 Dec 2024 17:15:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E4196B0367; Sat, 7 Dec 2024 17:15:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F2EBC6B0364 for ; Sat, 7 Dec 2024 17:15:35 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A31EB42835 for ; Sat, 7 Dec 2024 22:15:35 +0000 (UTC) X-FDA: 82869570210.05.3F55F92 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 4677D40019 for ; Sat, 7 Dec 2024 22:15:18 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kYvM+LwZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3A8lUZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3A8lUZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733609724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CCT5euYl/qJQPeGGazPVJuZRHShIdP4mKhuDso53GD0=; b=picuV8ZFHx5NCopsuWkAuNpTlQS7bFaexx8QUWOK7SKIwAGtA4WnBRrMi2S/EvqZA5wVot 77A8AdeLChEjPK/sWIxNWrzk02Q+b3Q6+4p99PoRXnqmcE3lxKn62bugrZ0GqMJyyZ3eW3 xNh4t8O2iJOegdFa2zZiHSyhsnh6KXc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733609724; a=rsa-sha256; cv=none; b=lKfvBVtnrMUS4UfyTelw3VI+WgdHhc6WG8XJwvT010uwc7Z4doz102Z601f0/vYWH0raL+ b3mzW5f/SaOOQwiDUwDYuvzbmoVLCq9w4P52GbxtEABJM9JofmgP/rjjmQST1prXpuayro pXCA2HsQ5IynMAaQrgArb+IXILV7IOE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kYvM+LwZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3A8lUZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3A8lUZwYKCCoeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7259d005dc5so3401967b3a.3 for ; Sat, 07 Dec 2024 14:15:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733609731; x=1734214531; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CCT5euYl/qJQPeGGazPVJuZRHShIdP4mKhuDso53GD0=; b=kYvM+LwZA83hqjPZTkraFlDWAY/nVaneVoUr+WHdykFbJjSvv+MgBmDufNOlruaHTy P6QOTBul5IiC3ovaiKjanMxkFBIa6Vxw6bRYfA1p3nV+BvCTcGSHmDREI3gG26OInyo9 CAZWXkKUCogFZO8wW79BifNKkeZcD3j3sFTG30g5WpM0xC5S4CmxnxvdzpD9ugfELYWT 78Z1Kd7QZ9W/1ZyCSqsysfs1cLoEKJdhCjrAUxegK4wBMKmMQDZiRNbJTJvzcbh4xAJw qSMf/9oYWQvNGRVQCDrEO+hBmlLqtgKrcTO+cOyvHtukk64N1b6nR8902lb+R+llBGUf Buzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733609731; x=1734214531; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CCT5euYl/qJQPeGGazPVJuZRHShIdP4mKhuDso53GD0=; b=l0r3d4xxvdiP+N8Q/GwhM8S0107XkZQu/QfE8LpLYjx8R0mnmERgjqBAkCofW0w/qv LuDvBqcvDS62WFXLhgfMfISzQqGHQifeXhoICdggaiuOp7F+UQ6b3akOcaKl8AB//pKA vS/qtnElytwyIRRUoDNOL0TLlgWiKASYuNGFBSCfaeRSMQs2eu2qiZg9dJxoPkwKafvE +zXnu6A1LD9PLNAHP+L4WdQYqCF9dIV1pRiOCLNQpZ4nQkQfDouaQAOWr6G6+szi63F4 anK/K8Py8XYo2MHlPoyhIcyWUSJw0kAj1Q+lMcFSihYQYVertoNR3Vbr452OBYV+u01T zOrQ== X-Gm-Message-State: AOJu0YyVzrQx2XZTNLF1fIUFhTdOeFtOCmABSC/aDz6G5PoIN1y/ZrkD WdRyp1CqCCcIh4VTRIQn8oQFqrS2jagkYw1ZAYjAGAk2NjIXyaMU100L09OhvuXlCHhDoHb+vm5 pLw== X-Google-Smtp-Source: AGHT+IEX0/0XlihpBl2av7TPcnGj//5IebmVqNnUYSl/vdQwzILN65v6v1O4wV+Mrra4vxLhPsMYzWB6m58= X-Received: from pfbbe12.prod.google.com ([2002:a05:6a00:1f0c:b0:724:ec9d:6e1a]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2381:b0:71e:573f:5673 with SMTP id d2e1a72fcca58-725b8144a42mr8420603b3a.15.1733609731167; Sat, 07 Dec 2024 14:15:31 -0800 (PST) Date: Sat, 7 Dec 2024 15:15:18 -0700 In-Reply-To: <20241207221522.2250311-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241207221522.2250311-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241207221522.2250311-3-yuzhao@google.com> Subject: [PATCH mm-unstable v3 2/6] mm/mglru: optimize deactivation From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Bharata B Rao , Kalesh Singh X-Stat-Signature: woogp57atmtr1jtgmaymwxs6szexdcu4 X-Rspamd-Queue-Id: 4677D40019 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1733609718-264678 X-HE-Meta: U2FsdGVkX195FKmcEe6DqNqq9l1va2guXiXqGjYsYexm/Oc9wfV5xPCb8z8MID3zQ5liGJxQUMJfJpb1dhQ7YYlvExqZvVJjztPRWMfiYrZkSnc/dF/zsYYshy0+UIyhgYRRbrb1nvGfutS4BEBGmLV+IuZbT84VPRo8Rrk4c1EA45KRirFNxwT8hnBf7ODxxDt4kF7vEn9F6GzOttAG50dq4+z3pi320JNheYqXjRpsP7egv1bIeWOrGj8WeKGIkedxS8mY1p3sMzh2xZtxYUEMIudEheQZkYdVzGachDyo+47yU6lqyhkdtUCpkRHE1o8VPrDC0aHhwklyZvWC4/dgYgU339rEen3orifYfr4BO3mMRMZRZf/wQeXjcXlHmEC7BE+xjH9BWUHBNsePt3jO3wsDxjjpHNWY8Pxj7et5O6oDCfDzdGcuPj/jltkUySX5RhH0uuEaV+PB6IASUx4AFkvAQ2xBrqrrflcgnAGSNzu0m6OUbKYgp1ddrTugC6b46cRo7KIwcGzTsBEmu0+LiVFKDsJgPtf7vMOwJU4CbxePWKPa36ogSGtoyJ3nJzihw2xXm5j58AAWFMBnu26k1Z8ScJWxWygvs+TdqAZcCoi6THDbqssQCyZTv1hGJZ7iOM9Da6BWMK7FRc4llcNfxu1l3IBKzVwjSjmOKAXbZ8EV5mardhukCPpUD+Oe/nzaqn+0xWbs39fqeXWHbaruFfSPUdhrEhfYdC725Q94IJ1DII4spXzkcFz67vpDxIC3E8cbtZmv/M4bFalUKt5fTkcp1LiNNeXBZ7B/Z75050q5BhCWPqDehP6Cv1F8J7oEzfUlgQS1qc5UlEIlvBmJO8bkFJMob8RiJH9ZTPiCxp/mi5G0F+IfTrm2hauUw8lasSxl4fpgAJ/v+krfFwJ6+OdO2EN9KKyYyAmgvuhHPsgRt8hxAViJAv6Cz9dsDHZTV/LQARgBXlLSisJ Mv5mdlDh X6xYDtyjKbdAysUspY0jmjZbg1oB785BYfTOb9Bode6y/2VyE1H8TOthe3hWbc04xcPQ2+Z49bdE92KxJ7Y3hwUGd6UDLI03CNsep6p5LFBGqKZ91nrYxmQ0CWkVBDWoFXCG3wCpw47wj1fjCRzEf93ZcdWCG3ML/wa7uzAlPggtdwETb6xKwse8tVJi5Nme5413R+SOi0Y2OAOWy51NG1Zv3BmGslcmyLYpRuRyh6243P2UbaVz5VnPuDGrux6wr9vzw6RRBGu9iTATDqaOdcLAzRDb0NDvNIiThaMwYzqMla0NK04/XUx63V9pQ9xbUtrdcDo48fQ2vcuJebdaZ/TX1b5m6nhBQq0StMh/fTPtoGVRr2DuPz5jRUXPWtj1Sm/9xnufpd/Bw1KynyRa7wQDqXnFjzsj1JX9IDDhptQJ+1ornua75jfnpoC9pnSdCV7lP7ujO3KYJeABCIYWVfcYWgIGGEGHRTrdnIWnqgXx8qspHzbu9Scu3upqlzE0bxEMkiVRRHt3J7iRdwCSZbIxSNetWsq4Wt2GiNvc1iFycGnCrRhhpzj3yi3UtVSI+s3nBq0fh1JvYbGKVITlYtAYHCmukKddNKaxJsDOnm0PNL+Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.011409, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not shuffle a folio in the deactivation paths if it is already in the oldest generation. This reduces the LRU lock contention. Before this patch, the contention is reproducible by FIO, e.g., fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \ -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync \ -bs=4k -numjobs=400 -runtime=25000 --time_based \ -group_reporting -name=mglru 98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl After this patch, deactivate_file_folio() bails out early without taking the LRU lock. A side effect is that a folio can be left at the head of the oldest generation, rather than the tail. If reclaim happens at the same time, it cannot reclaim this folio immediately. Since there is no known correlation between truncation and reclaim, this side effect is considered insignificant. Reported-by: Bharata B Rao Closes: https://lore.kernel.org/CAOUHufawNerxqLm7L9Yywp3HJFiYVrYO26ePUb1jH-qxNGWzyA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/swap.c | 49 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 8 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3a01acfd5a89..756b6c5b9af7 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -379,11 +379,12 @@ static void __lru_cache_activate_folio(struct folio *folio) } #ifdef CONFIG_LRU_GEN -static void folio_inc_refs(struct folio *folio) + +static void lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); - if (folio_test_unevictable(folio)) + if (!folio_test_lru(folio) || folio_test_unevictable(folio)) return; if (!folio_test_referenced(folio)) { @@ -406,10 +407,33 @@ static void folio_inc_refs(struct folio *folio) new_flags |= old_flags & ~LRU_REFS_MASK; } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } -#else -static void folio_inc_refs(struct folio *folio) + +static bool lru_gen_clear_refs(struct folio *folio) { + struct lru_gen_folio *lrugen; + int type = folio_is_file_lru(folio); + + if (!folio_test_lru(folio) || folio_test_unevictable(folio)) + return true; + + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + + lrugen = &folio_lruvec(folio)->lrugen; + /* whether can do without shuffling under the LRU lock */ + return folio_lru_gen(folio) == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); } + +#else /* !CONFIG_LRU_GEN */ + +static void lru_gen_inc_refs(struct folio *folio) +{ +} + +static bool lru_gen_clear_refs(struct folio *folio) +{ + return false; +} + #endif /* CONFIG_LRU_GEN */ /** @@ -428,7 +452,7 @@ static void folio_inc_refs(struct folio *folio) void folio_mark_accessed(struct folio *folio) { if (lru_gen_enabled()) { - folio_inc_refs(folio); + lru_gen_inc_refs(folio); return; } @@ -524,7 +548,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) */ static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio) { - bool active = folio_test_active(folio); + bool active = folio_test_active(folio) || lru_gen_enabled(); long nr_pages = folio_nr_pages(folio); if (folio_test_unevictable(folio)) @@ -589,7 +613,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) lruvec_del_folio(lruvec, folio); folio_clear_active(folio); - folio_clear_referenced(folio); + if (lru_gen_enabled()) + lru_gen_clear_refs(folio); + else + folio_clear_referenced(folio); /* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal @@ -657,6 +684,9 @@ void deactivate_file_folio(struct folio *folio) if (folio_test_unevictable(folio)) return; + if (lru_gen_enabled() && lru_gen_clear_refs(folio)) + return; + folio_batch_add_and_move(folio, lru_deactivate_file, true); } @@ -670,7 +700,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled())) + if (folio_test_unevictable(folio)) + return; + + if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio)) return; folio_batch_add_and_move(folio, lru_deactivate, true);