From patchwork Mon Apr 17 07:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13213418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBB90C77B70 for ; Mon, 17 Apr 2023 07:55:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71BFF8E0003; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CB258E0001; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5939B8E0003; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 48D738E0001 for ; Mon, 17 Apr 2023 03:55:25 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 187881A0491 for ; Mon, 17 Apr 2023 07:55:25 +0000 (UTC) X-FDA: 80690123010.03.4E528E8 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf29.hostedemail.com (Postfix) with ESMTP id B932512000E for ; Mon, 17 Apr 2023 07:55:22 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KvH8DLjs; spf=pass (imf29.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681718123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n+Co3Djx/FKBpBr6IJkJ4VOe8yza5ZJvAJ+PK4QALyo=; b=Mm8s3+q2qcOuMnCruo7Is6hM4PaRr2a4NBDoN0xfCge2tlJTj1/TufbZK8k2wd1+9nyuiW XC2t6Z8ZSVHa657L28K1QM5CWkopTmloMmXrKrND6cPcUvnDFSF91PoaLZin03FUYaV0pY NhJI7APTFMxRD8Lo8QWLafmqFe4szDU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KvH8DLjs; spf=pass (imf29.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681718123; a=rsa-sha256; cv=none; b=U+3UOjhGF8w+nqApepMl4U2kJnNT5WRF8Hv4WXb3wk7Zs6uhW7faQfuYy+eLHOoYS/ghlA n2w58DNdbzSoATwvYvLibaCeEqAUYCFPXkcVQZoSN96IHsQWxMlQGehR6e8zuwM852n5V2 4bOr0mZFVBtGOeCDhx4STKIsL9fVLfQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681718122; x=1713254122; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Lrd7bJnxVLV7szkBP4122haJtiLJgqqBdVKx0Hm+jA8=; b=KvH8DLjs+NPj5j/NOyeNe5qmo4okrnel5XkM+oQ6RRk2YBChc2JYCjKm 8cdJcOvBDDgNPOoqmtFvdF7jgOvsNrINRpTiF5ymmFJHuCxC7UoOepH3y 5X/zsK3kWhzONOJsZ8m9AHA55e4BP8rmrqjMu2P8j6B7aNRoThnpWHWl5 APgySE/eZmLk5KRzUXEqs4LoaKLEC8fUns0zyiHZRNkQAgdGYr3uN+vim 5HaAnHzNHvQqNlyIy2eP4dL/vGkdOdeVEyGEq/Y9KXwSa1HTF8rfg0bKU WRFHBUNCue8qFEnRYjr0RjFd/yz+AGGBTF/J23GRD/UzDpVHqQPTjMd8L A==; X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="343586973" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="343586973" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Apr 2023 00:55:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="755226640" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="755226640" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga008.fm.intel.com with ESMTP; 17 Apr 2023 00:55:19 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com Cc: fengwei.yin@intel.com Subject: [PATCH 1/2] THP: avoid lock when check whether THP is in deferred list Date: Mon, 17 Apr 2023 15:56:42 +0800 Message-Id: <20230417075643.3287513-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230417075643.3287513-1-fengwei.yin@intel.com> References: <20230417075643.3287513-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B932512000E X-Stat-Signature: qdfhzhb6gb3zbrbkf6144socw89m9xyj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681718122-583375 X-HE-Meta: U2FsdGVkX1+aUDiJpXGAmjUyLHIU6xKaf6cIhpDOK/MHQelbtKbQv/OEkCwDhZ7JnV9a3M+749rWyB0bVH8DuslBfQZshVrzik2im8nHPl9ZKX2DgWPpnU4ooGLXSRhKrUZJ8G2H1yATFRxhXTCX7RLX41b8je5jSS1dj2AD+L3oDFg09Ee1AZYnX9tdjFoK6SU6C5Nc13FFdedcqn4uQMOkG91R7AkFYTr0H7fNa4YDrmtdCmJEyRh3ykOleOSl9U6ByHgD/28jlWBUvTTaDTZSJVQ1vzBPRXoEV05W9VNYZX7goqYap/kER5BwZQxGCejjYO19qw4gylnP5/YMjZltnNAv+I83EckG8cpIZYRW9KxX7YAMqB3qx8GEM7duJ7SvLkCmQUM5cLrtPBQ8VU0sBBZdaEnX3yZYKCqRC9hIKwvmV/08Un/E9x2lppQVof60xRWoCKwQf/SX9SiGW+Ik2YV5ALRpaxlKeKU4vMcNGng4Kn5qcuIEv1AC40UgO8OVWY1uLTpVCMSQAXhtWzZCRFW94Qq4ZZnlp1AHRDJb61q3w62siqWhDD7NlMMzpA5ychCmHpSNg5wZ7olzr63+Iv9fyyzipUcuSsbXf5ezPfJk1yXPiPa4xriI5NSR/j6IWtjnp5l5fT7I4XFZELuq+yoWMtu5+fUIsthHkGF12NQ1+NNmrK6cejtdeiomWefZgdQl27Lm3L9JtAkxlfNVySnKLBRbrZctI2D7X3YkUC4PBrTy4WwodqSQkZIVFcxCiAXeV7PLWuKaguNnIldrQAHa6Isl+e6MBDrPPX0ovHVBOnHt0dWOiQGHjMMBstNXq/B0l9hnn9kfIr9coUy+P141+snOOZHmhnAaziKUgtajWq5nquuIR0NVpO7B68C7m0ZgJq7zHGR5RAdb5AMnnuSG6pTschbtaxUJyAR0W/yNj65Pav7jaExaoIgOKs05eWnVhE4PeDaq6mp 8rUDNiXe NG5P2wnRD/oXuYmCQ32hKN9DO0/hdiT1r+EoKxeW4UEVJkK+v8cUh8gNy+AiR4sjiARrnQ+c6DKVNe8jFYcUyBh/TwA417M4OaSIaLTwFTLvEzRp0+zDsfDCW4CZj0JyOoFyGKhzYw07wqvUvTrleJetaKqYGA1VM88YBQYPsl1FRlog= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_transhuge_page() acquires split queue lock then check wether the THP was added to deferred list or not. It's safe to check whether the THP is in deferred list or not. When code hit free_transhuge_page(), there is no one tries to update the folio's _deferred_list. If folio is not in deferred_list, it's safe to check without acquiring lock. If folio is in deferred_list, the other node in deferred_list adding/deleteing doesn't impact the return value of list_epmty(@folio->_deferred_list). Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 71.28% 0.35% page_fault1_pro [kernel.kallsyms] [k] release_pages - 70.93% release_pages - 61.42% free_transhuge_page + 60.77% _raw_spin_lock_irqsave With this patch applied, the split_queue_lock contention is less than 1%. Signed-off-by: Yin Fengwei --- mm/huge_memory.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b0252b418ef01..802082531e606 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2842,12 +2842,25 @@ void free_transhuge_page(struct page *page) struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(&folio->_deferred_list)) { + /* + * At this point, there is no one trying to queue the folio + * to deferred_list. folio->_deferred_list is not possible + * being updated. + * + * If folio is already added to deferred_list, add/delete to/from + * deferred_list will not impact list_empty(&folio->_deferred_list). + * It's safe to check list_empty(&folio->_deferred_list) without + * acquiring the lock. + * + * If folio is not in deferred_list, it's safe to check without + * acquiring the lock. + */ + if (data_race(!list_empty(&folio->_deferred_list))) { + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); ds_queue->split_queue_len--; list_del(&folio->_deferred_list); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); } From patchwork Mon Apr 17 07:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13213419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A14EEC77B72 for ; Mon, 17 Apr 2023 07:55:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40D7D900002; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BDF98E0001; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 285CA900002; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 197D28E0001 for ; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DAC7B80426 for ; Mon, 17 Apr 2023 07:55:43 +0000 (UTC) X-FDA: 80690123766.07.EBAA053 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf23.hostedemail.com (Postfix) with ESMTP id BD39C14000A for ; Mon, 17 Apr 2023 07:55:41 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F2lP4krB; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681718142; a=rsa-sha256; cv=none; b=TwgqWW2Cac8vDXmvlIUBxm7aoWkuc066KBQ2IPLSoBPDZRek8AVn8SrPLWFsgdm2nwOPKO zVzXbNCdH5f715ynlUezmnr4o3SsYEXOZtse6ZEqJXKoPUAc4/7uMbnqBuX0t8SySJKCQV Ka7p9rwdRvODO9fnvMym8Joyber7j7k= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F2lP4krB; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681718142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YplWFnCAw5REiCL+91O+oQDL0q3EFb5pySxQGdi9SXo=; b=QHYrgu2fm57K7ZXMmivTQs45q+Pxi+c3as3huCo7PFrc+U2UfzhcqoP0w1cfLJqEm62T2q r2tIBZrJ0vl4XCVF07TxmcvN/ut4yVpwD0qLsBmMEIr/INO/81sIRzj7z2YafPVuKnMpRd UpNcr5fu1jz+xHBv0wNYNi53YMDOKhU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681718141; x=1713254141; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2aOPxNaioygwyhundx6GQZJnjyHxRbJrogJB60G/n8k=; b=F2lP4krBqMbpiozk+szBVE/KAXiuF9+8toakv2Hg9B1FTqPoswtjsSrF +6ijcl7wh7f9H1cuB6RHGloEdRSxJgfEkJ55wXwlAMakKxm49mQOYmPFd Hbr4hfApXX9o8n7cGm4Quo0ZZM5yEMFqWclco1PXrw+WYErq7TGTjdLXa Wj69s7LfiGRxfz12FNW46k1Z2N/ueU0vieFt22PTVOT9bQqrCl/p9+Scb qASXDpLXQkOAuYHdzFXaPObPnqPue6JdeqzvVjyY8E1ubfa7msdr4DjSQ NFibzJReZ/eHHA2bi26cbo+wAflGiH7dUHnBiz2UTqwCtCYEJWLP0A+I0 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="324453865" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="324453865" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Apr 2023 00:55:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="723167476" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="723167476" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga001.jf.intel.com with ESMTP; 17 Apr 2023 00:55:37 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com Cc: fengwei.yin@intel.com Subject: [PATCH 2/2] lru: allow large batched add large folio to lru list Date: Mon, 17 Apr 2023 15:56:43 +0800 Message-Id: <20230417075643.3287513-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230417075643.3287513-1-fengwei.yin@intel.com> References: <20230417075643.3287513-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: BD39C14000A X-Rspamd-Server: rspam01 X-Stat-Signature: wf8j39q9e3bsemahc8fnpt4ry9i9x8rs X-HE-Tag: 1681718141-994292 X-HE-Meta: U2FsdGVkX18U1u7x/JHu0uyOyg9cZhf31cAvHlVFQRox4lQut73DtKOAzR67sNATLk9a61HCtKfO27oI7+1WKCuWSIH5YH2yRdpZgRHxHV7kqO83N0XjsWGhp99j9NjQOKkAAPGiYxw/iIR8e8X8sYL/i+oVtMmoB5pKwc9jPxcW0lxAJ6CXNHNRL0epqD040RE+0hVJPdRjhnEzG8KKjzKEDE9jkom0+AWpnGBf+I1xVQ0xV5ZLfmbPjiZqunbEGyt27lYhSmx9aoa6fJC6d9JaMCJLobe7hHbu+rBZRoLeXa3/zAStiQn7Hqk2ZsqsClqKo1ARzMiUUMhYUJgEBF1eZdepbdjgwxSViI/lr5ChAnHpWbEEir40TukG9HChdXEN2cZmhMa8urZ9B4332jbEgz+ULNvCskZQGebC627rpW6Q3hbrXv/mMXgM/EyO8O54xMKnnPLqA7vta621q0LF7yOLeMIfwm7qo2RJNDpM4X0IpsO4NOoVl5NW+yDzj5kB3TQif0AtsR2ao5xoFfXC21UCjVHG/kLp5KAqDksF2SyRXI4DvJorimemmzWB2hvjj5dN7VeOO+lqXyD2UqXXmgLlAQwUxHGXv/8brHjb/QOMJdnCs/Ard3hyXr1Q8PRFag/+BvqTdTttTZCWlxbryLBxXxJ4U/mCqUA4kUaqjbMsqtyB47T7w4zwX7lP3+jdbj5PHlKxZYgOniqwyJBNAyPp9sXNY6QPKBho4+a/8jDg4iA3s41VaSX1GOkejuEAw3QL8v8M686h0+eG1hQ+chEqz7KdMfC2/hyn28GkVxnWlYsn8rT3zjwr8zK1sir5sjZMgHN0i+NbytLVf6aCWEYksCe6DlI3DpPEA3wv7Pru6UDo5WteXkgZgR8tX6QBeMn1mFnUH6akhupTExHF2SGsQULGjfCD5NSRMbBgIjsUyxd0g7x4i5ikznqi32IWqki1kf0JGzJDAx7 pyC4SSLf AOmnGp89auaFZqIxnyjNUofPTnlVqSfoYqzJiKxNKP13UDxk2isCclLkw3hDQ9o/6jr0vNxl9xQSAk4PS1zQ5nPAo6PTdEj5lAN8RhQbUdqzXUUpp7jmBrijmtu6T3vOqikPF24aVKhkxVmgVV3xiTH3nSHoBEqHUM+tdxotGIvVMTns= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, large folio is not batched added to lru list. Which cause high lru lock contention after enable large folio for anonymous mappping. Running page_fault1 of will-it-scale + order 2 folio with 96 processes on Ice Lake 48C/96T, the lru lock contention could be around 65%: - 65.38% 0.17% page_fault1_pro [kernel.kallsyms] [k] folio_lruvec_lock_irqsave - 65.21% folio_lruvec_lock_irqsave With this patch, the lru lock contention dropped to 45% with same testing: - 44.93% 0.17% page_fault1_pro [kernel.kallsyms] [k] folio_lruvec_lock_irqsave + 44.75% folio_lruvec_lock_irqsave Signed-off-by: Yin Fengwei --- include/linux/pagevec.h | 19 +++++++++++++++++-- mm/swap.c | 3 +-- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h index f582f7213ea52..d719f7ad5a567 100644 --- a/include/linux/pagevec.h +++ b/include/linux/pagevec.h @@ -10,6 +10,7 @@ #define _LINUX_PAGEVEC_H #include +#include /* 15 pointers + header align the pagevec structure to a power of two */ #define PAGEVEC_SIZE 15 @@ -22,6 +23,7 @@ struct address_space; struct pagevec { unsigned char nr; bool percpu_pvec_drained; + unsigned short pages_nr; struct page *pages[PAGEVEC_SIZE]; }; @@ -30,6 +32,7 @@ void __pagevec_release(struct pagevec *pvec); static inline void pagevec_init(struct pagevec *pvec) { pvec->nr = 0; + pvec->pages_nr = 0; pvec->percpu_pvec_drained = false; } @@ -54,7 +57,12 @@ static inline unsigned pagevec_space(struct pagevec *pvec) static inline unsigned pagevec_add(struct pagevec *pvec, struct page *page) { pvec->pages[pvec->nr++] = page; - return pagevec_space(pvec); + pvec->pages_nr += compound_nr(page); + + if (pvec->pages_nr > PAGEVEC_SIZE) + return 0; + else + return pagevec_space(pvec); } static inline void pagevec_release(struct pagevec *pvec) @@ -75,6 +83,7 @@ static inline void pagevec_release(struct pagevec *pvec) struct folio_batch { unsigned char nr; bool percpu_pvec_drained; + unsigned short pages_nr; struct folio *folios[PAGEVEC_SIZE]; }; @@ -92,6 +101,7 @@ static_assert(offsetof(struct pagevec, pages) == static inline void folio_batch_init(struct folio_batch *fbatch) { fbatch->nr = 0; + fbatch->pages_nr = 0; fbatch->percpu_pvec_drained = false; } @@ -124,7 +134,12 @@ static inline unsigned folio_batch_add(struct folio_batch *fbatch, struct folio *folio) { fbatch->folios[fbatch->nr++] = folio; - return fbatch_space(fbatch); + fbatch->pages_nr += folio_nr_pages(folio); + + if (fbatch->pages_nr > PAGEVEC_SIZE) + return 0; + else + return fbatch_space(fbatch); } static inline void folio_batch_release(struct folio_batch *fbatch) diff --git a/mm/swap.c b/mm/swap.c index 423199ee8478c..59e3f1e3701c3 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -228,8 +228,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) static void folio_batch_add_and_move(struct folio_batch *fbatch, struct folio *folio, move_fn_t move_fn) { - if (folio_batch_add(fbatch, folio) && !folio_test_large(folio) && - !lru_cache_disabled()) + if (folio_batch_add(fbatch, folio) && !lru_cache_disabled()) return; folio_batch_move_lru(fbatch, move_fn); }