From patchwork Tue Apr 25 08:46:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13223082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C04EC77B71 for ; Tue, 25 Apr 2023 08:46:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0896A6B0074; Tue, 25 Apr 2023 04:46:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03A606B0075; Tue, 25 Apr 2023 04:46:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E44436B0078; Tue, 25 Apr 2023 04:46:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D58AF6B0074 for ; Tue, 25 Apr 2023 04:46:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A13D3C02A7 for ; Tue, 25 Apr 2023 08:46:43 +0000 (UTC) X-FDA: 80719282686.10.B4740D3 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf11.hostedemail.com (Postfix) with ESMTP id 973EE40015 for ; Tue, 25 Apr 2023 08:46:38 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gSi0OB4K; spf=pass (imf11.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682412401; a=rsa-sha256; cv=none; b=oJ1tnlmNla6XQsURcoBGfeZql9qoRU+G7iLpLMfLYn3zpki6lwl+brM/M543bUmi/pJg2+ yq/9RaNHmlrapc3RL66mABjx4L967HQ9ytCZDhRsfGxnQ7Z02yqNQV90qxbfeIi/0gTCI5 BroRLWMKFBcioAzrRE92wYcLLEHikec= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gSi0OB4K; spf=pass (imf11.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682412401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NtLUJYvqGFEpl+U/E0TiUclpFyCf3kGdSKtkoK2Z5Zw=; b=MjNmm5YvZzr6CIoWPypMmGej8yGEF6jgI90uwBZG230vsFS4Pw3rHZrO2YQ8AkrLQ9ueR1 CzKk6X80rIjZOJY9bRL8FdqojgNZ4aj1fDomHujYsIiHKzHI7oqwLa3EZZeQmMXVrQIPUp S48ubQDbx78XwVrHXivGJ3ized6+Au8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682412401; x=1713948401; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pk9XCUgoTnR9WpxWD2YhzR7meCtyBUVgtKd3rO4gYZ4=; b=gSi0OB4KIWLgLK+MZE3JL/+332NPyppQMf/R4isk10AOkmaLXg+8Su1m WG/iTI/EUpro+LtAzTW25zacF5cUvVcImKgx3hbCMmOXadhTdpDA49hv/ kPRIKv6Pjp75/V0xS3eyLZeQfzleZg27ld5YI7GqyVKJhOIgoSIuOPQJV N7lLSlr8cVUZQp3NqsP7TwVIU8IT2VF10vFq92NhRLR4wz61dTO0kmF9X n3u5+sU7sWHZUm2tKPOsb89lggCVjDZN6K76VrywcY4/kCse3mZs3jnoU MRTyYkrJPeUopul7wSMU8zsS8jksFppGADsRACLMRPcovZ9FpQN0hMxyo Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10690"; a="345454721" X-IronPort-AV: E=Sophos;i="5.99,225,1677571200"; d="scan'208";a="345454721" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Apr 2023 01:46:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10690"; a="696098037" X-IronPort-AV: E=Sophos;i="5.99,225,1677571200"; d="scan'208";a="696098037" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga007.fm.intel.com with ESMTP; 25 Apr 2023 01:46:35 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com, ying.huang@intel.com Cc: fengwei.yin@intel.com Subject: [PATCH v2 1/2] THP: avoid lock when check whether THP is in deferred list Date: Tue, 25 Apr 2023 16:46:26 +0800 Message-Id: <20230425084627.3573866-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230425084627.3573866-1-fengwei.yin@intel.com> References: <20230425084627.3573866-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 973EE40015 X-Rspamd-Server: rspam01 X-Stat-Signature: fpsjwo6srk64g87x9wizwxehahud1981 X-HE-Tag: 1682412398-916771 X-HE-Meta: U2FsdGVkX1/rxoPWGVdB8Y8Ad6G20zbmlPHybtPL9DlbYcv8oTdM8aYy5tzpZ+WzO9ebZfXIT5yXjBUuMnRwWguZ7HFWSUPF/CrVgXcCwJRBtSC+qTOUVj8qSP1dPjJbXQLDRCM07fPJVctArT/AK1jFxZK0tugRZK/K7CHRmZ3aczw1u3Y4V1nKGL37EnPblgbQ/Zjhrou4DX4M6arFiIbhmv/roxUhqz7b9UTfykgsv/K8AEGN1QKCn0Ojp+P63bdvcv8sht2h/7utlr640MCXPF/FGzmRO2LzpLneMa9UM3+ZXBSmZtKr/VUKxoocVp92rTSz57BxvcIbxMXEOaxk3JbQivR071Mz41ZFSaS3UFLcddVKzU8fMWE44e9iXzwbHX9vQzJoKyA4DnXiwVgiKq2F+G0Wack9iT+PFVj3a9LQFZ9pEWVCqLqRHtJTInu2QcpnrKkvV1ArNbNwlYmEkQU2DxtcD0aga4DaGMapdxd7YWUF5dhAgZoFkHF5BC9qZsS+fx3ZKy2Puoxj9dhcSZTMN395yNg3t5hGZbSYGvCl1reX0Gb6qr9VVh7NbVGpg5Iwkq3jNnj997xD5SSQJ4V9zqXQlcnd6S1SnwEomFmCjP2b+XDZVL0QI3SmSUh/IAWol4U+nzSBaC3Zi4/7VLAw3J9Fr9NNnm+Ekb6UXjy6BLPiM8YnYEneBrg1yZxDHjMcYvb43NovQCOKovdGNe45sfgnO8kzW5HVUjTBvCmES7lFmooBKYVJX2Cfnv6Hviw2pNclLMrNYb+tlwUw7Bu1Q/WmsskRdcuocSCW0eCUJrSlBCEwWnIuwxfsITu6aiVexsX2Lrqi8MokkNsJEQlCq5e9tyxaPxIXvFGIuQ+WIrW3+vaLi9fYqB5Dw5dBQCPjSoJC4xbsA8XbGNmROGXQhL+K9gM+qrfShh2IjRw+8JPLnCCvjWUlQuo4UgUtEKrSIP+B4f/3r8o VptlO+KY UupqtP0HOp9CLQrmTGjFrmcJHQWHuCybEbcfDtjG1dwHTs9jj1BpgL9CTjRehdQmz+C6T3tlSukMnUSo9nmMAzmrN1ZCprRW/lYvocDxL0p58MFVvlKyNDEDC9WUL6wOOVYme6eA4ZYZ2WKsmYgdFkU2yNGIks1uIZiTrg7tGXJzuJryMVsJNRGg6f7ckpsxMU4FsSjlEK4STs/kdpPP8UtxvkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_transhuge_page() acquires split queue lock then check whether the THP was added to deferred list or not. It's safe to check whether the THP is in deferred list or not. When code hit free_transhuge_page(), there is no one tries to update the folio's _deferred_list. If folio is not in deferred_list, it's safe to check without acquiring lock. If folio is in deferred_list, the other node in deferred_list adding/deleteing doesn't impact the return value of list_epmty(@folio->_deferred_list). Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 71.28% 0.35% page_fault1_pro [kernel.kallsyms] [k] release_pages - 70.93% release_pages - 61.42% free_transhuge_page + 60.77% _raw_spin_lock_irqsave With this patch applied, the split_queue_lock contention is less than 1%. Signed-off-by: Yin Fengwei Tested-by: Ryan Roberts --- mm/huge_memory.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 032fb0ef9cd1..c620f1f12247 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2799,12 +2799,25 @@ void free_transhuge_page(struct page *page) struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(&folio->_deferred_list)) { + /* + * At this point, there is no one trying to queue the folio + * to deferred_list. folio->_deferred_list is not possible + * being updated. + * + * If folio is already added to deferred_list, add/delete to/from + * deferred_list will not impact list_empty(&folio->_deferred_list). + * It's safe to check list_empty(&folio->_deferred_list) without + * acquiring the lock. + * + * If folio is not in deferred_list, it's safe to check without + * acquiring the lock. + */ + if (data_race(!list_empty(&folio->_deferred_list))) { + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); ds_queue->split_queue_len--; list_del(&folio->_deferred_list); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); } From patchwork Tue Apr 25 08:46:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13223083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3A4FC6FD18 for ; Tue, 25 Apr 2023 08:46:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FBBA6B0075; Tue, 25 Apr 2023 04:46:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AA906B0078; Tue, 25 Apr 2023 04:46:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1726C6B007B; Tue, 25 Apr 2023 04:46:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 09D556B0075 for ; Tue, 25 Apr 2023 04:46:54 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AFE3BA0278 for ; Tue, 25 Apr 2023 08:46:53 +0000 (UTC) X-FDA: 80719283106.28.5AAC798 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf27.hostedemail.com (Postfix) with ESMTP id 8DBB940004 for ; Tue, 25 Apr 2023 08:46:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="G+7Hbb/G"; spf=pass (imf27.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682412411; a=rsa-sha256; cv=none; b=fYobxS1T1jjngeBdsQeABOzYMqnVPWBueG8ExK1mrKd3YOFNIqEbqCrR0ISwou3IF2ThyC rn7nDm2rqPDtoFqsObr8ZDeDZnBZMCol4uojAueOTMIQpwCdXgDos4wf3uyL0Z3PRY3G5z EuWfG5WHhpAfTYhBjXpE4v2clf14ZXQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="G+7Hbb/G"; spf=pass (imf27.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682412411; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B9a60uYmHapHplwK9leCSNLD1IQB8tnKmzDhZx1EX1A=; b=w1oEoOudY97HMdbCEYT+Qm5f7vOhlc847ALZLXVrxVObmrIzIbSSbLaOFm8kRJuFz/siXE oz/w0Jq978iYchB6RsFV5od+t9ZCaU4TeSoY/2KbyWaTjcrKCs4RRFWZZzpHQ74M7Ie/++ qAHGeu2dp+PyligQStXAOQnfUIlldyo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682412411; x=1713948411; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=s2Sp63pCj4ThAHewqyBtIzNBTpSoN+c965kAGpeyoSY=; b=G+7Hbb/GMnpFCphdwh9lon7+dFaxR42Z/GZ5yXNvDsPHBjLgTh3Ex2ba bGV6q6fYvWIsdUzDt3hc5a5nR+EL6DRVFsI+U3rSht6KmmEDvFm+VnQ6S iSifb8NZmuIa9/scV07vrtGUOVHu8yYj6D0qHCNh9l6PiNjrghCq/BpK3 vByoyGRfI/W+gHsZqIijc6zfmTegd5tOeLOmeFLh5nDcVMkLkJjx52eqa 6gllsIi2TZKurFQSgBHocXSC1l+zDgKP8JJZDbUmWtKGs8QV7kxHvWUf+ naSsNJj7yOpD3y7n+1l/ydkSAKEgGHXaH8jffTv6y/AgRVNMrh7VWPFkF A==; X-IronPort-AV: E=McAfee;i="6600,9927,10690"; a="326302955" X-IronPort-AV: E=Sophos;i="5.99,225,1677571200"; d="scan'208";a="326302955" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Apr 2023 01:46:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10690"; a="939685610" X-IronPort-AV: E=Sophos;i="5.99,225,1677571200"; d="scan'208";a="939685610" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 25 Apr 2023 01:46:48 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com, ying.huang@intel.com Cc: fengwei.yin@intel.com Subject: [PATCH v2 2/2] lru: allow large batched add large folio to lru list Date: Tue, 25 Apr 2023 16:46:27 +0800 Message-Id: <20230425084627.3573866-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230425084627.3573866-1-fengwei.yin@intel.com> References: <20230425084627.3573866-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 8DBB940004 X-Rspamd-Server: rspam01 X-Stat-Signature: pszuwcir57u3jc4buteqrak57i9mrwi3 X-HE-Tag: 1682412411-501746 X-HE-Meta: U2FsdGVkX1+JRmgl9eD8jubpv4eBv4xotOKBSmyRZ3l6OCs7y138WLONFUhIFw5Ei6NKs0VH9gJVLY5GhxGwGl6iPDQyGp0NV+8fQi4aAl7/Y+loIMGCjV3l3bGNlBlpxX7LRDzNQQCpjIJ0LCGcjUAtcr3k+StZW71ZtU3HPe+Fx66fFjgwID3oCFPTSEuxSgOK22S4kLmReXZuB5xzC1ceG+j2NZxE4Rd7qlwYv+elyKtRqxCRNcY+Y5yW1r2A7/V80nb16o8uYSDd+hTPaJ7ej5rl8K9K1v7+zzJ3x551rQVmiAk/h7adHNhPQdPZ1rw9z5sbsQhzh164edBo5Zt42i7wv8a82WoTy3q0F1/VkZA+HPlnaxeqiHDzlB5Mjmw2vdUl355iI2wYKx6PYk+AfFExWqs+eWNYIcSsUZ5Mag/5jpVe1w8ueY1WXDgdv5NigxLQ8NZI69WGQ5IvrIi9TlEF/k2rLsFIo65U1MyCr4z/ACUPpmQPIs/rdvMBhzU7NET1x/THOX0hCjF49tun1fynj74XF92OgoWjT7xjvRqdsYgMFHBOHSoo/1s2f3s+1KfgaaXANp8a5P0eUMpv+EMb+oLMPG5mz7eT6D6x05JOclDsJqY7Co3ltS4rtmX+n96iujCttkI9qWlHKH9rauDm7dmXyOgvoxNEFNlBe1S4CO1HWgYWyfHDHBF5cxWnULcYtc7AjWdSHsgja7jj+EUvC3yr79r+eOqME/ykwjZbGbabfdn5V+btcKqqpj3ppVVlRm3GqsyzXMupoVGAfiL6NhY2q8Lo9KsdU0x4EejRLqOa18SghoY3zU6hLTxgBZekyCUCeu6ZoUX33LvWUaORZP5H2iz26Kxt1WSWe82+fPpK+xiU1XAAxkR8DLHmmNmFdRwiXSqtOOO4gidpIOuknjY53xpWQeU22m7hJqv5BXdIAyafXiScenF6U8Vrii3EXQQGIbldoAf 6livL1Jg u1C9N88v4fMAO7tV/GQ85fFh+buKa7Z5/7levelfb3yrORSQEQkYuri4tK1BX5XLe3+r//nEKmG7+XTKMmNpGon315OzEHqI0tefpw/TtvEbHTzgv5/HTrb8om1hNHcr1oBz93JaKCSED0ISscn8oGa3q6V/gzhYHe632gLlt7WLR9f9He74O5y9KGKxdjmReYiZIsCkGVeytU6/POjUFSpajIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, large folio is not batched added to lru list. Which cause high lru lock contention after enable large folio for anonymous mapping. Running page_fault1 of will-it-scale + order 2 folio with 96 processes on Ice Lake 48C/96T, the lru lock contention could be around 65%: - 65.38% 0.17% page_fault1_pro [kernel.kallsyms] [k] folio_lruvec_lock_irqsave - 65.21% folio_lruvec_lock_irqsave With this patch, the lru lock contention dropped to 47% with same testing: - 46.64% 0.24% page_fault1_pro [kernel.kallsyms] [k] folio_lruvec_lock_irqsave + 46.40% folio_lruvec_lock_irqsave Reported-by: "Huang, Ying" Signed-off-by: Yin Fengwei Tested-by: Ryan Roberts --- include/linux/pagevec.h | 46 ++++++++++++++++++++++++++++++++++++++--- mm/mlock.c | 7 +++---- mm/swap.c | 3 +-- 3 files changed, 47 insertions(+), 9 deletions(-) diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h index f582f7213ea5..9479b7b50bc6 100644 --- a/include/linux/pagevec.h +++ b/include/linux/pagevec.h @@ -10,6 +10,7 @@ #define _LINUX_PAGEVEC_H #include +#include /* 15 pointers + header align the pagevec structure to a power of two */ #define PAGEVEC_SIZE 15 @@ -22,6 +23,7 @@ struct address_space; struct pagevec { unsigned char nr; bool percpu_pvec_drained; + unsigned short nr_pages; struct page *pages[PAGEVEC_SIZE]; }; @@ -30,12 +32,14 @@ void __pagevec_release(struct pagevec *pvec); static inline void pagevec_init(struct pagevec *pvec) { pvec->nr = 0; + pvec->nr_pages = 0; pvec->percpu_pvec_drained = false; } static inline void pagevec_reinit(struct pagevec *pvec) { pvec->nr = 0; + pvec->nr_pages = 0; } static inline unsigned pagevec_count(struct pagevec *pvec) @@ -54,7 +58,12 @@ static inline unsigned pagevec_space(struct pagevec *pvec) static inline unsigned pagevec_add(struct pagevec *pvec, struct page *page) { pvec->pages[pvec->nr++] = page; - return pagevec_space(pvec); + pvec->nr_pages += compound_nr(page); + + if (pvec->nr_pages > PAGEVEC_SIZE) + return 0; + else + return pagevec_space(pvec); } static inline void pagevec_release(struct pagevec *pvec) @@ -75,6 +84,7 @@ static inline void pagevec_release(struct pagevec *pvec) struct folio_batch { unsigned char nr; bool percpu_pvec_drained; + unsigned short nr_pages; struct folio *folios[PAGEVEC_SIZE]; }; @@ -92,12 +102,14 @@ static_assert(offsetof(struct pagevec, pages) == static inline void folio_batch_init(struct folio_batch *fbatch) { fbatch->nr = 0; + fbatch->nr_pages = 0; fbatch->percpu_pvec_drained = false; } static inline void folio_batch_reinit(struct folio_batch *fbatch) { fbatch->nr = 0; + fbatch->nr_pages = 0; } static inline unsigned int folio_batch_count(struct folio_batch *fbatch) @@ -110,6 +122,32 @@ static inline unsigned int fbatch_space(struct folio_batch *fbatch) return PAGEVEC_SIZE - fbatch->nr; } +/** + * folio_batch_add_nr_pages() - Add a folio to a batch. + * @fbatch: The folio batch. + * @folio: The folio to add. + * @nr_pages: The number of pages added to batch. + * + * The folio is added to the end of the batch. + * The batch must have previously been initialised using folio_batch_init(). + * + * Return: The number of slots still available. + * Note: parameter folio may not be direct reference to folio and can't + * use folio_nr_pages(folio). + * Currently, this function is only called in mlock.c. + */ +static inline unsigned folio_batch_add_nr_pages(struct folio_batch *fbatch, + struct folio *folio, unsigned int nr_pages) +{ + fbatch->folios[fbatch->nr++] = folio; + fbatch->nr_pages += nr_pages; + + if (fbatch->nr_pages > PAGEVEC_SIZE) + return 0; + else + return fbatch_space(fbatch); +} + /** * folio_batch_add() - Add a folio to a batch. * @fbatch: The folio batch. @@ -123,8 +161,10 @@ static inline unsigned int fbatch_space(struct folio_batch *fbatch) static inline unsigned folio_batch_add(struct folio_batch *fbatch, struct folio *folio) { - fbatch->folios[fbatch->nr++] = folio; - return fbatch_space(fbatch); + unsigned int nr_pages; + + nr_pages = xa_is_value(folio) ? 1 : folio_nr_pages(folio); + return folio_batch_add_nr_pages(fbatch, folio, nr_pages); } static inline void folio_batch_release(struct folio_batch *fbatch) diff --git a/mm/mlock.c b/mm/mlock.c index 617469fce96d..2c7e5043aebe 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -243,19 +243,18 @@ bool need_mlock_drain(int cpu) void mlock_folio(struct folio *folio) { struct folio_batch *fbatch; + unsigned int nr_pages = folio_nr_pages(folio); local_lock(&mlock_fbatch.lock); fbatch = this_cpu_ptr(&mlock_fbatch.fbatch); if (!folio_test_set_mlocked(folio)) { - int nr_pages = folio_nr_pages(folio); - zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); } folio_get(folio); - if (!folio_batch_add(fbatch, mlock_lru(folio)) || + if (!folio_batch_add_nr_pages(fbatch, mlock_lru(folio), nr_pages) || folio_test_large(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -278,7 +277,7 @@ void mlock_new_folio(struct folio *folio) __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); folio_get(folio); - if (!folio_batch_add(fbatch, mlock_new(folio)) || + if (!folio_batch_add_nr_pages(fbatch, mlock_new(folio), nr_pages) || folio_test_large(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); diff --git a/mm/swap.c b/mm/swap.c index 57cb01b042f6..0f8554aeb338 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -228,8 +228,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) static void folio_batch_add_and_move(struct folio_batch *fbatch, struct folio *folio, move_fn_t move_fn) { - if (folio_batch_add(fbatch, folio) && !folio_test_large(folio) && - !lru_cache_disabled()) + if (folio_batch_add(fbatch, folio) && !lru_cache_disabled()) return; folio_batch_move_lru(fbatch, move_fn); }