From patchwork Mon Feb 12 16:35:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13553619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12BB5C48297 for ; Mon, 12 Feb 2024 16:35:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A11218D0002; Mon, 12 Feb 2024 11:35:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C1A78D0001; Mon, 12 Feb 2024 11:35:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 863798D0002; Mon, 12 Feb 2024 11:35:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 73FDD8D0001 for ; Mon, 12 Feb 2024 11:35:18 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 104BF40AB6 for ; Mon, 12 Feb 2024 16:35:18 +0000 (UTC) X-FDA: 81783701916.24.38BA7A6 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf05.hostedemail.com (Postfix) with ESMTP id C6C7C100026 for ; Mon, 12 Feb 2024 16:35:15 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b="L znyQ+W"; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=weVyV5yW; spf=pass (imf05.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707755716; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=BbXi9w879xgqvVwZonwpLom0NLIzmEYgtiUFEjEguAc=; b=FJ/hLnNWwsHuruJxe6S1IdUY7jobDjRBoDsyEi7OcTVLSLYgb+be61JHRhb9r+no05ZM1c uuaJ4EBznsqj4cifrIyc5+2EGRrU5qGyieUCqeN3xW5bAZcA+UoH469ETQHoH1SSS33Mga YO1tUr4VVA9oIPyWRdbg1oUhxauhsMk= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b="L znyQ+W"; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=weVyV5yW; spf=pass (imf05.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707755716; a=rsa-sha256; cv=none; b=JwKwtQF0AxAq2Bf+G5F+TVt6xH/OMlECzKHYJXwFCJTrBQt5j7hVXaR9EjaLor0JFXgI6I giS7oOTw9EHzCb471oVvyYbnC4M3VPfIKup8Maaq1qx40/+g3Dhzfodk2i4VU5SvEUCWLG o7Zp26fIHXjLC9nBzzzBeF/qUNGN1eM= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id CA4755C0055; Mon, 12 Feb 2024 11:35:14 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 12 Feb 2024 11:35:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :reply-to:subject:subject:to:to; s=fm3; t=1707755714; x= 1707842114; bh=BbXi9w879xgqvVwZonwpLom0NLIzmEYgtiUFEjEguAc=; b=L znyQ+WVMPxNsRO3tOLxO9ib+qWCTHfQwe3DJbVfyYJkf0dAkyV6hYd7nCCtF1Hyj S2nb1v7aID/oMk4SMmPRwwa6BiVcPm/74dVEafQlQBGwk1wTSj64kXBaRZBnXgAV Lr23/yCTSYuZGva7eWuwKH6Jscq7VZQ7HSFUpYnqPyoNvkVhSIcQ9s/m8WM4J0kX vXYuwQrcPoh4ziFC+isSUbCOU3slvHWqcd1u2Oviu+pHoEoZ6BCL3OQaK7SX0l7B 5pVSsI6vOu5SGLPRkvMVj0t7b4OenL+FVGkAh1zQQ0fGIh+HoPA7Gy+0s9XqePXD ROBGVDKonaa6FMtIv/ryg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:message-id:mime-version:reply-to:reply-to :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; t=1707755714; x=1707842114; bh=B bXi9w879xgqvVwZonwpLom0NLIzmEYgtiUFEjEguAc=; b=weVyV5yWZswZrDC6d l7jn0x36uDHOdOkOU6JgsrYBfIpyGbJVhtXwltLW6IH4ZHAE9TAQIRjuwrH6UBHv oca8xWShTiLnMlgLsYghs/QUPgpU6Z51WORxyL4AqH9j62Yhy8fRvh3ZkHDTkMhz TLjaBQ8h/wW10YBfjbstKypqkFG7HRjB9pyFdMPI/7CD8lp51J4MEKmJYJ6C3HHC /hYYdPBMc8Ja6dj4RjkP9XpLnHBJrKZrLmpG6I4CtvtoQ+Z13vrAS3qlMmp6X7d1 bdbuaPhPz4LAo1ew3aHi2BcOThwaBS8CmXhSEJgMgXng6VUt7VrIRQA0IgUiE5jz oZxWg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrudefgdeklecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkfforhggtgfgsehtkeertdertdejnecuhfhrohhmpegkihcujggr nhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeekhfelud fhhfejtddttedujedtueeiveeutedutdeutdfhffffgfeufffhgeejffenucffohhmrghi nhepkhgvrhhnvghlrdhorhhgpdhfohhlihhoshdrmhhmpdgtohhmphgrtghtihhonhdrmh hmpdhsphhlihhtrdhmmhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgr ihhlfhhrohhmpeiiihdrhigrnhesshgvnhhtrdgtohhm X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 12 Feb 2024 11:35:12 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Zi Yan , "Huang, Ying" , Ryan Roberts , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , "Yin, Fengwei" , Yu Zhao , Vlastimil Babka , "Kirill A . Shutemov" , Johannes Weiner , Baolin Wang , Kemeng Shi , Mel Gorman , Rohan Puri , Mcgrof Chamberlain , Adam Manzanares , "Vishal Moola (Oracle)" Subject: [PATCH v4 0/3] Enable >0 order folio memory compaction Date: Mon, 12 Feb 2024 11:35:07 -0500 Message-ID: <20240212163510.859822-1-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Queue-Id: C6C7C100026 X-Rspam-User: X-Stat-Signature: isa4qaktbutrb35k6n6b4kod5nccckr5 X-Rspamd-Server: rspam01 X-HE-Tag: 1707755715-146135 X-HE-Meta: U2FsdGVkX187ZtAetmFvQCegNmpY87D0D6neD4Uz+Lh1sDEpjiC4imoLBHAB3tXRzoXLkvQsvcZ/kNv0m4TCCoexHIStUTnuAXMIMJK3Sh74XZ8nndCus+B5AC1GUE2iFzB4faksx60vRfkcn11YwUAohB1Ffu3iQUVcOFJn0VMRlzRrJmkm3l2M4GpfWmsAe0851Us4JFwsHL8jMN8CmzpG6KAmMn6OV63CIeNq14ZG66gZwQ162Fk08dMOo+FKbLK3HiNBQEPM6nig6qYC6Wwn2MsoBAtdDbYXmOugQbJxwiqKIR/nkmtWeIk3Fr84CeHjuA9oDQT2QUrmab1D9zHcc5YuMWtEmnNTT5katnkA2GGfYPBBi+Et12UN0knP0dNMYKVf0pEpLdqn8AJCgnGKsq73OH1xZLqZs6ZQwuYBfNnuqiVImdHmtgdApnN2IbcNUqfR1KVu+qOB26q1E2yQxnL4BfcW93Bvkr1f2w3Ggiy3DjplZyxUlFCUbPWHIHFTCipIaZYONLkXNCdYWfl8Gfcj3jJPy76SW3RMs97ON3sP43mvS5nBxahAoXrxunz46KA7xYcEBcd7h3mnUD/xmaGn2FVqglGvyCoaaItac+XM84FtvJQd+vV1Cym97CC2h+lvrA52LyYOBT3HB56gF/uewM3pKSEm4ePpFYt7gcXgfqeTYORmVFeZvmacUY4VjOnp/6NuvV72KH3VFMdrDGpsvadw5v84Rnlwlj/IcfCf34roZ2AahaCKNO7VlnnW17sff+B697QBepN/O/X/5yyAtrbB8XJhNsop7a9aSS6uHeUXbchmA/9e6T2t24cmWZWDiD2BfQyHDJtbPMD+gkX0/2HL88hY4/pCkSZOtW6dbiel2WIEQPGupEb/Iq57U0tNlyhzaSG3Ns+a4euO2qz9NcTSsIu2XAoYx72L3ETMRgx7TJ6v6EeCkzpjH5HXrr7eNylhCCcM1zt VFhQsNkq 9qYDBBBWxbBI6lBxzWHzVh+UtUxQttbCrET6OEt4JiD65pHRTMDvBzRm51psowFOdHZf+5zqCBEjNz4SgAco5OVN3J4ZbMz6jb+BjxbOp4qIRRhNy8+OdyMaERJkKeHJCx7FSM9OUDY40qMQdC4zBXKkVCGlliU5hJXbr7fAP7wqqU44GSdTX9NiTog== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan Hi all, This patchset enables >0 order folio memory compaction, which is one of the prerequisitions for large folio support[1]. It is on top of mm-everything-2024-02-10-00-56. I am aware of that split free pages is necessary for folio migration in compaction, since if >0 order free pages are never split and no order-0 free page is scanned, compaction will end prematurely due to migration returns -ENOMEM. Free page split becomes a must instead of an optimization. lkp ncompare results (on a 8-CPU (Intel Xeon E5-2650 v4 @2.20GHz) 16G VM) for default LRU (-no-mglru) and CONFIG_LRU_GEN are shown at the bottom, copied from V3[4], since V4 is only a code refactoring of V3. In sum, most of vm-scalability applications do not see performance change, and the others see ~4% to ~26% performance boost under default LRU and ~2% to ~6% performance boost under CONFIG_LRU_GEN. Changelog === From V3 [4]: 1. Restructured isolate_migratepages_block() to minimize PageHuge() use in Patch 1 (per Vlastimil Babka). 2. Used folio_put_testzero() instead of folio_set_count() to properly handle free pages in compaction_free() (per Vlastimil Babka). 3. Simplified code to use struct list_head instead of a new struct page_list (per Vlastimil Babka). 4. Restructured compaction_alloc() code to reduce indentation and increase readability (per Vlastimil Babka). From V2 [3]: 1. Added missing free page count in fast isolation path. This fixed the weird performance outcome. From V1 [2]: 1. Used folio_test_large() instead of folio_order() > 0. (per Matthew Wilcox) 2. Fixed code rebase error. (per Baolin Wang) 3. Used list_split_init() instead of list_split(). (per Ryan Boberts) 4. Added free_pages_prepare_fpi_none() to avoid duplicate free page code in compaction_free(). 5. Dropped source page order sorting patch. From RFC [1]: 1. Enabled >0 order folio compaction in the first patch by splitting all to-be-migrated folios. (per Huang, Ying) 2. Stopped isolating compound pages with order greater than cc->order to avoid wasting effort, since cc->order gives a hint that no free pages with order greater than it exist, thus migrating the compound pages will fail. (per Baolin Wang) 3. Retained the folio check within lru lock. (per Baolin Wang) 4. Made isolate_freepages_block() generate order-sorted multi lists. (per Johannes Weiner) Overview === To support >0 order folio compaction, the patchset changes how free pages used for migration are kept during compaction. Free pages used to be split into order-0 pages that are post allocation processed (i.e., PageBuddy flag cleared, page order stored in page->private is zeroed, and page reference is set to 1). Now all free pages are kept in a MAX_ORDER+1 array of page lists based on their order without post allocation process. When migrate_pages() asks for a new page, one of the free pages, based on the requested page order, is then processed and given out. Feel free to give comments and ask questions. Thanks. [1] https://lore.kernel.org/linux-mm/f8d47176-03a8-99bf-a813-b5942830fd73@arm.com/ [2] https://lore.kernel.org/linux-mm/20231113170157.280181-1-zi.yan@sent.com/ [3] https://lore.kernel.org/linux-mm/20240123034636.1095672-1-zi.yan@sent.com/ [4] https://lore.kernel.org/linux-mm/20240202161554.565023-1-zi.yan@sent.com/ Hi Andrew, Baolin's patch on nr_migratepages was based on this one, a better fixup for it might be below. Since before my patchset, compaction only deals with order-0 pages. vm-scalability results on CONFIG_LRU_GEN === ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19+ 6.8.0-rc1-split-folio-in-compaction+ 6.8.0-rc1-folio-migration-in-compaction+ 6.8.0-rc1-folio-migration-free-page-split+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 15107616 +3.2% 15590339 +1.3% 15297619 +3.0% 15567998 vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19+ 6.8.0-rc1-split-folio-in-compaction+ 6.8.0-rc1-folio-migration-in-compaction+ 6.8.0-rc1-folio-migration-free-page-split+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 12611785 +1.8% 12832919 +0.9% 12724223 +1.6% 12812682 vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19+ 6.8.0-rc1-split-folio-in-compaction+ 6.8.0-rc1-folio-migration-in-compaction+ 6.8.0-rc1-folio-migration-free-page-split+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 9833393 +5.7% 10390190 +3.0% 10126606 +5.9% 10408804 vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19+ 6.8.0-rc1-split-folio-in-compaction+ 6.8.0-rc1-folio-migration-in-compaction+ 6.8.0-rc1-folio-migration-free-page-split+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 7034709 ± 3% +2.9% 7241429 +3.2% 7256680 ± 2% +3.9% 7308375 vm-scalability.throughput vm-scalability results on default LRU (with -no-mglru suffix) === ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19-no-mglru+ 6.8.0-rc1-split-folio-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-free-page-split-no-mglru+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 14401491 +3.7% 14940270 +2.4% 14748626 +4.0% 14975716 vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19-no-mglru+ 6.8.0-rc1-split-folio-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-free-page-split-no-mglru+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 11407497 +5.1% 11989632 -0.5% 11349272 +4.8% 11957423 vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq-mt/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19-no-mglru+ 6.8.0-rc1-split-folio-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-free-page-split-no-mglru+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 11348474 +3.3% 11719453 -1.2% 11208759 +3.7% 11771926 vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19-no-mglru+ 6.8.0-rc1-split-folio-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-free-page-split-no-mglru+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 8065614 ± 3% +7.7% 8686626 ± 2% +5.0% 8467577 ± 4% +11.8% 9016077 ± 2% vm-scalability.throughput ========================================================================================= compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability commit: 6.8.0-rc1-mm-everything-2024-01-29-07-19-no-mglru+ 6.8.0-rc1-split-folio-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-in-compaction-no-mglru+ 6.8.0-rc1-folio-migration-free-page-split-no-mglru+ 6.8.0-rc1-mm-eve 6.8.0-rc1-split-folio-in-co 6.8.0-rc1-folio-migration-i 6.8.0-rc1-folio-migration-f ---------------- --------------------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev %change %stddev \ | \ | \ | \ 6438422 ± 2% +27.5% 8206734 ± 2% +10.6% 7118390 +26.2% 8127192 ± 4% vm-scalability.throughput Zi Yan (3): mm/compaction: enable compacting >0 order folios. mm/compaction: add support for >0 order folio memory compaction. mm/compaction: optimize >0 order folio compaction with free page split. mm/compaction.c | 229 +++++++++++++++++++++++++++++++++--------------- mm/internal.h | 4 +- mm/page_alloc.c | 6 ++ 3 files changed, 166 insertions(+), 73 deletions(-) Tested-by: Yu Zhao diff --git a/mm/compaction.c b/mm/compaction.c index 01ec85cfd623f..e60135e2019d6 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1798,7 +1798,7 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) dst = list_entry(cc->freepages.next, struct folio, lru); list_del(&dst->lru); cc->nr_freepages--; - cc->nr_migratepages -= 1 << order; + cc->nr_migratepages--; return dst; } @@ -1814,7 +1814,7 @@ static void compaction_free(struct folio *dst, unsigned long data) list_add(&dst->lru, &cc->freepages); cc->nr_freepages++; - cc->nr_migratepages += 1 << order; + cc->nr_migratepages++; }