From patchwork Wed Mar 20 18:02:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAD53C6FD1F for ; Wed, 20 Mar 2024 18:04:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9BDC6B0083; Wed, 20 Mar 2024 14:04:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E24FD6B0087; Wed, 20 Mar 2024 14:04:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC5996B0088; Wed, 20 Mar 2024 14:04:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B7F3C6B0083 for ; Wed, 20 Mar 2024 14:04:54 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8D3E940555 for ; Wed, 20 Mar 2024 18:04:54 +0000 (UTC) X-FDA: 81918193308.10.256CB08 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf27.hostedemail.com (Postfix) with ESMTP id 4F34F40023 for ; Wed, 20 Mar 2024 18:04:52 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=V0bMBeff; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957892; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=hok1is7g7Zb/3Dq95nz5gC21SpgfU8OMT2/ZL0zycfk=; b=CuITqGpN3EDI26lhKVwOt8NmbCmKqM2vJ5ZvkMb+8A5tchRy1xQ+DmlpW5eA/xzXPhBBzr j+QrMaS72x5cdNoi2Yk7WLmYIWySLU5FqTBtKBPW3ncwDHtv9Zs+a/FAGwLYOQIOwlnFef iluyfm/e+FGOkVT9UehB/3I8DccDnSM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=V0bMBeff; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957892; a=rsa-sha256; cv=none; b=LdGlTFQdgxjvmphMYP0z5MsyOele7Uo5G3nMiqwYG76uG13U+l4Fgg4FHbBpXeTIXg+teg 6adkd5e6jIBVLcr9t4fjXMlQVm+sFDbL9vWsNMyCvRCX6exd+j9WmKp6ApDMDFfEpReNGN 2SiwPvQXrSTPNkgAwDSO5djpHmRfXr8= Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-430acf667afso984561cf.1 for ; Wed, 20 Mar 2024 11:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957891; x=1711562691; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=hok1is7g7Zb/3Dq95nz5gC21SpgfU8OMT2/ZL0zycfk=; b=V0bMBeffIJqcE67lUzHC1gYpP0ne9QvSH4xEoc1h7gr/1/Cyo+LsIkzB6ByZSyW1ji zWvkVIiK1UKcjmyEKVj+2WQGBloeK02ach+Q9COXUz8hiPAz9elojAsP9loNx9dkB0rx gw5U8HES6SZfIxGFNOxJaDsklg+Ue0A9HqUnJx/YELrM9u1OU2m3ebM5IuxJKsbwJ1fY sk8HfvbypfwOJvWxuedTXuu3UEUbvqL5uLrDr/ThfNk6db2ER2+xRJmWIossHGPgdeTr a1mYsASfaV02+s0L/1EKmE1n70xl/fRmZSrv4TO3eVoVeXW7tCW4jU1mb32JAGImFJy+ 0UOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957891; x=1711562691; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hok1is7g7Zb/3Dq95nz5gC21SpgfU8OMT2/ZL0zycfk=; b=Buc1S5SEuk/bjP0bAK8RWRrlM+JKsSFNPihWT424J4ht513UtgBM7Cc72r63CxjzDm cs52LKLkpoXJmKizJJdh/AfHIhhjt+FcxJ59ViAuylALp0Efyy72467Ao55PWQz0neue RLFHxbEcTzg3GWLiJGJw5ysrnlx1/euQ+3fd9y2UQdTNCq88nO605RjhIKoCMCYTbZk3 Gbd1mHfEtWdY02Y6AFFjr12fKImW9FHaoqMQruKEiPyed/1GX6KjjvHcJy8eq/a7Kt0i CL1WfnqF9VFDmsmHpMUJ8Nn/eqdPKi5XQhWcrhceqVz3rffNSF0hx6TPXbLO3vDO7TKY cyKw== X-Forwarded-Encrypted: i=1; AJvYcCU/p3YRwXcfDhHvsJPLzXOGWTXM9A+A3akZCawblmYiEXymGqmLohExQVk3zYd+CVX1MjuxzgRsgChzEojzniKJ+HA= X-Gm-Message-State: AOJu0YywLssjwu21l9k/mQJY9bI7l1d6hqBSoT8/p37u9whsXh48kfpD yMc95c+0wvgPKWL/iWye9WsGtXMBs8urhbILqK+VnMbYQw3JBBN8ZFWgayDP43Y= X-Google-Smtp-Source: AGHT+IFw5Y6Fl0V5Gpoz4wStEblDo0MiKsqpxBMt3xu4UrsKdUC5SoTseBm34qKf4NrikAG0kBhbrA== X-Received: by 2002:ac8:5809:0:b0:430:ddc3:d559 with SMTP id g9-20020ac85809000000b00430ddc3d559mr7217593qtg.12.1710957891149; Wed, 20 Mar 2024 11:04:51 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id cn7-20020a05622a248700b00430b60698e9sm6156005qtb.32.2024.03.20.11.04.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:50 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH V4 00/10] mm: page_alloc: freelist migratetype hygiene Date: Wed, 20 Mar 2024 14:02:05 -0400 Message-ID: <20240320180429.678181-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4F34F40023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ugtqkfy7fdmi64pibz19piw7gentohfh X-HE-Tag: 1710957892-306739 X-HE-Meta: U2FsdGVkX19qXzw+ybnri3W4lFREJhRs7AkEzEmmTh1ZfFRZ+K3HJCFPit8xKcoHb19NJtEQ/WcHh7gBJ0VTtxI2PviskUrkvV4Hh3T2HA+YJy7QWL55ZsP4khYZ6rUFxhHusFXCo2OFlkzX28z4vh2FW4/gTZAk7lNUsrHraeFS+dsWb47mTWjYSdz5DxgKGdPaTcVJqFD4AM0+BqYfDH9uDZPOkSBa8OgvaObHzp7SESnMUDoFS/aus898zXahQYoa2Gy0Wbf6UbSgHU9RftMxzzRskfpUFdFm+eMjeVXLj7kSJnsgQi4QkIFAGJOqscKDanTIrCfJCjT0MVV4xETUv6TRSY0XzDeWAEho+O+yYRmZv3bDfHgcUJqn5fqBbDHyAfCIGlzOK1c55iVJIuX53GNs+RrBxdEjo0tAQZjkLzwbs0wGygEsmwOMAYIXs+Vm3VbxRHsGrkEzAi3Ear9mCWwtXCmkBvys09gP5K9YsR7JCE2Rqbq3igloOJIqCBq8Q53+vixhoWN+TvAtYsBjlptrQjV0EvJhfc6r4GIXLSxiDgGad62kDoTH2qh+bbojMTOxRsNQrAdWnfguFmFTf9sM76hzERDwoyczboC05XVTEqbHXtteo2Z8rDG6/RbH7YARgTHbHsdSOj4TNsDOw4i8IgG2JYFC98TJoyXbkaeJCa6yM5bE4tFU2CmMF27tb+91+A5o0SBVU685pwp4q2yI9TeK7o1+9p9abg4CvrXxTF9RhJ3Wd7ariTHWCuIJ4CbpP7wZ+OdRZaV5u4boMggUG26k4VWg3zzZIEPYEJ3PpY7iK3K5skCJ6kLxgyCQmHHtSRdmSEXx2Cks1M5i5Ncj23hgq6H4klHp9PEVZDS7MjDlbtuEdDwMaS8QQfioS+et0hx3KYyE7IYjxc4DE7uJTGBSmeoFl2JzaWj7lYJkKv8bALG/tnGakL29C4w6RJ+WOWGr7PMvhbd d9Q/m84l odN9StV29YoEdMyOtNLcKhJYOq9zgOjdMEEMnHDOuD+7Q4o0UGD1hZQfkxbpkALd/bwkbMgAbNghuAj/hh9djuWD2rIUXJxqiWeLl0DltKpXJ+M9HYAEkNopMIiUwCiAzbuig4e1ybWu54p6hoEGcs9Jn7n9hAz1xXv6QHdCoLC4SmbdKb0x6/sByj0PxYuqMtOzH4JqyKU+gVzleGyWUjxdlAHiBNT1ToCGLm9iuIuLhQ/OBUr3BkQtMhniddL2m7a75V9n5jG1g+eU+hP62cux9/BVBX/KvG8Z8WOyTRRpz8KWjud6KqWsdThY7YpX5nRr3mvziTPzhprcbHWKH71NhQN4eZmXXiexsQjWb2ewy3HBlC9Jq9gTegW8mpdAlyDnwUK2Ww2tYcPBMcaoHZmuIIIq9jfLBwNVrVPdQ13dfXFO1ATTBqTp7WVPZ7Jqr8rgLvkZVBhrPISeQcfSa/Tsu7zXvFW3DiSzSbFuQ2PYLoc6krcjmJgJRuitTspew1GERsNzfOmy/LgnSBt3ubod0ivVqls0AYMmaExZAJXsR7G4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: V4: - fixed !pcp_order_allowed() case in free_unref_folios() - reworded the patch 0 changelog a bit for the git log - rebased to mm-everything-2024-03-19-23-01 - runtime-tested again with various CONFIG_DEBUG_FOOs enabled Signed-off-by: Vlastimil Babka Acked-by: Johannes Weiner Tested-by: Baolin Wang --- The page allocator's mobility grouping is intended to keep unmovable pages separate from reclaimable/compactable ones to allow on-demand defragmentation for higher-order allocations and huge pages. Currently, there are several places where accidental type mixing occurs: an allocation asks for a page of a certain migratetype and receives another. This ruins pageblocks for compaction, which in turn makes allocating huge pages more expensive and less reliable. The series addresses those causes. The last patch adds type checks on all freelist movements to prevent new violations being introduced. The benefits can be seen in a mixed workload that stresses the machine with a memcache-type workload and a kernel build job while periodically attempting to allocate batches of THP. The following data is aggregated over 50 consecutive defconfig builds: VANILLA PATCHED Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) Huge page success rate is higher, allocation latencies are shorter and more predictable. Stealing (fallback) rate is drastically reduced. Notably, while the vanilla kernel keeps doing fallbacks on an ongoing basis, the patched kernel enters a steady state once the distribution of block types is adequate for the workload. Steals over 50 runs: VANILLA PATCHED 1504.0 227.0 1557.0 6.0 1391.0 13.0 1080.0 26.0 1057.0 40.0 1156.0 6.0 805.0 46.0 736.0 20.0 1747.0 2.0 1699.0 34.0 1269.0 13.0 1858.0 12.0 907.0 4.0 727.0 2.0 563.0 2.0 3094.0 2.0 10211.0 3.0 2621.0 1.0 5508.0 2.0 1060.0 2.0 538.0 3.0 5773.0 2.0 2199.0 0.0 3781.0 2.0 1387.0 1.0 4977.0 0.0 2865.0 1.0 1814.0 1.0 3739.0 1.0 6857.0 0.0 382.0 0.0 407.0 1.0 3784.0 0.0 297.0 0.0 298.0 0.0 6636.0 0.0 4188.0 0.0 242.0 0.0 9960.0 0.0 5816.0 0.0 354.0 0.0 287.0 0.0 261.0 0.0 140.0 1.0 2065.0 0.0 312.0 0.0 331.0 0.0 164.0 0.0 465.0 1.0 219.0 0.0 Type mismatches are down too. Those count every time an allocation request asks for one migratetype and gets another. This can still occur minimally in the patched kernel due to non-stealing fallbacks, but it's quite rare and follows the pattern of overall fallbacks - once the block type distribution settles, mismatches cease as well: VANILLA: PATCHED: 182602.0 268.0 135794.0 20.0 88619.0 19.0 95973.0 0.0 129590.0 0.0 129298.0 0.0 147134.0 0.0 230854.0 0.0 239709.0 0.0 137670.0 0.0 132430.0 0.0 65712.0 0.0 57901.0 0.0 67506.0 0.0 63565.0 4.0 34806.0 0.0 42962.0 0.0 32406.0 0.0 38668.0 0.0 61356.0 0.0 57800.0 0.0 41435.0 0.0 83456.0 0.0 65048.0 0.0 28955.0 0.0 47597.0 0.0 75117.0 0.0 55564.0 0.0 38280.0 0.0 52404.0 0.0 26264.0 0.0 37538.0 0.0 19671.0 0.0 30936.0 0.0 26933.0 0.0 16962.0 0.0 44554.0 0.0 46352.0 0.0 24995.0 0.0 35152.0 0.0 12823.0 0.0 21583.0 0.0 18129.0 0.0 31693.0 0.0 28745.0 0.0 33308.0 0.0 31114.0 0.0 35034.0 0.0 12111.0 0.0 24885.0 0.0 Compaction work is markedly reduced despite much better THP rates. In the vanilla kernel, reclaim seems to have been driven primarily by watermark boosting that happens as a result of fallbacks. With those all but eliminated, watermarks average lower and kswapd does less work. The uptick in direct reclaim is because THP requests have to fend for themselves more often - which is intended policy right now. Aggregate reclaim activity is lowered significantly, though. --- V3: - fixed freelist type violations from non-atomic page isolation updates (Zi Yan) - fixed incorrect migratetype update ordering during merge (Vlastimil Babka) - reject moving a zone-straddling block altogether (Vlastimil Babka) - fixed freelist type violations from lockless migratetype lookups in cornercase freeing paths (Vlastimil Babka) - fixed erroneous WARN in the bulk freeing path that was intended to catch mistakes in the now-removed pcpcache (Mike Kravetz) - fixed typo in patch 1's changelog (Zi Yan) - optimized migratetype lookup in free_unref_page_list() (Vlastimil Babka) - batched vmstat updates in page merging hot path (Vlastimil Babka) - rebased to mm-everything-2024-03-05-20-43 (v6.8-rc5+) V2: - dropped the get_pfnblock_migratetype() optimization patchlet since somebody else beat me to it (thanks Zi) - broke out pcp bypass fix since somebody else reported the bug: https://lore.kernel.org/linux-mm/20230911181108.GA104295@cmpxchg.org/ - fixed the CONFIG_UNACCEPTED_MEMORY build (lkp) - rebased to v6.6-rc1 include/linux/mm.h | 18 +- include/linux/page-isolation.h | 5 +- include/linux/vmstat.h | 8 - mm/debug_page_alloc.c | 12 +- mm/internal.h | 9 - mm/page_alloc.c | 650 +++++++++++++++++++++------------------ mm/page_isolation.c | 122 +++----- 7 files changed, 415 insertions(+), 409 deletions(-) Based on mm-everything-2024-03-19-23-01.