From patchwork Mon Aug 21 18:33:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13359740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A169EE49A8 for ; Mon, 21 Aug 2023 18:37:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BDA0900007; Mon, 21 Aug 2023 14:37:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 46BBB900005; Mon, 21 Aug 2023 14:37:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BABA900007; Mon, 21 Aug 2023 14:37:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1839A900005 for ; Mon, 21 Aug 2023 14:37:50 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E819F1C9AC1 for ; Mon, 21 Aug 2023 18:37:49 +0000 (UTC) X-FDA: 81148970658.04.2C17FC6 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf04.hostedemail.com (Postfix) with ESMTP id 250CB4001E for ; Mon, 21 Aug 2023 18:37:47 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=uu324QBc; spf=pass (imf04.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692643068; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lLUpAEhwdHses5nFOU0azVoQ7mhHkrTPDw+MACyT3Uo=; b=0AHGxHTwL2qcubyekOm0oIusCiTcMuo8EhJqpVy3KfXNUk3TKtQy/q+lshkptEb7RNYVH9 iRe2bAP66JrlgBJntZQ0w60/2xCyqa4Zw9STlEJ1OVuBd4tp8w1FnXwHl81RB6s+OPrffX I/rNHVfq10HLH39e29UXst/W8usDhPI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692643068; a=rsa-sha256; cv=none; b=f59UXo/GS0PO9rskA/N+8ixpRkdq4vmB/hlhwDUqP5roRFOIj5QFi5G0RgDCU+kaX+Oypx 1DMc/tH+I+53VbMj7Y1wS31i2GFOso+OzCl3qCc8fNIUBCJ5daRrqRlXubg85TplHnOKrG Fh8qh9SWMFjFijK41ktNYwwpQS+YFMM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=uu324QBc; spf=pass (imf04.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-649c6ea6e72so23018446d6.2 for ; Mon, 21 Aug 2023 11:37:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1692643067; x=1693247867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lLUpAEhwdHses5nFOU0azVoQ7mhHkrTPDw+MACyT3Uo=; b=uu324QBc+rtb2rb6ojdEsAGPTmfCn1pVoKqBeTm/7P0dPa53c/018QiTZAxvjfk4tR gd3isXbWBnvmXwKnm/HQBQdv4CAkaT2elzDp9mjtKLdLitCtyeNEdOyJDD8OSdoNt9gt mDcPb4pJbQH+QjEmQqQQM2JBTmXebCbCMtWni1qZHwHNDASVRcK5jqvoscTv3vRHHDcn EEIaW0f4LPbjd1nU7A5uR+22sHHRvxqew4mEd3GgeorJsCcDp+bzTfVNcKiiZjTxI3MX GCyYm3Z2VbUNvA1yCSYnin8EXYwtYy2a0maw6rJNQldN7CHPfZ3e+BwVlaLApl+QVHTl V/3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692643067; x=1693247867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lLUpAEhwdHses5nFOU0azVoQ7mhHkrTPDw+MACyT3Uo=; b=fZxi+5ZXOYRBZNoWPiW4dlBuZO+c2tj7T1hTbrKrHB0bitnj2PjrDk4lrhx93urBKH 5EaldDdFddFF9wF6igeNhb61f5DUFmRblqw1jiU5O3/+hnAvuPl+Wb6qb711/ulQY1f/ OQJIOY487kv0hPAC3mSJwFSSX9NWhDw18rt+XiRapg89ipy/fF/7RpOskZPxMVcCiW5w Dgx4D/PcglDcBXaMAYvxfVZWnEzMgpQn7aUI5EMQ0ntWXTqP03pjyu9o5NTfefkSGlvC NZStHU7EPJAzKBFsN0ZFVx9t7JKLgkqhWmrqFCMkiB0zFauuLaAVBiSJxl9Y0Xp2DvYS jP1g== X-Gm-Message-State: AOJu0Yw6oHmwwqdJ9ohg3OGaTKZ61bIBOfRP05HJvu96hFK+jbv1Ksy0 wy/7/7GFsY/+KRrZ2xdtw4WddA== X-Google-Smtp-Source: AGHT+IEJOOhim3Cp793d/K2ahJpfEq1PXlo2PZVHOqjBu5gwxhsUCzov2/4CrV0FJmCuIF2w2QYyuA== X-Received: by 2002:a05:6214:192c:b0:647:39f1:5237 with SMTP id es12-20020a056214192c00b0064739f15237mr7472065qvb.47.1692643067350; Mon, 21 Aug 2023 11:37:47 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-699c-6fe1-d2a8-6a30.res6.spectrum.com. [2603:7000:c01:2716:699c:6fe1:d2a8:6a30]) by smtp.gmail.com with ESMTPSA id d1-20020a0cf0c1000000b0064f46c719fasm547661qvl.31.2023.08.21.11.37.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 11:37:47 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/8] mm: page_alloc: fix up block types when merging compatible blocks Date: Mon, 21 Aug 2023 14:33:36 -0400 Message-ID: <20230821183733.106619-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230821183733.106619-1-hannes@cmpxchg.org> References: <20230821183733.106619-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Stat-Signature: hqgtduduayyjzf9o5euoestdm5mpyaoh X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 250CB4001E X-Rspam-User: X-HE-Tag: 1692643067-721428 X-HE-Meta: U2FsdGVkX1/P6sT9+MAb0yGerpaAJWOvkchC/dZr4EEbtZkqgAnsTwcXO0q+gsWCTAtHGTLrghDbxkP519VAVzlEv7bCoMA0sUNzNthUZEEixOAe7BYCz8X9ABVDTHBXbHsPKsYoBcGVq9x+YHEH0uBlJQUX5SQLTaQHyfa74OVBcvU0wVKzqGt97YSMnd8XzXVzezBl6S7bmy6TN+xUTp1GhVYhW2hkUqYF+aW10B67UrYXzl7Fpa44OsOClM5YRdfDu6EHZIQeMiE9YO21OPBxPiHLPZ5rd5uOjvVF7pvv8IRpnQQOQ/RO8UZNSFUjq9WZueyGlXIEozHEK25xl2I6SEQbtubLeudnV0LC32s8PUosg4nIzxvPeui4YuuJICV+zEQ5OI+gBb7Gfj2PSM/lATfClmcUPH9OhIfekCPaWZF/XbT8XW/GppCROF+JPxZ9J5DGMxFtQWbQ97OGPkX89ZC9tzST1K4Q2MRRsPQkFFs7HaWQWEPjKPz50iODi80QdAco/oGBnOD2rqy/wt11FBFjcfBO+udFikDr+SzUcyXEelyvs1HVBebC25JcCp1hxfJHM4YDUJOKlsK/lCrSVPNCkKXLre3MAi+kZfyMTw7ZxhoZ5aSDZWrYWbf4PZ3Nrs9vewkDjx+kDo47kuWUMyIjKZkl/vprJA+cVvByTlu5mtTebm5dHZ+vCzxOZadfEFPo8imdPqzymMCr8DD0bhaikhuqqMW9VFU4SHNdDrNQSRZgmuGwRL7QYDkkYP6VKYdV5kQnzxN26F/eQRJIU+7PloppvODSR66kAeAXh7B7D3R7trUZ3xTymDKHXeCLUk3+f/rSdxWFeMmxVwEyV+Cy+UfuA4yIAetmbxQuePZ+S3PKoUYIoZh3Im0Vby6Eounue6pDbiegT2iA8WZp1dbyaI6Yl7neXPJNh2KdIx52yqJgrPh0BFyVtTDgFlsaYRFrACWCuXJ8K8t hWxOcwbc GI9dM30KemqKqZQPFOPchdiNGuJrnQ4WSQH0LYWNHB5/aDDDu5zZroiadjqcf+K3LGIA/+nL1eAkt/T4at5Pp6i2dye4MEQThU3mcNyeXM55vAWkkV+y7eVR6GvK1fgoh88DgxPh7p++wkSNO4tibjhuradv9KQKDSawgr/Nm7NPD32L24ZV8WZqeXmh1fWQ9dduAHxPqaLJtUeR4iMCsbjmfGndzYbJZfP4ai8KhXE49MzvxtEJ/PDkmBNUnVNWHeCzYkUNl+iTE3Hswnoin7YQQeX/GGq3QNooHg+a5odOQvJ6szsEeQ8aGcawEyHsIO3V3Ve4RyoVkHJj6WUibOk0jSpsEtRkF9q6rCfrkYZcyNRqRzVWlVVlOucE05pC4Li059mXSRS31+FHegYMGul7+OfPMsXIuNhbYk1fZ1BN/3hmFq+IHxIzpGPyB1HhSKTMoUT0Et//TjE9y/YFl1q47AVSKVw+acZiizD5iWsqG7MI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The buddy allocator coalesces compatible blocks during freeing, but it doesn't update the types of the subblocks to match. When an allocation later breaks the chunk down again, its pieces will be put on freelists of the wrong type. This encourages incompatible page mixing (ask for one type, get another), and thus long-term fragmentation. Update the subblocks when merging a larger chunk, such that a later expand() will maintain freelist type hygiene. Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a5e36d186893..6c9f565b2613 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -438,6 +438,17 @@ void set_pageblock_migratetype(struct page *page, int migratetype) page_to_pfn(page), MIGRATETYPE_MASK); } +static void change_pageblock_range(struct page *pageblock_page, + int start_order, int migratetype) +{ + int nr_pageblocks = 1 << (start_order - pageblock_order); + + while (nr_pageblocks--) { + set_pageblock_migratetype(pageblock_page, migratetype); + pageblock_page += pageblock_nr_pages; + } +} + #ifdef CONFIG_DEBUG_VM static int page_outside_zone_boundaries(struct zone *zone, struct page *page) { @@ -808,10 +819,17 @@ static inline void __free_one_page(struct page *page, */ int buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); - if (migratetype != buddy_mt - && (!migratetype_is_mergeable(migratetype) || - !migratetype_is_mergeable(buddy_mt))) - goto done_merging; + if (migratetype != buddy_mt) { + if (!migratetype_is_mergeable(migratetype) || + !migratetype_is_mergeable(buddy_mt)) + goto done_merging; + /* + * Match buddy type. This ensures that + * an expand() down the line puts the + * sub-blocks on the right freelists. + */ + set_pageblock_migratetype(buddy, migratetype); + } } /* @@ -1687,17 +1705,6 @@ int move_freepages_block(struct zone *zone, struct page *page, num_movable); } -static void change_pageblock_range(struct page *pageblock_page, - int start_order, int migratetype) -{ - int nr_pageblocks = 1 << (start_order - pageblock_order); - - while (nr_pageblocks--) { - set_pageblock_migratetype(pageblock_page, migratetype); - pageblock_page += pageblock_nr_pages; - } -} - /* * When we are falling back to another migratetype during allocation, try to * steal extra free pages from the same pageblocks to satisfy further