From patchwork Wed Mar 20 18:02:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D183C6FD1F for ; Wed, 20 Mar 2024 18:04:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2908E6B0088; Wed, 20 Mar 2024 14:04:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F1DD6B0089; Wed, 20 Mar 2024 14:04:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F36EB6B008A; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E0B396B0088 for ; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9F819C039F for ; Wed, 20 Mar 2024 18:04:56 +0000 (UTC) X-FDA: 81918193392.07.F721262 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf24.hostedemail.com (Postfix) with ESMTP id D259E180008 for ; Wed, 20 Mar 2024 18:04:54 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=2lYvnaC0; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957894; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HCOZaDSQS3uSogr2y2Y4MtomH1kr/Fi4b+AnkoSHYG0=; b=HapHiLN9DEcIYDYRuUXfTkxaEW2AmRK8PtLAHTYh0SEJcDwSgTmJmIFJmrdh0pH5JmcKR3 BaK4F/gD8GOyzFXoKCw9fwIICksjm/sKZji9tPNFpVRK+FfArAZpOF5Mfb0EcJlGwc1mpP wNW2ar5bzCpZ5zlaawiB6uC6tCc8f3w= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=2lYvnaC0; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957894; a=rsa-sha256; cv=none; b=WfCuyEObqDvfjsoxG/g7tJUsCoA6quLnESSkAULUEAJMu+VSQPtzIJ5bPreNnA0uBCYA6c XV/UQAqjfp1jH2p1yqJ23Ni4lBRdYVVCjWpUbmyoWTxh1Ji+7e5A7pOGuXyL8ik+5IndcF 0XnP/O7Wq/0eK1AFOImctWFnbBrRei0= Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-430dcd64e58so1105761cf.0 for ; Wed, 20 Mar 2024 11:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957894; x=1711562694; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HCOZaDSQS3uSogr2y2Y4MtomH1kr/Fi4b+AnkoSHYG0=; b=2lYvnaC0fqwyb2XLtX6o+qf3Vtuuec/llyGMsb930l8vfHc/1PbAZRz7LntbWv4WN6 x0xxaSrK2P21w/x37ky/2r8/tcX6ktTQl/JjxfrzkPhaOgoyNJ2XeltU4UjQ3Um6kHNx IWxpTIEuz49OVB7uf/SS4QWveDQRksUg459OSER/Mbw2CAt5gkZhkph9H+gb7HPF2BBp pirGoIIbWK+y8hFc44j4B1CGgkCW9FcwE7C+TaxL0P/VYL3Czacs8fVB0xPnJb2AKDto Hwq2B+ICYY6M0gUoT9NLItKem49QH6ClY0TzVd7M1q0t3oX3YkxE1GV/6Nc6aiWZ+Rbg JICA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957894; x=1711562694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HCOZaDSQS3uSogr2y2Y4MtomH1kr/Fi4b+AnkoSHYG0=; b=HsYT7dvg9+FwAtddpAW7PE5OnpV/Y5M97Hj2UlRgsVljm3/ESC8uKJSqRx11Q/x0+C PLMtzxl/Y5K3kO23Z1r/o/sUxrobNvtSeEWrqGRii1sJv/RnKyNx5bINgxkGaNewI3ZA HCGaea1i/QpHHNimpA0dHY99B6AZNitd2eL78e/U6KJbmD3t4095SLKpz+puoFTZLjrF V/vymWY7j1FU5reSNidjybi+FJ5gt3ZP2NIzLOmwJCWUtGe+YEFMlzSEwqkeJShSl94L cDycvg46L8ZCUND33BV0HEqK0h3iuNv/aFKQ+cREB+yN8Fe3XPT509f2D2vR1Ufd7Yv4 LhYw== X-Forwarded-Encrypted: i=1; AJvYcCVjLkWkOHf6zC+aBDMGfdbSa2HX/4R1WNMtrJSUv3ptgIwAgUf5jCG9KvJhipmhgaHfoKLuFDzLsyW4pV+d9aR8ynk= X-Gm-Message-State: AOJu0YwQ2AHVurf2m07ZhOiCXuq7DAgr9/CuiSzQpNSUTucW7H8lIB3A CZVfXtMiXCOnbVRhLFpBZp+WMtbfvYMUukKgZSzSWSF7eNRg+jgaE++oN9i+arE= X-Google-Smtp-Source: AGHT+IF5UfXc5lTV+1+ZbEqDKnB2S/2dMfIITfLAmk8SgkXLoh8oySBupjmVBPzQZ9ttFuOzueZSzQ== X-Received: by 2002:a05:622a:54:b0:42f:20f2:c4cf with SMTP id y20-20020a05622a005400b0042f20f2c4cfmr2979645qtw.31.1710957894003; Wed, 20 Mar 2024 11:04:54 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id y17-20020a05622a005100b00430d8e11bebsm3231779qtw.64.2024.03.20.11.04.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:53 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/10] mm: page_alloc: optimize free_unref_folios() Date: Wed, 20 Mar 2024 14:02:07 -0400 Message-ID: <20240320180429.678181-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D259E180008 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: w19k1b3ck9zr1qhn3srqf8jphkc4q63z X-HE-Tag: 1710957894-478711 X-HE-Meta: U2FsdGVkX18RIUmrQJGsUaXwQr523hu6LHQqbzabhcsmTPhCPxM0/ZDHylEy8L6djfRXWw36gA6rlAQIeE+sIXN4FnXkMvAzknBpxqle0cgr8LQhgRbtWntu1lkSmsa2ptQ45GHMQtCn7lNpG7eZ+achtyn3BkqXB0xQKxiFeDYFJIM3lfbE4JOSP+7Xe26jjgM8lc0YUbRHv0J86lAZF6oh5uCQpcwlVHiBuIqNBV1AWW1pgTOy/CqR1yQ/Nj9X+B6iHnKyni0XBJJlGkkBXlo3F4PNkkSHLdVcGy87tJYnEBi1oYTeJl3iSmIFNMn5NL+V6IEpHgGkc3Tnkad9e1Yy9snr2sYjCCQshOGhhngdP5WlZoR9xDqskHjNRdYNZ9jPvhFHBbRoiD88bs8dXmjz35X+OmVybRWdn2zRuE0Y0rZlOSGtrRrmTCIW7nwEjT8r6nIPI7W2MryBkB6uWzGYTt/fZ8oWpI/khr/+W3aU/wdKQxBGjIhRdS0PqB8msJS2AujnKC28pPs2oCcv3XKNgSk91mdcz2/JOOzB7JuwnRP28A2FG6AsNhwQ/kNwxBs+WcJzE8PxNmESUuBN8cXn7Mwei6LEuopbjGJOf+u6ikAnXYFuc5iwVcnNGDBRYDTiEYmMlytNBMLND9Spy2GfFr/ebXhWSs8qI3+WDH25P7pCk1YiZ8iYgGcbxpXhsktN021+8Kr/Sl4lD7/sR1qzbWjN5JqHQISRfL54nZ5622vtlcTIPgrtQSCXBA8Rv8r2IdpNvX9/Fgd3zz67To27RplTk5k0lKoocuppI0mK+e/QENSckdIS5Fifv8StS8Dq7OsrlrG1Y8G5BeGjZ2VqEBboMlLApw0edwuKzmEposzQbMk7cLdf5mIBy+mGIpOj5uvP78a9l/lea8yO8MRj+PuXIBI85O77tI3CnYU0GjAn7w89zMep4ZvRz57VEz6pXwjWozPuhVP8web aFC4vrQv thXvQSFgIInHixJ/ZqoKzOzHghKLT8JJbY6xwB4socI7dTFqRPC0qirvbgTFTg11rGfRTK4S1A4NKcUvLCd8ZklddrnALC8JMVHNZIZ9EWOPETbiihhm0OPj3fm2qtWfQ3BD/WsM3PgyVfpdRZjAUEX72NciCCCD3GsecSCtni6HAwL2d+q78wfjYFxLPbZ/g8qCZlgxkoftDxPwZoDezwvHkyG9C8I+6DW2Ru55IsZa8Y3TQujkSU0b0PB8ijPILrz5ptVZy+5tFLBc0Xl4+4yN3p5L3w8zi1YqQi3KdB65b9sQs7wGfhXzjkKpnyURWcTcSzEh++zjNYwh09NHjWHt62ZlDGxtg4Ob7sMlnlwj1SRQNoHmICkpFTeeh/o0M6ljDmg8+v1t91571w86bS73rHfdGMWG/WxI06wUc24YhYpnLCFh5eMEXkN6gIcoOQW+L8rsJv0Wjy9t4dWtm0exvk1Wcff2pyaWhUwNqartY3R4i20ALrBJDh0tK8K/UgRUIAxbgZW9BUh4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move direct freeing of isolated pages to the lock-breaking block in the second loop. This saves an unnecessary migratetype reassessment. Minor comment and local variable scoping cleanups. Suggested-by: Vlastimil Babka Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 60a632b7c9f6..994e4f790e92 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2524,7 +2524,7 @@ void free_unref_folios(struct folio_batch *folios) unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int i, j, migratetype; + int i, j; /* Prepare folios for freeing */ for (i = 0, j = 0; i < folios->nr; i++) { @@ -2536,14 +2536,15 @@ void free_unref_folios(struct folio_batch *folios) folio_undo_large_rmappable(folio); if (!free_pages_prepare(&folio->page, order)) continue; - /* - * Free isolated folios and orders not handled on the PCP - * directly to the allocator, see comment in free_unref_page. + * Free orders not handled on the PCP directly to the + * allocator. */ - migratetype = get_pfnblock_migratetype(&folio->page, pfn); - if (!pcp_allowed_order(order) || - is_migrate_isolate(migratetype)) { + if (!pcp_allowed_order(order)) { + int migratetype; + + migratetype = get_pfnblock_migratetype(&folio->page, + pfn); free_one_page(folio_zone(folio), &folio->page, pfn, order, migratetype, FPI_NONE); continue; @@ -2560,15 +2561,29 @@ void free_unref_folios(struct folio_batch *folios) struct zone *zone = folio_zone(folio); unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; + int migratetype; folio->private = NULL; migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ - if (zone != locked_zone) { + if (zone != locked_zone || + is_migrate_isolate(migratetype)) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); + locked_zone = NULL; + pcp = NULL; + } + + /* + * Free isolated pages directly to the + * allocator, see comment in free_unref_page. + */ + if (is_migrate_isolate(migratetype)) { + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); + continue; } /* @@ -2581,7 +2596,6 @@ void free_unref_folios(struct folio_batch *folios) pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, pfn, order, migratetype, FPI_NONE); - locked_zone = NULL; continue; } locked_zone = zone;