From patchwork Wed Mar 6 04:08:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13583359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D9A9C54E55 for ; Wed, 6 Mar 2024 04:16:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74B8C6B0080; Tue, 5 Mar 2024 23:16:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB266B0081; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C2DB6B0082; Tue, 5 Mar 2024 23:16:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4FC8A6B0080 for ; Tue, 5 Mar 2024 23:16:24 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EE337160368 for ; Wed, 6 Mar 2024 04:16:23 +0000 (UTC) X-FDA: 81865302246.07.076D8B4 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf05.hostedemail.com (Postfix) with ESMTP id 52668100015 for ; Wed, 6 Mar 2024 04:16:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=VxBW8bOt; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.169 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709698582; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xpqItcyH2EDAXtPSFuVCpgEvxVG9s6bJRth9b0kyPMA=; b=H/qXeLBFUsamkEbXIMOtMhrSEtghCZOiiftwI1E4agZf8RYA0EP/MZd7TTTMLWhyqWKCuy rglpQmGvqAT7H8+L+7xSwQUBakd3bodYngkmhT9TQnEW/zN2DTluSI5UFXY+J+pvJidE2+ ddZcid7qLsWzOZDvBNxrX/Thkm0Ez3c= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=VxBW8bOt; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.169 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709698582; a=rsa-sha256; cv=none; b=jBYluGks+GBMSKlLvI26j3UbEmOSeCTmAJJ/XJ14fhSgvniPzNJ5b9o3Q+TmQb/FziVDS7 r3eFLmo1xPBkNN2OwH9krRU6pMATHrohJ2k9BNqX6rcQEyhOAf3gSabBpallgHCoWJFuc6 be6foVEgF0tYgugOKNnXW2QxXEaOCuY= Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-42f0d09f567so1372491cf.1 for ; Tue, 05 Mar 2024 20:16:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1709698581; x=1710303381; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xpqItcyH2EDAXtPSFuVCpgEvxVG9s6bJRth9b0kyPMA=; b=VxBW8bOtgcF6pXXw21YnmKsp/O1njeQfURVaXXNttLt843f0sKF0zoHLrsptz8FmhM /5mGYXV2wi1U1e9PPYincARtZLs1yb6PWEYoEs71s6c6vWnzyYjpWYVm2AgRmqI5h+vb Gn44GoRsB91VewMqJ95ake1RQwcFnV6gUqL0dk6w0Sr2eHqUTv/wC5oANZPT8NdKqsOY sIjf84z4cwCtKrqXvg2DD3FVgMW9oLxUsXStmiqIN9QzmLzG36hKtju6IhUo1SjqtSwm LDqlCdmBnG6L0tWOza4lnXMgg3XnJXZV9qcqenTnqKTRMZ3KYHnrBWnYztt469BQslfA tDIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709698581; x=1710303381; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xpqItcyH2EDAXtPSFuVCpgEvxVG9s6bJRth9b0kyPMA=; b=Q/TU2Juj4IaRFLXoZFcVSi5RUPIHNcQCBTZO+bOITPZV6Ojkic8LwnncNcIFExnc9E jh2ON0eYFxnpY5r/Skl7VB3S9bjikgwqy9QLA3FZE+Ww/zS2TlHe78uHcvGnnnI7oiVL N1prGECWzwQ3McrLYfj8Xc1xaHrd5Gaj+SzYSie7kyeqwuYawi/CmQsAWf9+g6oY0bC3 hDlRaCWoJJ24joa7fxC0wEj7ZHe6ZMVzxx3axf7Tj0U8ibqLdg0ayJFPWH0n/HOLK1t8 wFasdlsNAwwWGgyYrdEXIrr3rQpFDjJ0toLqA+HSQ4SvZyI1OW3f7oERgOqji2t8ZjsJ 4kgw== X-Forwarded-Encrypted: i=1; AJvYcCX5mF+YgVMnzP/54OJV2bYX6NjsA/FnsYcedRsAWoFUouXcZyItKpKs4QUq8EQqIDbEC/DJrSovoHnNBhKLtjO3g0Q= X-Gm-Message-State: AOJu0Yw04ehrhwwanc/D7miNt2dSjIL2C5oD4WFSMWRGChenjfokHB2o mu2GdbTERXMG7t+sZvp2HraeaUJqPn6aXBtFJQQ8/P6nST3n6g2I3eptOoCQBcY= X-Google-Smtp-Source: AGHT+IFlnJve3Rb16fSRkg2cnwF59nsIv8qKr5z8db/VPIGVmj2pSzSHnn6osAw7nOgwMuODvBzfPg== X-Received: by 2002:a05:622a:110b:b0:42e:f56b:91d1 with SMTP id e11-20020a05622a110b00b0042ef56b91d1mr4214660qty.32.1709698581469; Tue, 05 Mar 2024 20:16:21 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id k6-20020ac80746000000b0042ee243ca95sm3509250qth.50.2024.03.05.20.16.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 20:16:20 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , Mike Kravetz , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/10] mm: page_alloc: optimize free_unref_page_list() Date: Tue, 5 Mar 2024 23:08:33 -0500 Message-ID: <20240306041526.892167-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240306041526.892167-1-hannes@cmpxchg.org> References: <20240306041526.892167-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 52668100015 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: c1sist7skg5ubrmcy7gu9ha7oz9txi6q X-HE-Tag: 1709698582-590586 X-HE-Meta: U2FsdGVkX1/y83N09Y4ot6JRLM55s1FmLal00FrRrXw7Vr1ZiYB4fXTGFCC10tgbakxt7V1VLReh8iNAIF5tH6/aB3uA7IzXkFZDvSWoCF2+03Hm29PTmDqa0A8zfi5BZUcRa0PiAKAqwLLlA8viKE/TbfiJW6j9z2hW1EK95crID/dzCFNTgeTB+E1MHiEdJB0BqY6ZvvRemeC3e7GHhfZmARnwgFTOrzSG6x4pbb6aI+5vjd0s7ftJ4wtIe4H3P3BX8HWUhrdYZFGjmfyZ5oWaKPUykSSk0rDnWrSjbJjXRf+bTz/CQGfoISBuvoR0Gn6WEHWn+K29NNrV8YO/WgyDsqZEh1+vrE02/hfxO6WVYYybgJB5ARCt5XaQ9BL9Qbb/NpbR4oMjD4nz+adAnSMr6gPXdsn0LmNrJehe/5ldJdf4aYY2F9NXiWm00fOtz1kgIB3E3dMSkWKZq3ct2YqIdzcjDHmTuHu/OKtjYn/cA9vs66tH+oA5VlIQ2DT7amFzznberpVlYsA0cGFRewEBGx52UbrDy2oMZ6K9i9XKa5LpxNPU7YkwP4FBr8qP/VIdy2WIlha9p8SRq2Gf36iPfHFf63hCFXdn3GuYqFv83LQdxt6SkLA7mtFnnvR0tEhcT0Rg5wEBr2nlqSuzJ2YwxvzROL3CdfaqydqzrMy/M9FxtU4vkviB26gxb0S5KpcwjkS124mMIx7X64ZbWieC91gu8K/5Bn5t6FuC3HVJPfOvcSRYrRxBPAeOI7V+nyehaef6IuYd6qpAzD6EBneOoW79yGu8ksvxkLZNcON73T+fbD7G5iMVmQ6krjZxsDVoahJLTqG82WsDsBafTlTrir/k0vBzFd3lV5t6Wgtv4wLtP4310cCg7PJGmNy/z6QDOpZW5P1dFYweGszVw9Kb+He0EfzUKyVbU4RFSpZRdy7g7gO+ynT3b3doyNf+mEre/d6U5jE3IxXoZ04 z5ZNO8Em Tj8THEAl0WDNEeXizryNtLD9d+WOw+UMDBalJ4WAbzTeXEHT8KKYrF/8+zkuQXVFzYz7AHUKcM7rBiH+NEdqrra/QaBn/G2JJO13119OWwLir3PfakJc0bhRZTpBjyoKND335M6nKq38qvVYdHy2seAmbrUmuJ+Qk5nARfEXE2k+CHaSlbOOUIhjdPyBFlAwc+m7CdowJw2REA+/eSzQJjBgSOld437aD6WOe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move direct freeing of isolated pages to the lock-breaking block in the second loop. This saves an unnecessary migratetype reassessment. Minor comment and local variable scoping cleanups. Suggested-by: Vlastimil Babka Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9665a2db3b53..88f4e08726ee 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2521,30 +2521,17 @@ void free_unref_folios(struct folio_batch *folios) unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int i, j, migratetype; + int i, j; /* Prepare folios for freeing */ for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; - unsigned long pfn = folio_pfn(folio); unsigned int order = folio_order(folio); if (order > 0 && folio_test_large_rmappable(folio)) folio_undo_large_rmappable(folio); if (!free_pages_prepare(&folio->page, order)) continue; - - /* - * Free isolated folios and orders not handled on the PCP - * directly to the allocator, see comment in free_unref_page. - */ - migratetype = get_pfnblock_migratetype(&folio->page, pfn); - if (!pcp_allowed_order(order) || - is_migrate_isolate(migratetype)) { - free_one_page(folio_zone(folio), &folio->page, pfn, - order, migratetype, FPI_NONE); - continue; - } folio->private = (void *)(unsigned long)order; if (j != i) folios->folios[j] = folio; @@ -2557,15 +2544,29 @@ void free_unref_folios(struct folio_batch *folios) struct zone *zone = folio_zone(folio); unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; + int migratetype; folio->private = NULL; migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ - if (zone != locked_zone) { + if (zone != locked_zone || + is_migrate_isolate(migratetype)) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); + locked_zone = NULL; + pcp = NULL; + } + + /* + * Free isolated pages directly to the + * allocator, see comment in free_unref_page. + */ + if (is_migrate_isolate(migratetype)) { + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); + continue; } /* @@ -2578,7 +2579,6 @@ void free_unref_folios(struct folio_batch *folios) pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, pfn, order, migratetype, FPI_NONE); - locked_zone = NULL; continue; } locked_zone = zone;