From patchwork Tue Feb 27 17:42:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E671EC54798 for ; Tue, 27 Feb 2024 17:43:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B30E96B00F9; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CD90940017; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44DD094000E; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 05EE46B00F8 for ; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F3CE114091E for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.04.34093BC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 4DA15A001B for ; Tue, 27 Feb 2024 17:43:01 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Y37CTPdS; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dz7HcV0fyY0T2HVHE6hhj30IoC72xWK4he3mTjgcdts=; b=R5ljEdaWRzx8zJHvZwCWGJ95dConq+Yc5a5Px86hvh4H8uf9i7gYjlOcrJfsqY/5Nic9CN jDBTrc/dB9yrMLY5DGO9/QSYrMptejhvL3O1MX45rd8m1EmZjiYPTNeKBwvvjDwlTSpybg tIhNpWo7I7SY49uzjSfdy2G01m6xx5M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=Da9ilAux46Sr71SfWDlgkTIt4CiE9lZ7AaetWKw3XFjRwqEWZ7wXSfHCBpYvthvw675edk e6PPCT5CX85d4rxb0clrweCwGKTgASYxih0mKVAQTD2j9hj6wUnITsQW4UBl/oeJDSdwAr Jm+kwbBFFG8epzCoxxmfVUsCFa7tfDM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Y37CTPdS; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Dz7HcV0fyY0T2HVHE6hhj30IoC72xWK4he3mTjgcdts=; b=Y37CTPdSNLy571+DOBHZ4uFXxR Lvl1PRGXZRo7/LwpqGkbdr8AuS8zWI8rFYc9gbAiYjkd1urHVCsoNtxpb6AvvKK6Cs4j2U2WFvflE Lnk8y0dTR4t8ZNIxR4LjyWUGkg43i/RMfAVuAm75q5NNaaRuWoRGe0v5+urYDmBHqw4Lz5NOud6Je MxicavMvQq9cxKoi1CHltXMQcHY2EaJ411hD+TJf6FLkfQjky7GyyGChRZvi6qHyV1J7Kx5UXsyRP vjuw3Zlw+DsytCIBCe1dRpbLWnl846sqqyyLLQqJevQSyoQeOKZUyLPhBWDhoT+w7wxhpANIkuFB7 SUUxngww==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tl-00000002yqs-3TgF; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 01/18] mm: Make folios_put() the basis of release_pages() Date: Tue, 27 Feb 2024 17:42:35 +0000 Message-ID: <20240227174254.710559-2-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: jd54cghxidsecwpj5sjbi6axif1f3yu8 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4DA15A001B X-Rspam-User: X-HE-Tag: 1709055781-704555 X-HE-Meta: U2FsdGVkX1+PXZD76qvwiSMxOifzMe6kDmowU/dpTnaUn58kz7ZynuKwPk7H/pJDu5OPNVz2L+G9yhLZAt8Qki0GHqdK692hVbUZYgEn9PrtBkS/f9796iz0XdKxNHRUBqmmC+QyLI8TvF1xAIHWh86bjzpYmkI3D48vteloLqGyQX5tC9Bh5c2/QRpDFr6FN0HPReGqgzFCElo2wV18MCseed9Mqpo1FBT+51uRGNFbiRlEels7esRifc0hrVHjp6PyKRTJL9x4ySaDX9+pbdzqs6ZNNh0BRXJ3oFc56uJ4UOO3SUCWFvHcV7bv6x4hAEqGpRkhKXCi29Ubo8Spbfwc0wEsBQhXbFqrhXRaQyyW6oR4/NA2BfPaYM/wxwj6rQgd3Ndhg+zQUC4t64NeC26MBwMwKeg9J5gQG1QXsC9B4/OEAo1h6Ptie5VgGzi0x1bs4cI76NzdaVanTx/wfBrKiuLB+EVFWEUsCysOUvuXShThiIRaykU1FxNS3OLLdttyY4d00o/qv/hhe9GuG4i8yUUdVHo6E/GlPhwGCvxBYnPUcpc71owe9KOZut+HgeHLXCZDkwG/jgZjhcARZ2PFOxiMVII99bCfs3RreywL4c66I0QnBeAK+2BNPHH8RtVQ9rasV3bdb5dfacwUeMbjhfuMk2dbsO/YefnSoQC/63t3Rt478EbywSSIeCFMOg3RzQRh7pp0arNMqc/TvS3rpyPwKjw8mCM4FY3mAvvh9yLZMGprbCXv0YzoJV+9PW0zqylbQ3fGFYQyW0sZwOJ/b4BkWexKYRB4f5zSN3Uj4P98bx3lkHkQ0XMp/Nwn2pj7ZxSKIhZ8WduV5ruk5ZU1wGCC5i2DlyI5/J4blh95K7IX9VX9+8Bor+Tfw2K4Cuf7P8IrXGYEsn5E/OGiqnPNq/SCuojiMfeFQG0h3KNQv40BeHAefUqa0KNlkxFA9F4ItbqctpPiWyH1GzU s3yDlszV CJ7x5gvndm4Fj5+akydi/s43tAk/31rkkfBdEoa+DEy0AVqNLeSlTtSGXYfjCwOB33ERpmzyMtGQjs7PMo0N+z/I+psTAfCX3N4B0N7jQmbFj2WRYJxJZgz04j2W4k5YwU8UdTCnTI4sXru3An0gRNYaqJgMAIYVtu5UVg++leL+HcxN4FJTIOlUfjT59L/hAhMJuYFPU5SC6/C9Q3U8ydTcnASCmm5N0eDZU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: By making release_pages() call folios_put(), we can get rid of the calls to compound_head() for the callers that already know they have folios. We can also get rid of the lock_batch tracking as we know the size of the batch is limited by folio_batch. This does reduce the maximum number of pages for which the lruvec lock is held, from SWAP_CLUSTER_MAX (32) to PAGEVEC_SIZE (15). I do not expect this to make a significant difference, but if it does, we can increase PAGEVEC_SIZE to 31. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 16 +++++--- mm/mlock.c | 3 +- mm/swap.c | 100 ++++++++++++++++++++++++++------------------- 3 files changed, 70 insertions(+), 49 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8c65171722b6..07d950e63c30 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -36,6 +36,7 @@ struct anon_vma; struct anon_vma_chain; struct user_struct; struct pt_regs; +struct folio_batch; extern int sysctl_page_lock_unfairness; @@ -1533,6 +1534,8 @@ static inline void folio_put_refs(struct folio *folio, int refs) __folio_put(folio); } +void folios_put_refs(struct folio_batch *folios, unsigned int *refs); + /* * union release_pages_arg - an array of pages or folios * @@ -1555,18 +1558,19 @@ void release_pages(release_pages_arg, int nr); /** * folios_put - Decrement the reference count on an array of folios. * @folios: The folios. - * @nr: How many folios there are. * - * Like folio_put(), but for an array of folios. This is more efficient - * than writing the loop yourself as it will optimise the locks which - * need to be taken if the folios are freed. + * Like folio_put(), but for a batch of folios. This is more efficient + * than writing the loop yourself as it will optimise the locks which need + * to be taken if the folios are freed. The folios batch is returned + * empty and ready to be reused for another batch; there is no need to + * reinitialise it. * * Context: May be called in process or interrupt context, but not in NMI * context. May be called while holding a spinlock. */ -static inline void folios_put(struct folio **folios, unsigned int nr) +static inline void folios_put(struct folio_batch *folios) { - release_pages(folios, nr); + folios_put_refs(folios, NULL); } static inline void put_page(struct page *page) diff --git a/mm/mlock.c b/mm/mlock.c index 086546ac5766..1ed2f2ab37cd 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -206,8 +206,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch) if (lruvec) unlock_page_lruvec_irq(lruvec); - folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } void mlock_drain_local(void) diff --git a/mm/swap.c b/mm/swap.c index e5380d732c0d..3d51f8c72017 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -89,7 +89,7 @@ static void __page_cache_release(struct folio *folio) __folio_clear_lru_flags(folio); unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in release_pages() */ + /* See comment on folio_test_mlocked in folios_put() */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -175,7 +175,7 @@ static void lru_add_fn(struct lruvec *lruvec, struct folio *folio) * while the LRU lock is held. * * (That is not true of __page_cache_release(), and not necessarily - * true of release_pages(): but those only clear the mlocked flag after + * true of folios_put(): but those only clear the mlocked flag after * folio_put_testzero() has excluded any other users of the folio.) */ if (folio_evictable(folio)) { @@ -221,8 +221,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); - folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } static void folio_batch_add_and_move(struct folio_batch *fbatch, @@ -946,47 +945,30 @@ void lru_cache_disable(void) } /** - * release_pages - batched put_page() - * @arg: array of pages to release - * @nr: number of pages + * folios_put_refs - Reduce the reference count on a batch of folios. + * @folios: The folios. + * @refs: The number of refs to subtract from each folio. * - * Decrement the reference count on all the pages in @arg. If it - * fell to zero, remove the page from the LRU and free it. + * Like folio_put(), but for a batch of folios. This is more efficient + * than writing the loop yourself as it will optimise the locks which need + * to be taken if the folios are freed. The folios batch is returned + * empty and ready to be reused for another batch; there is no need + * to reinitialise it. If @refs is NULL, we subtract one from each + * folio refcount. * - * Note that the argument can be an array of pages, encoded pages, - * or folio pointers. We ignore any encoded bits, and turn any of - * them into just a folio that gets free'd. + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. */ -void release_pages(release_pages_arg arg, int nr) +void folios_put_refs(struct folio_batch *folios, unsigned int *refs) { int i; - struct encoded_page **encoded = arg.encoded_pages; LIST_HEAD(pages_to_free); struct lruvec *lruvec = NULL; unsigned long flags = 0; - unsigned int lock_batch; - for (i = 0; i < nr; i++) { - unsigned int nr_refs = 1; - struct folio *folio; - - /* Turn any of the argument types into a folio */ - folio = page_folio(encoded_page_ptr(encoded[i])); - - /* Is our next entry actually "nr_pages" -> "nr_refs" ? */ - if (unlikely(encoded_page_flags(encoded[i]) & - ENCODED_PAGE_BIT_NR_PAGES_NEXT)) - nr_refs = encoded_nr_pages(encoded[++i]); - - /* - * Make sure the IRQ-safe lock-holding time does not get - * excessive with a continuous string of pages from the - * same lruvec. The lock is held only if lruvec != NULL. - */ - if (lruvec && ++lock_batch == SWAP_CLUSTER_MAX) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; + unsigned int nr_refs = refs ? refs[i] : 1; if (is_huge_zero_page(&folio->page)) continue; @@ -1016,13 +998,8 @@ void release_pages(release_pages_arg arg, int nr) } if (folio_test_lru(folio)) { - struct lruvec *prev_lruvec = lruvec; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); - if (prev_lruvec != lruvec) - lock_batch = 0; - lruvec_del_folio(lruvec, folio); __folio_clear_lru_flags(folio); } @@ -1046,6 +1023,47 @@ void release_pages(release_pages_arg arg, int nr) mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); + folio_batch_reinit(folios); +} +EXPORT_SYMBOL(folios_put_refs); + +/** + * release_pages - batched put_page() + * @arg: array of pages to release + * @nr: number of pages + * + * Decrement the reference count on all the pages in @arg. If it + * fell to zero, remove the page from the LRU and free it. + * + * Note that the argument can be an array of pages, encoded pages, + * or folio pointers. We ignore any encoded bits, and turn any of + * them into just a folio that gets free'd. + */ +void release_pages(release_pages_arg arg, int nr) +{ + struct folio_batch fbatch; + int refs[PAGEVEC_SIZE]; + struct encoded_page **encoded = arg.encoded_pages; + int i; + + folio_batch_init(&fbatch); + for (i = 0; i < nr; i++) { + /* Turn any of the argument types into a folio */ + struct folio *folio = page_folio(encoded_page_ptr(encoded[i])); + + /* Is our next entry actually "nr_pages" -> "nr_refs" ? */ + refs[fbatch.nr] = 1; + if (unlikely(encoded_page_flags(encoded[i]) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + refs[fbatch.nr] = encoded_nr_pages(encoded[++i]); + + if (folio_batch_add(&fbatch, folio) > 0) + continue; + folios_put_refs(&fbatch, refs); + } + + if (fbatch.nr) + folios_put_refs(&fbatch, refs); } EXPORT_SYMBOL(release_pages); From patchwork Tue Feb 27 17:42:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4305CC5478C for ; Tue, 27 Feb 2024 17:49:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3FD194000E; Tue, 27 Feb 2024 12:49:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BEE5294000A; Tue, 27 Feb 2024 12:49:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB58494000E; Tue, 27 Feb 2024 12:49:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9B64794000A for ; Tue, 27 Feb 2024 12:49:40 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 73D00402E8 for ; Tue, 27 Feb 2024 17:49:40 +0000 (UTC) X-FDA: 81838321320.11.53E0D05 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 052ADC0014 for ; Tue, 27 Feb 2024 17:49:38 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HsfOVIQK; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709056179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JS2zWSxA9BHMVin23i/d0wNgroAnOrkk1liM0VEFQ4o=; b=ZdpCzMk7q8qv08OIzFxXS2FsqMcZnqdofC8AfZdzQIglG103qP0BADg/vulu3xAIekbd9T pjB5box36bcGjUC/5MPPVszLHe7krFJsbGQuSdxM1SwmPW7VOEe5WJgu0iFGWr8KXvOfAm oAPA5gZArLzfqPdcwjLaLpvXKQ2Bvss= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HsfOVIQK; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709056179; a=rsa-sha256; cv=none; b=J0lqQ7QtyXshS0euQUylm/jOnexcpLTsWQSVFnnFkt1mAMEE83SpjqVZ+7yZ9bMyZFThwe QwS7UGcVfPFTskIYQC6TvZjZ6XpK6M2FoYxqz0HXUHQAoEpzcvlqZp2eqifq4dPR+X79D1 oqxFlNep/s1P94yWZNrnoeoOvPYFWXc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=JS2zWSxA9BHMVin23i/d0wNgroAnOrkk1liM0VEFQ4o=; b=HsfOVIQKWyC+Rn+0o/6cyT8msq AM39rApF5sFTGBeNN24uLfQYSO3tdLsSbeptoezM/K5z1IEiV7hDLdiv+t5xEDLd3J1pM9Chjw6FC nDzssZOplts/RbWv0O5uHLFAnI3OaGykJCMRbp+sjWcVEylkYjQU3vb/iuQFSyMXAviJeRPP/jwD6 qY8qspuf0I4fZUBzlkzA1s+xx1FeHzNIW1skY4go9/0G6S122vbVjfspYHlykXOJZwiXt070Ix8NV RaLVKdSuZPZakKmuHAOmPvAtje2gZZDAOr6yuMDNL4XZHeQvinny5TAOcVwTX4sS0Ga7x3yLQ5Bpy /Zi2V6OQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yqu-0RXL; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 02/18] mm: Convert free_unref_page_list() to use folios Date: Tue, 27 Feb 2024 17:42:36 +0000 Message-ID: <20240227174254.710559-3-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 052ADC0014 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 6kwtw6iqi7jk5ryref7w9eds76xgcc79 X-HE-Tag: 1709056178-380765 X-HE-Meta: U2FsdGVkX1+D7XfAEiO/Zw9ieksmYSADe4TPDFFxsLYknK8jWGq35CMua/T683/8y12CVp6SKBMhl3VuMWBT0EtteAdcGsoXB6uyq9jtYB7pRoeENYOTjLUkG7KbFhlcPmiJ0Tl3RJc/MaLF6ljS36hIo9VJXuCIQ4pbOY5oKc6X3N008IA3SJ0cizda6gCdCgQAD2zXX5537seFlsAngrKfuUGowVCVZ4lVIMjPe6/3oldribmqrZxM0axyB2ta0U7WJF6CRTCDAUwjp6PtehgNeaS81U4o2Sc0Gbellh3z0nhEdlOH/ZbwtaqOama/yaKBgDK1/kGu/DUlTQqWccQoRhl2p7AC0gRNqYjKyrRMtgljMyOkvVruvc8v8dbbwIPbh92f1bhUUuamRStBu3O/GQuVkovH/fn6wBeRpxzjw13lxxL5h0p8N+2caKdDSzQAcf+tefr2XISnMZFHggZBpqhDwuPjZIVgNHYUtuX/hYCpINTruYQhQdTggjC1Yf0ypJxEnMs2s3jD7fWglmscPb0zqdeGioRgVryHPNUDZb6L6GDySPwekZVervqMhjtw5sfd9+e9cfaimo/X+HxW70/nC1u6yB4UkUfX/E1KkCReTB5AzUCMymI0H4sj9R9H1aSYTfnXz66EZP36aIXYZLDEHp8QZE5noOe2sIyWlaGJDb27sRXM3GV9EhFNDzCQ1QOdBYrFjc0dYxV40aBgnKBLZc/2P7czgvXMinNW91BWLiNz0eAGbShaCZ+QMqWrSOAeRWRREb6hR2T8giIggXUW6NhkS/F9SJKCLEB4noQT4kZPTbtrb8dtZua98w6m8SxhPigLBqq1E8FkUEJkl0O9WWgJHPXuYq+9Lj9XaYPRFgiaYtizKRNo+lGUxpcDDq6kJ6r1vnpsuxIiwi0wLyvhQ0ne X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Most of its callees are not yet ready to accept a folio, but we know all of the pages passed in are actually folios because they're linked through ->lru. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/page_alloc.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 96839b210abe..24798531fe98 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2556,17 +2556,17 @@ void free_unref_page(struct page *page, unsigned int order) void free_unref_page_list(struct list_head *list) { unsigned long __maybe_unused UP_flags; - struct page *page, *next; + struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; int batch_count = 0; int migratetype; /* Prepare pages for freeing */ - list_for_each_entry_safe(page, next, list, lru) { - unsigned long pfn = page_to_pfn(page); - if (!free_unref_page_prepare(page, pfn, 0)) { - list_del(&page->lru); + list_for_each_entry_safe(folio, next, list, lru) { + unsigned long pfn = folio_pfn(folio); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) { + list_del(&folio->lru); continue; } @@ -2574,24 +2574,25 @@ void free_unref_page_list(struct list_head *list) * Free isolated pages directly to the allocator, see * comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(page); + migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&page->lru); - free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE); + list_del(&folio->lru); + free_one_page(folio_zone(folio), &folio->page, pfn, + 0, migratetype, FPI_NONE); continue; } } - list_for_each_entry_safe(page, next, list, lru) { - struct zone *zone = page_zone(page); + list_for_each_entry_safe(folio, next, list, lru) { + struct zone *zone = folio_zone(folio); - list_del(&page->lru); - migratetype = get_pcppage_migratetype(page); + list_del(&folio->lru); + migratetype = get_pcppage_migratetype(&folio->page); /* * Either different zone requiring a different pcp lock or * excessive lock hold times when freeing a large list of - * pages. + * folios. */ if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { if (pcp) { @@ -2602,15 +2603,16 @@ void free_unref_page_list(struct list_head *list) batch_count = 0; /* - * trylock is necessary as pages may be getting freed + * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. */ pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, page, page_to_pfn(page), - 0, migratetype, FPI_NONE); + free_one_page(zone, &folio->page, + folio_pfn(folio), 0, + migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2624,8 +2626,8 @@ void free_unref_page_list(struct list_head *list) if (unlikely(migratetype >= MIGRATE_PCPTYPES)) migratetype = MIGRATE_MOVABLE; - trace_mm_page_free_batched(page); - free_unref_page_commit(zone, pcp, page, migratetype, 0); + trace_mm_page_free_batched(&folio->page); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); batch_count++; } From patchwork Tue Feb 27 17:42:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7098BC54798 for ; Tue, 27 Feb 2024 17:43:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D27F6B00F7; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C29F6B0102; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 003016B0100; Tue, 27 Feb 2024 12:43:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 977566B00F8 for ; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6FE7CA099D for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) X-FDA: 81838304730.21.F608B42 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id D5F6620017 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YthG6Hrj; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hIqW8ZzSmIsa86syKHo9BFjI8Rh2TkL2UCfXwvuToM8=; b=PoEMLyWjCygs/aDqIyPvI5BkBi3VRwOhlovA42Dwq2LPxQHYrmMiAP+SXUoQ/sstyVpJDT MFpV3bR1qaZ+pNYfu93W6yg9jtYDGX6wcSZuylG5PuRmbiSlYJtC9dXrBB0caGrWIL2nMA hnLSETj8XYijowVOZgmHWt9xqYg04pA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YthG6Hrj; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055783; a=rsa-sha256; cv=none; b=u9iP8o1btb/fAs+ANHRtO3OypKefSZpwYc/F5nnxRSH11HEaaYRKgeW2NehrUbnBo1nuCm 6KMMOWGpVaAqDTAQ445xm3tcwrTUvOcEhoDeLPRdb8SxVf57Tv/Cn7xpdfpr9Emq6BgiWi bWAb9yo+xC2wPaC5Ya+KwjtUKIVpIo8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=hIqW8ZzSmIsa86syKHo9BFjI8Rh2TkL2UCfXwvuToM8=; b=YthG6HrjgqHL2w7P6eAMjYwBgh 1NHeg5ilvyLwR3G4lvyzwC2R/xM+5JsRAxFtTKYTUhm7yKpe/nn8BT9vyMDVJfmHZDf586PZ6ymyJ Nun9YMEZNrcwBuPaLZVrO0gVnKzrvhC9hq/AbNrFEDi6X8IspCEYGuQw8biJR3IAkmbnQthTw8OmL EiNbW3SYsxJQOZmwjgiEsouKFCK1T9+KTc0SJXqYFHWLv4yzFoOx1JQg/bLw9G/Mij6HORxfY515m sebby4hweYlX2Udzmb5nFVyehNO0beJZZdPvYqNq6+cprBBxgwLLDRJZFwvfIirQGsZBZnPSvtDR5 IiTuTAGw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yr0-0y0R; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 03/18] mm: Add free_unref_folios() Date: Tue, 27 Feb 2024 17:42:37 +0000 Message-ID: <20240227174254.710559-4-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D5F6620017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: mpn9o7hagh6xt14kx5yp8oetdd6bxfrh X-HE-Tag: 1709055783-416954 X-HE-Meta: U2FsdGVkX1+Rpt5wXza3peAB2xrXO3YqeX1PGp+FeIk/S76MKZ+e3E8+qzz49RkxbaiCRN1f/H+e+U9E8ZmdlyvAh0kmPZtBjXdclfMsnbHgo3P3nGvzqORi5wEsTCrUrNBjHb5c3K7nK2/SBOKkL3NyTQ8HZFCTdKvtzkXlz/30sRmDEuyI+k9SFQuZbgLWAk2M5sf7gjXRnOn1ZAECRr0KNxa3ZDau6E5R2gk8T0ZDbUpW1qVUOQMbrxNQ9B/suJXrgEz6Ds9kF0cc4SfE7BJ5kJKmz2LpH2llFtTqspPo0mI6uhQph5WQeYBdv5db3KCGUYSgvHh5Sgp1kSCRiLuOpP3oJjh08v0UIdgSP3Sffukt3jkudpH5XhscoX1M4oXiPXNVfgUu3af75tlkkOwwo4afzwpdI/6bTeZ85PNTp03eekvAMqE1wKgxJ1lhUfhID5vGh33ekGYue29e2faKEynh0dG/Oa9bsIX2B4FqXEy1jrwIu727pv9tiFlVJ/7E0EMhYAmI8xdV6DyxhY7iYRT9GUbbv35HWIvvhNEwqOQrRuCeLAwgW0MFQ1D1Ff71RBO+q1Mbp0AHtXZSFVQZUiEN4TL39p4p0NpBAKFfqn6lEOLxMXsLhFNEZFOZ9kCbyRlz+KPbFQA5oIn6bESQwslfE2i1BBRCCJ0PKF/xPU7oPtyAtfDA3/Oku/f80hLQCLt46YVdc2SJ519n5jSdGg4heDPl6z31MtzVRaL3QszPL23nKeXxI04h1fRNw8x/fvjmoKI431jwhsG5OSD5GPhQF5lip5OcoZdxXMSayYVjMjGO4e/d9pcQbBewwJb6zkX/KzCw6Fen1F1prqSEXNRk5DWke1OwHSFQjFqv6TJ4NvR5GtcKE0rNv9JQF/dee2iDXgHxp8Vm5Tu2KPuCk43Q/ji+Oh4X+xobEvQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Iterate over a folio_batch rather than a linked list. This is easier for the CPU to prefetch and has a batch count naturally built in so we don't need to track it. Again, this lowers the maximum lock hold time from 32 folios to 15, but I do not expect this to have a significant effect. Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 5 +++-- mm/page_alloc.c | 59 ++++++++++++++++++++++++++++++------------------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b680a749cc37..3ca7e9d45b33 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -452,8 +452,9 @@ extern bool free_pages_prepare(struct page *page, unsigned int order); extern int user_min_free_kbytes; -extern void free_unref_page(struct page *page, unsigned int order); -extern void free_unref_page_list(struct list_head *list); +void free_unref_page(struct page *page, unsigned int order); +void free_unref_folios(struct folio_batch *fbatch); +void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 24798531fe98..ff8759a69221 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2551,57 +2552,51 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a list of 0-order pages + * Free a batch of 0-order pages */ -void free_unref_page_list(struct list_head *list) +void free_unref_folios(struct folio_batch *folios) { unsigned long __maybe_unused UP_flags; - struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int batch_count = 0; - int migratetype; + int i, j, migratetype; - /* Prepare pages for freeing */ - list_for_each_entry_safe(folio, next, list, lru) { + /* Prepare folios for freeing */ + for (i = 0, j = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) { - list_del(&folio->lru); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) continue; - } /* - * Free isolated pages directly to the allocator, see + * Free isolated folios directly to the allocator, see * comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&folio->lru); free_one_page(folio_zone(folio), &folio->page, pfn, 0, migratetype, FPI_NONE); continue; } + if (j != i) + folios->folios[j] = folio; + j++; } + folios->nr = j; - list_for_each_entry_safe(folio, next, list, lru) { + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); - list_del(&folio->lru); migratetype = get_pcppage_migratetype(&folio->page); - /* - * Either different zone requiring a different pcp lock or - * excessive lock hold times when freeing a large list of - * folios. - */ - if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { + /* Different zone requires a different pcp lock */ + if (zone != locked_zone) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } - batch_count = 0; - /* * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. @@ -2628,13 +2623,31 @@ void free_unref_page_list(struct list_head *list) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); - batch_count++; } if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } + folio_batch_reinit(folios); +} + +void free_unref_page_list(struct list_head *list) +{ + struct folio_batch fbatch; + + folio_batch_init(&fbatch); + while (!list_empty(list)) { + struct folio *folio = list_first_entry(list, struct folio, lru); + + list_del(&folio->lru); + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); + } + + if (fbatch.nr) + free_unref_folios(&fbatch); } /* From patchwork Tue Feb 27 17:42:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAC3BC5478C for ; Tue, 27 Feb 2024 17:43:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D96A86B00FA; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3CBD6B0103; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5843A6B00F9; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 088126B0101 for ; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5D128140A03 for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) X-FDA: 81838304730.20.02FCF25 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id DF4AF40009 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=g1L8o4YG; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pz2bbWtwmdw9j8cypGnGl7rt/rSb7O0Hc5eYv7pd3Vk=; b=ljaWuvcHuOasgA27SMYTBxodu1VgoTGeWlEnTY3GekLddJk8v3FXURCxxUStkH092a2Sgy GG1DfH1vH4wVzKs0oDBUyS4pHgy1sg160meDmN0QUgS2ehFsusOR1l7WoceRtkG8jUQGuk 5d0ptsc8vgMmNAe+fQiV3cS8yIpPSQw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=g1L8o4YG; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055784; a=rsa-sha256; cv=none; b=EkdB462hh3TEKA6ktZCJj4trHx9cgv26Kniu7QIQplFxadhQHtA3BX0r/F3Pm1bAhxfouy NzZtwdIP9zmneojzeW3jcGtKw0b0bkTfYOeoafN67MoE/8Xzwcl3T6y6Cs6NQqYOiUgqyv sYy2czbwjDLEd/TZ+NjFDbU2J7r4Sy8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pz2bbWtwmdw9j8cypGnGl7rt/rSb7O0Hc5eYv7pd3Vk=; b=g1L8o4YGNT2lx561EE3X8d5fi8 wOEsv7PUCqVAYekH2yF7/MDJ8f7eS5GHQxHedfGY+wdFIlmG2ikkF1S0HlWnQ9dFe9gEPMnHrV7xb Zbwh42nUQEQOiTw2ans3hOJGkXSelrS9CKOav+YQJ/vmbAk4I0XWJEoimr4RjNKd8CGMbYAxQ/CVn fPtgHPX+UC/4eaq+2tRTTIPGB1lBr/CCyTJpP3ROcHcSYAoCKD6s7PZ7a8cCbs28mwS9oFTlBpt09 q6Y70tkln2i9BrLhNkodydJoEzr9wM8NAzx//4kbR1+5M2zt//Ol39c90Eb7rmbVg1BU29yy/C12y T/mkDHSQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yr2-1MjL; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 04/18] mm: Use folios_put() in __folio_batch_release() Date: Tue, 27 Feb 2024 17:42:38 +0000 Message-ID: <20240227174254.710559-5-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: DF4AF40009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 9r9bc5c3fsdmfwwxr4daiek4g7ceojcf X-HE-Tag: 1709055783-429194 X-HE-Meta: U2FsdGVkX1/DJHLsn3CSOgl8g1RI7a7D3rUg12iv9VYk+VADXksU0hVGrDnxo7RdoMWXWjQdlMztBkqFaM2yr17/0rx7Zh603vsATGpFRGCIpLNYTNlue+spLGOPoIORwzBLZiuXIGCffQmCR7XaNT+ByEXroq4SqBYE6IsNKW0G1XqHIwjmCAqK86l8VNeH2OFOoWT2d5i3j7ooxITWJu+J+XtOrKvk8masGMcYKLiFfJOzoDcjBm1xbSCm5nRlexoMo8qczg0Kynb/8t7ipVJN+s8snDwRZrrBGaFTVsh+QyWGw21U1ZQl2LdYiGJdIUpNlJRtLC9n4GR9wYx72NGkqorpxSSLt2Q3M0ipux2U56DZHRwH8x1KebEtnppdB0fBqTmUC6ic6iZHSwnNFaXPYZbUsU4dcJnS8v1+y2d2x09yB1R0WnXzo1u0fqWU+v71tpJJ4jJEUe7gxa76zKpSP7txNmibdwJI1wYcaiaS9525BhCX0JMFdFlIls+iYVBn2L98kWbQ56eP6iFK0HLKiqDB3z1wDUDHwLqStG7e834yqESSEZrdRRuf24jvCyihZwJerzu5JPdn61pg3658k8Qzb2780MkREHdq51xPNUZnUdKrYI7F7U/5XYJbi0kjzxn3pFZZF7gQTZCILHkvNEAxqhAFCB5NCUewywjP5uhZvluHLStFQl/yBLAnRWimc08upTaMGmlpEPPf6nUWoMaCc7KaCvBsscAwk13Qo3xWQCbZjkRox3TUOnl4ugpibCZghUW59dFEudVMAdLAGITD10GQDGKf3JFD7/laORfDpAmTFUyN/VdIP87TTioLCzIlCY1+CxJGYAmJp7rMUIIrq9KrLzWSn/v/KPgcPxPKujkYyfIfomwLs8iySW96cDnVocNfH00Fhb4tHPZsEEgAm5BF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There's no need to indirect through release_pages() and iterate over this batch of folios an extra time; we can just use the batch that we have. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3d51f8c72017..1cfb7b897ebd 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1083,8 +1083,7 @@ void __folio_batch_release(struct folio_batch *fbatch) lru_add_drain(); fbatch->percpu_pvec_drained = true; } - release_pages(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_reinit(fbatch); + folios_put(fbatch); } EXPORT_SYMBOL(__folio_batch_release); From patchwork Tue Feb 27 17:42:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A36F4C5478C for ; Tue, 27 Feb 2024 17:43:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E4EB6B0085; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FD936B00EB; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AFFE6B00E9; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EC40B6B00E4 for ; Tue, 27 Feb 2024 12:43:03 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A302AA10A1 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.07.3777D05 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 21E1D20008 for ; Tue, 27 Feb 2024 17:42:59 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oLA65UaV; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KZvuxZ8Z0GdHFwXlTLgUAXn7W0I0rBhhGqeJNWJMEA0=; b=BTFGTRMG/iCSzt5CJu8UYpi70E5nZgJ6hZD2WDGSmoOFjJD7uzQefojpGVaFbdXDCL0nXb qD+D+4hMC9IMvqHIB+Cw6mieShQKG/vscBPeHLioLC72RwCQeYSWbZuKVuFpiVgxXJEk/B 2f8iZjx+UReN4lmOe7EhoF6Yos7GAr8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oLA65UaV; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=BCS52T956abdAT3eYT5LXNeiuIEVZnjL7SyvaJmr3vS3h6B8VMyndhp78L5eZp2rKd8sBV lMvHWHbyT2C3j26k5A8FpCy33yR8mB2UAMpmFv0C735W8Q6xcLeJVzoq3nAlUDNKp4c/lG W+NAduBHn8Pi0FqB5OfnSp+/RR9tkQ4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=KZvuxZ8Z0GdHFwXlTLgUAXn7W0I0rBhhGqeJNWJMEA0=; b=oLA65UaV457eibUT18eWCWgnFG 7i9qJyvLmglWC9HBBE3Owtm1FgSzkyUxBqVCsR9BOJ0sVYSoPHIFf0izlh7JdY8xPyUFpGqO7JYWz sxz5AJJUyh61l5Y/XP8Ym3YNGW5+L+0bB7LbM0mbLZ91tuxoNmTCF+H+ZkvWctgklTc/DPVztoEnt NYsHUlWxiKUjBxZmoahu2RVgTdwCW4tz3GowMgkwMqX45ZCtfsyYJn1K0LGpMMIX4FaYe1LYHfKab nk5ENNetdfgBxmFV0T0gdyeqa9sFRzl/qRUBFMUQnQTcoAK0nMj/+KSlOtebOuU7lCdpkz0ZBrAR4 z+yJ7iZA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yr8-1vHD; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 05/18] memcg: Add mem_cgroup_uncharge_folios() Date: Tue, 27 Feb 2024 17:42:39 +0000 Message-ID: <20240227174254.710559-6-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 21E1D20008 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 84hk9scngjymz58sc8tzdezfn6fpfbpe X-HE-Tag: 1709055779-415265 X-HE-Meta: U2FsdGVkX18OLLpZZkDgjtq5Jq9I0mfXi1GgF9drdpRhbrF4NX0nVDSqXVCDw0NCzbuazWlo8KMhmWVG9M4RqHo+40Fa2J6Kgxp8Sb05guJgXgkjM2eVpfYEeFk7gkNfjFUgSnE6HvkyD7OKKcwzgAwmPqK8dvWPI7+xiKaYFVEaExSGQccdLnf46jTps0WQcWOkVLRePI3ZW2ogV81+s8RPxq91KWSUgP+SbJgE8lbF65UZ3OKlJWi1UtdbKOlxO+5nCO27noU3cnFLFZzrSpgtSEGtPi1ILUqTUc2I46VzHPWZvP73bF4LyZB985eLR5j0aGmC1dSnnemDtz6+sQFYktbBDmjMyLPynWn0ZL1S8B7PCdiJmjyrOI5ByAWv/7mljP5lRe7ddXblIFHXpRVODjlVK/ih7vljF8gUpyFS/UCEyWcMK6AJmgcD++somUkrcrA7u5+FvCulLKLp0U4JfGxJ0BELgGlx7w/sFFwlPV6TtQp3yvSXkgYJ1OLxYZhf81PNS5aMiymtB58Brp6JlVlWIA3jjW5mUe3UmiyhWZFXVUtsLYRxxR8mwxBgq+APEnSMr4CdqdE4YQ2J7r5QzPTrwHn8gdQX7eEk1jvfRtme1a/C7whEA9ZSCYTbZHDLiVvOS8O9IKJlCAx3UyavEEXpyY+1DtNAtvJtjxnEm8H/A1UhS6r80AfuhsxI8lQSmme7JOZoXbHx8U41wSYDS/AIuAzfJzuG1OJfCFRLAhSwcWakoPY+u4Y/k4+aCGH/mHuyEExtK+C9H9F9Xe0JYtQjvD1uH7JKOF00UJfH9ReK5GPKdjxWGgfA8ZnQRjXnom3GcRS8da54pBHekYDw6qNofczgPimWKFgGOg9rXOc30s2ckHM3RyO9z0b69Vu1MTS9MBZ71jAhiZnLylV1fxE7Ynvszybc2katVOo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Almost identical to mem_cgroup_uncharge_list(), except it takes a folio_batch instead of a list_head. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/memcontrol.h | 14 ++++++++++++-- mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4e4caeaea404..46d9abb20761 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -721,10 +721,16 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) __mem_cgroup_uncharge_list(page_list); } -void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); +void __mem_cgroup_uncharge_folios(struct folio_batch *folios); +static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ + if (mem_cgroup_disabled()) + return; + __mem_cgroup_uncharge_folios(folios); +} +void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); void mem_cgroup_replace_folio(struct folio *old, struct folio *new); - void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1299,6 +1305,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) { } +static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ +} + static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 95c3fccb321b..4be37c9a0759 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -7564,6 +7565,18 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) uncharge_batch(&ug); } +void __mem_cgroup_uncharge_folios(struct folio_batch *folios) +{ + struct uncharge_gather ug; + unsigned int i; + + uncharge_gather_clear(&ug); + for (i = 0; i < folios->nr; i++) + uncharge_folio(folios->folios[i], &ug); + if (ug.memcg) + uncharge_batch(&ug); +} + /** * mem_cgroup_replace_folio - Charge a folio's replacement. * @old: Currently circulating folio. From patchwork Tue Feb 27 17:42:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBA13C54E41 for ; Tue, 27 Feb 2024 17:43:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4C7B6B0109; Tue, 27 Feb 2024 12:43:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CFD206B010B; Tue, 27 Feb 2024 12:43:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC31D6B010C; Tue, 27 Feb 2024 12:43:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AB7BE6B0109 for ; Tue, 27 Feb 2024 12:43:10 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 829CA1A0BC7 for ; Tue, 27 Feb 2024 17:43:10 +0000 (UTC) X-FDA: 81838304940.10.8E6B135 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 1E60FA0018 for ; Tue, 27 Feb 2024 17:43:08 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="j/AMIWaJ"; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ejGLYMhF0Jxik9msp0N1jcC/bsh2AhaNDjT+XtzCS7M=; b=XYJd0r5jR7vSxoIcyy+AMpZPVXov+wZklagP5uJCrNqPdAOZQNOsyEKM4GL1wIESch1NnU ouHIC/SkSgI2R5L7JikmZ0qHQs+umPrbvtqJgAugLGkSimIjax+4jfiru/W6WRpvQBEUZP nEW8779li2/950JnIIg/Qu6WMutJd5c= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="j/AMIWaJ"; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055789; a=rsa-sha256; cv=none; b=rjw1N0xbM53k+5Gbsff6h+xHTKgrJyfZ549QTkzsIO9M7RPSkZmh6VqINmlGzPPmd6RFYd 5hd82inYAcvgVKsoZQVqjIm17oRbgNhIbMpFyczJYr6uDNLQUJlDneItYhLBHLiAPfPenE Lnm6wwzI8qchIoYxzgEi7zkFt0wbn+Y= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ejGLYMhF0Jxik9msp0N1jcC/bsh2AhaNDjT+XtzCS7M=; b=j/AMIWaJBix9kZWucU+y6s41Pv 6eODNjC9GxX/iGj1UoUBPC/z2yW2XkhUvXVcfUzqv0XVo5642VvVt3QKHY0nSisoCUWKYvAckiD8O fhWtM0MkEEpCzTRQnEAdlh0eKUQpEBjYDnWj2HM4TX+tE9M2fbscTsSDMyyboET0M3NFVXinqtz3Z iThI5luEuBVAEtrJ07Jpo3as1HMEddx6F8goay0ApyP0cIHEN9wn5z5K7bDgQwiU8rppTvCxuTeeg m0tjwZBao/EmBsJzNjghpwQeey8jlOGWULPqLOvzAHJ2uez6qeCPpvgEXRWGc40JbESa04UGA9/Wq 3gnELJ3A==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yrA-2Ey4; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 06/18] mm: Remove use of folio list from folios_put() Date: Tue, 27 Feb 2024 17:42:40 +0000 Message-ID: <20240227174254.710559-7-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1E60FA0018 X-Stat-Signature: czoaeahh19p868ewten7gjpefnoouij3 X-HE-Tag: 1709055788-584415 X-HE-Meta: U2FsdGVkX1+o2/zzpb4enzXk9SwcGCXmHrK9HCCG4zmz88KbK2OqmzDBDM8ZrohRBQdr4JJzSvFenbI6qSKF3Pd4qdffNCgQpkidtmf2dk3/ZxRrAd9/51EpcRq+EOqT1m5kLRib/3/NPJIX/Fpyn/xVC+QFnXnTAPGo8YSt3SfeqA6uAvPBMy9bqXA/DFOqu1N1VKxKhtsbqOFseB1e+Kt0Eg+CtU3mkjLIzpfg1GtUtoYl4nT8oXbeuNvjm2L1YoIX7KlO0ds62YKHiDjGVdvpFJ8GBR8vRfMkeH2vu+y7tlgqoJYETW5L5opMXxCvnFvGNKw5X8HwC8pt6AaiG3rK16RBrQMOKW8BjHynm4cErNvzHoDUTczQLrF6VeZKcCL9NXf9Xqx4hHu11hTIODG/3C30/b6FExWb2F7INeYBmO4d22Io8W3QFBHLqCWice9BdlfyfUaL1Uu6zxFibHvicD8Q33/5YVXeQ1UqBIVZjvfpcY6/96Yel/CCEDZgD3xbtEQzj8ggp6ZFSbdqyPGxEqh26sNbjYEwpsbH73O58ORjVxK3wMjMYYVJOEKaxEMVC0bDqoKBnbf7LRPIsybNMf2q3MonKM9FEeucRQfPXRyFOT3pLIrYWoU9SJ5itU9BF3hBu6Y2SbbcC0cHXQK++I0xxh0PSht+jJopcD0uqJdteUyQe/EThi/Zk8+jBSXHYiPMaoEpASACkUNOBK6h4e0LGoD+5IepYWgaom+wFKExfLODPCEGVP9juDN31G4ByawsZZDI9Sn+6zPuRYXElTS0olVAg9DVJPyll6Rv/uWRhIGHOqifirLE5y9ZSaA1cwsiVNIwkl0MCQIRlSKqVBlM73xbB7DIUxxChKth8OSChjpOrVTJ1PVUqx+zL7DeSRKcoA2KNiA76+dRqUp6Gxv/KWi9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of putting the interesting folios on a list, delete the uninteresting one from the folio_batch. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 1cfb7b897ebd..ee8b131bf32c 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -961,12 +961,11 @@ void lru_cache_disable(void) */ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) { - int i; - LIST_HEAD(pages_to_free); + int i, j; struct lruvec *lruvec = NULL; unsigned long flags = 0; - for (i = 0; i < folios->nr; i++) { + for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; unsigned int nr_refs = refs ? refs[i] : 1; @@ -1016,14 +1015,20 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) count_vm_event(UNEVICTABLE_PGCLEARED); } - list_add(&folio->lru, &pages_to_free); + if (j != i) + folios->folios[j] = folio; + j++; } if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); + if (!j) { + folio_batch_reinit(folios); + return; + } - mem_cgroup_uncharge_list(&pages_to_free); - free_unref_page_list(&pages_to_free); - folio_batch_reinit(folios); + folios->nr = j; + mem_cgroup_uncharge_folios(folios); + free_unref_folios(folios); } EXPORT_SYMBOL(folios_put_refs); From patchwork Tue Feb 27 17:42:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1920C54E41 for ; Tue, 27 Feb 2024 17:43:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C3626B00E5; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D705E6B00E6; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CBA86B00EF; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1BCB06B00E4 for ; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DC813A0947 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.04.7B613AC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 173324001D for ; Tue, 27 Feb 2024 17:42:59 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sk28ikhP; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hW1lKddwChCwhgOj8zOdkpVbhWbm50HPXPJZ8BtEokk=; b=i/GOTXebRmdm2x/kVqRfRTuIeFWAwAp85mBauhs0RxMAZsnVDHjrMHJS4tgrd3XKWBTEPn 733ShdRty+0Zbi2Tn43WwF4UqgksQXHTXhleTNo4mZrooSUq+0boHxtEiGnG7v3rxdkf+h tXfDym+AlEEC3am+nkCL12qQRgGMkys= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=llRwOnlYetPpO/eUUlP//570jJMvYsIcz6dg0C+7kM3LKzDLlkkjq+aVDNBxaZNPq6ZfLS plfVFCIx7rx+Ob9P0Maum935mYU69VcMST+shjrbQNXqBn5RpzyaS2dfw6qD4i45Zdpgk4 Nakg64P835oqkQHn+ygcUgnmS/rPMYE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sk28ikhP; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=hW1lKddwChCwhgOj8zOdkpVbhWbm50HPXPJZ8BtEokk=; b=sk28ikhPtxlulqjzsEEhm6+rFc iI0jjNEg5txxUm2cOc8pcPTIzgs7Ki1gFzebZnoZujX3/NP7t6aRwKuxaxi9uPHogI3FRXXM0bHFD YprkMjmXWQLiO1E8bfNpoPDzOTuf/mKu0ZSS/tNzoizUfOLJeFCw98taEI/pvhVpe/ZFGokxb99jL TsW1xOby9au9D5VK5lqgtD1Ee/a6/WTqTR1geaxsRjrBtApCd9304anGZPCL9QI09jISeWWOK4j7g LVVALZQd1xZv5euFluHmeCLfQCstr2OJq6/LIF1TrSxRS/2oBWywcfbSblgvq47G+N74sK7M8UQUm 7lUH4v/A==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yrC-2av5; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 07/18] mm: Use free_unref_folios() in put_pages_list() Date: Tue, 27 Feb 2024 17:42:41 +0000 Message-ID: <20240227174254.710559-8-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 173324001D X-Rspam-User: X-Stat-Signature: yjhgfofy7bjm767wtuy3g3moiqopq6ab X-Rspamd-Server: rspam03 X-HE-Tag: 1709055779-206010 X-HE-Meta: U2FsdGVkX18p9pN2B2jsFWSi6j05f37gNOX+lED7TKLXwG5DqsQ7b2VRZeNBYrZRz4GrSZ30N0rhQxaBMYJZ9PWeAa0b68lG/198WaHx1MteJpwy4oIhPbA8M4coaopeofPXPYtddKR5521d/eVyW0VKQ1gfawSSG2bL+5rQxTKzANXCeI8xLQFIyEC4Xn/8K3Fn7JsEFDbGcYEA+H/YUzYmdsUIifbWHavItkruaafCqO2Fatnf61i1pWBKqbaFuEG/puXWfGPXd3ogR5lrO8E/EyRvM2VqrMZ0esoqIXDs9A4Mpb0Q3FyCa1FSyRVB0qP5LU8bqvPTmZXPP/IQbrS9J4S0bNp3xwWZuzUdav8ug1EnIrY4CzGpv4PrQeLuUKOTxofg9YPRSJFaLUjWPIcwmVPhOJGVsFhWuaFaqoU/TFFTsl6r39NqLaLye7Bi1TBlUkvMKvy63NhJtTGqWPEKb4LhUnf7SD+89NbqE9GXjUc2nCXmpC3w4lzCiyNx18Wql4jpzj4pxifvOiCyoI1tnF8yOcfzRTmyPMM3bjKuBGYxSiSzmoThSc0siV2TXPTZ0Wr6NrOlaCiAJusZDF4s8JmVlQ1sDx76VKfq69UMhq15MsZnU5lYO456rnp7qYlVnvUmaYN7RBNd9DqtBLRmGegl8XXOwf69HbktTKHEDVNYoWBVtrhhTbg6kcYQuKCwWyWqpe3U0zT50RG4utkw6FF17EurxRzOL3RkliT5qJL7zNr9lDn7D7D1EV1HlH6+YDVD1+LG417lkak0AuPub/nXo3+bvp6vnGoBMqv+zpb7Qsqk3+88CGddwe7wSNzksujgKuJ7LjWSPxrvUwqSr8egzrc9WUX2ePhdB34pu1DB81IuiIh1AVsnprkuG3cGfhN6DXp1vn4+q/1LKsRA4b0OZTaQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Break up the list of folios into batches here so that the folios are more likely to be cache hot when doing the rest of the processing. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index ee8b131bf32c..ad3f2e9448a4 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -138,22 +138,25 @@ EXPORT_SYMBOL(__folio_put); */ void put_pages_list(struct list_head *pages) { - struct folio *folio, *next; + struct folio_batch fbatch; + struct folio *folio; - list_for_each_entry_safe(folio, next, pages, lru) { - if (!folio_put_testzero(folio)) { - list_del(&folio->lru); + folio_batch_init(&fbatch); + list_for_each_entry(folio, pages, lru) { + if (!folio_put_testzero(folio)) continue; - } if (folio_test_large(folio)) { - list_del(&folio->lru); __folio_put_large(folio); continue; } /* LRU flag must be clear because it's passed using the lru */ + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); } - free_unref_page_list(pages); + if (fbatch.nr) + free_unref_folios(&fbatch); INIT_LIST_HEAD(pages); } EXPORT_SYMBOL(put_pages_list); From patchwork Tue Feb 27 17:42:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7A11C5478C for ; Tue, 27 Feb 2024 17:43:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BC3A6B0108; Tue, 27 Feb 2024 12:43:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18C896B0109; Tue, 27 Feb 2024 12:43:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD86A6B0105; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 39CF994000A for ; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B62A41406C3 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.05.118D8F7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 0B31E40009 for ; Tue, 27 Feb 2024 17:42:59 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FzBqebYw; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GbovS5FPii+pMRhRNiTk/ANTExABfXg9lzmO/c4iads=; b=1A1L8CvNhiWcH3neTojtWRJzK/2tWb2JVNb5Y8Ts7sXxQ0U6tiGiKOB+1drc/3o+AbR04I lYxYm8GQmy9AD071lylte1zKMnyCKfWyUNwMh9g3zho+x858wDXTWwHjewz0OKguVujKaY wgbIomsAFrAin0dOq7gonfVLWa1gvL4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FzBqebYw; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=JTw8C8dnLwqcvlempgpPDh8HU7qnpjIHQMen5O5tKiJDpCdpsw2I/zI63aTS3VwEKjGFZt YuCzGbQKnGFD8bjgeTXaBmatGZ9fBkSaqQzxsCJA7438TSUce5hYns7Z9/nrC6WPWAwyzg Tgz61gqnIsROXLBoBNraijGxLaLB0lU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=GbovS5FPii+pMRhRNiTk/ANTExABfXg9lzmO/c4iads=; b=FzBqebYwvSGCp8nWlvoCbpv+FQ I4WEGsuyw8nlTqlulYAN0xetVPSOJjcd8siGHSrecK2bSGyhXihq7BZc6hFZ8+6CI6Xl5O9tyrlgz ozKA4DeUMCg6XNzxuXVmYUmFs37Hb6HfwxKeW8gprfNT4DSaSK3QNOOUAXKXD+dgiJtN0G/+MI0hS mEM5vKnW1/Mk7CJokCspgQC4WhxADysRKoULkV9dtqdAQr1/fErwciODxYAraztF6LbOYft7j73A9 ZheFvxkdIsZ3FRwwSuZEDneXtZhxkiouzvnp/RZx8nkYgKrRFVDH/NH9IcUQsTWJrlHv/pF6oaL4T x3pBprLA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yrM-314L; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 08/18] mm: use __page_cache_release() in folios_put() Date: Tue, 27 Feb 2024 17:42:42 +0000 Message-ID: <20240227174254.710559-9-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0B31E40009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 6izx4yorbj7tuofiqyzrfwb7nmdrgkmb X-HE-Tag: 1709055779-323475 X-HE-Meta: U2FsdGVkX18+lsqDX4On6WnSEBnsjJYs7JN0lrAlRGPQ57OuRch/7PC2ECiLktMqHCgixE2l4pkoRAfKWQNMj7ApEOS/WyNNXVQX7gsG0UbZy0+Y7+CujyHYg7696318/+kaIxyDdfelhgYFwRGBVh7zKuV+MGYwfWlmZkROZF6PYzzFInFk0jmmB7vz7pStbeqF/spddiZgD2WbIxcaS3HombvpUXZ9xk1DlyLQ8bb6qcoNNcNIop6tVn2/FHp4TUQz9zt8LzBLMJAcrdCdU8rsNMxBR1G/K2i2xRzfiGahvVFAo0zRBKEJEhfsJtd1O3llX5PtOKhWWJqw9qCcVVCabaJqYSXPGvaWctaE/tYLOW70jC86t0/HJfH5CWf1evu8Loyk5SCc5mfSFQPzsY5W+VX6Dc/7sO/ludEUSIhckDTe6fSGydQjx7GcvKCM6gvOnovNGEItDzTSWjUwJfVZWihRdEi2T/XEWRb6pvvF8kV6YeIH7cbbuCD4NG3RGPxXDdf9R5oJq1kCbwkWYfSPr+xizO0uAHfRf4zJyl4hgpq+mfjGjByloxE/Co1P36MPG1ejDRqZp0uLvCE2seUWaaskwwqRTuwmX8pOHN+NBBonmhzRPzxXWtCBmiIbxtdA1bStS2URuewiG4HDaFV9zyUF20ncOeOF2JqetTNDxW+xETo6B5LcAWOr1Ei6rjDLJpJVL3GWBSOSBBJqcaBpsnbBkuynZAzK7+nXPdGrvBmwFMrSqg8j93hVusxi94bji/5/g6dWYDphvwwY8fgMDtSZYO1t1Wm9hG2gBy5QTKUpf4wNevk3b8OCo359V/PEjrHOuzdwtfKEu13+E4W0scGQWXcLSvViB6cJiChCeqxtWYIxQx6GAwsV+Ca7IX+URKSVvxaP3nDFOk6HtDY07lDJH1+JRYh7xkhHkwg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pass a pointer to the lruvec so we can take advantage of the folio_lruvec_relock_irqsave(). Adjust the calling convention of folio_lruvec_relock_irqsave() to suit and add a page_cache_release() wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 16 +++++----- mm/swap.c | 62 ++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 46d9abb20761..8a0e8972a3d3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1705,18 +1705,18 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, return folio_lruvec_lock_irq(folio); } -/* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, - struct lruvec *locked_lruvec, unsigned long *flags) +/* Don't lock again iff folio's lruvec locked */ +static inline void folio_lruvec_relock_irqsave(struct folio *folio, + struct lruvec **lruvecp, unsigned long *flags) { - if (locked_lruvec) { - if (folio_matches_lruvec(folio, locked_lruvec)) - return locked_lruvec; + if (*lruvecp) { + if (folio_matches_lruvec(folio, *lruvecp)) + return; - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + unlock_page_lruvec_irqrestore(*lruvecp, *flags); } - return folio_lruvec_lock_irqsave(folio, flags); + *lruvecp = folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/swap.c b/mm/swap.c index ad3f2e9448a4..dce5ea67ae05 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -74,22 +74,21 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), }; -/* - * This path almost never happens for VM activity - pages are normally freed - * in batches. But it gets used by networking - and for compound pages. - */ -static void __page_cache_release(struct folio *folio) +static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, + unsigned long *flagsp) { if (folio_test_lru(folio)) { - struct lruvec *lruvec; - unsigned long flags; - - lruvec = folio_lruvec_lock_irqsave(folio, &flags); - lruvec_del_folio(lruvec, folio); + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in folios_put() */ + + /* + * In rare cases, when truncation or holepunching raced with + * munlock after VM_LOCKED was cleared, Mlocked may still be + * found set here. This does not indicate a problem, unless + * "unevictable_pgs_cleared" appears worryingly large. + */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -99,9 +98,23 @@ static void __page_cache_release(struct folio *folio) } } +/* + * This path almost never happens for VM activity - pages are normally freed + * in batches. But it gets used by networking - and for compound pages. + */ +static void page_cache_release(struct folio *folio) +{ + struct lruvec *lruvec = NULL; + unsigned long flags; + + __page_cache_release(folio, &lruvec, &flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); +} + static void __folio_put_small(struct folio *folio) { - __page_cache_release(folio); + page_cache_release(folio); mem_cgroup_uncharge(folio); free_unref_page(&folio->page, 0); } @@ -115,7 +128,7 @@ static void __folio_put_large(struct folio *folio) * be called for hugetlb (it has a separate hugetlb_cgroup.) */ if (!folio_test_hugetlb(folio)) - __page_cache_release(folio); + page_cache_release(folio); destroy_large_folio(folio); } @@ -216,7 +229,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) continue; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); folio_set_lru(folio); @@ -999,24 +1012,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) continue; } - if (folio_test_lru(folio)) { - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - lruvec_del_folio(lruvec, folio); - __folio_clear_lru_flags(folio); - } - - /* - * In rare cases, when truncation or holepunching raced with - * munlock after VM_LOCKED was cleared, Mlocked may still be - * found set here. This does not indicate a problem, unless - * "unevictable_pgs_cleared" appears worryingly large. - */ - if (unlikely(folio_test_mlocked(folio))) { - __folio_clear_mlocked(folio); - zone_stat_sub_folio(folio, NR_MLOCK); - count_vm_event(UNEVICTABLE_PGCLEARED); - } + __page_cache_release(folio, &lruvec, &flags); if (j != i) folios->folios[j] = folio; From patchwork Tue Feb 27 17:42:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5275C5478C for ; Tue, 27 Feb 2024 17:43:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7EF96B00EF; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CFBD86B00E7; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 581666B00EC; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id ED02F6B00E5 for ; Tue, 27 Feb 2024 12:43:03 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A7FE5A11D9 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.06.A701EC5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 1DCAE14000D for ; Tue, 27 Feb 2024 17:42:59 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XIdZvzcl; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g+JIHiaEA+ow4Xu3v6j8UgfoxcK+pXriMoQCo3/vWmc=; b=x/OZZvoDr2y5mKCEqjySmhTejtNXVhHhKQ2QSOpOwzmAA46onjempzbGmw8H6C2P6jUzep iRIxkJ/DH2PHFv5UC1ruftfYXPt0rywnsv7Y7gbhDMV0NrCA8XZ06ivjwe4YHBemCICAvj JyBUow3bUPOvylpM1KjIiWuZTCQvoT0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XIdZvzcl; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=oPO800SyOfA0jlS76L0nXRsa4Nml61QleWKopgQzW74Js/aW/j/JSWgMwGvTL9RLbLTTrA fleNONybH2AX4p2bTfhm+/3S7NHIM48+uGfqU44zTnkTX2LQU2H9RurU8lZ/2iLA1MRk6K rLwjnUD1EVBCtDcinNbGh/XSZB7sAzs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=g+JIHiaEA+ow4Xu3v6j8UgfoxcK+pXriMoQCo3/vWmc=; b=XIdZvzclPkFmta7jCnoJjtGtPs /FvU6xUq0gn4elsefXd9Q1FqbQggaFE/zPsXlehVy/hLP9VoOh1YyTapVm4m4WCOIaU/WX4bhMuC5 gzNtENyg3WuRybX+PI/CGpfRzYUHWUTLYbfxRLnOeRvLerya8ysZSfT+7WvMFV7T90+z750sqYG3A b4MHEXbzUPsJ9fBQT7NcfJpHJWTeQ3wIetoOmdDNkq7UmpTMnZEVSRVleepgg4leblV2uybKGe/0C ayQ55C7SfcOCO4ugtmRbcoxcyECZNf8umMNJxtzF/+TrXQ70G65d85JJHX0o9a6a79yQj7yRggkmG fnJMYBOA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yrS-3QsR; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 09/18] mm: Handle large folios in free_unref_folios() Date: Tue, 27 Feb 2024 17:42:43 +0000 Message-ID: <20240227174254.710559-10-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: rx8yspgz61pfksjdy49o9eifwcd78afg X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1DCAE14000D X-HE-Tag: 1709055779-190555 X-HE-Meta: U2FsdGVkX1+usMfg5MoER55PGEYwB+aARee6IBVcsy/jfgtWKQxuSZ1pdPBi70kchBQ86Z4f8D2h1dSXG0ezPE4n/jcWokKJhzN72sMNNjfRwETm+/BNwA4WhRy0fmiINUbp6xc27iOhTWqVXIc6ZUZ6aoudg2YBT5+AW4nkSXYONHSy0h/Nm+qiUVkUHnsdcI51BzGGKJTn2z2fwLzeGFdWydGDYOKERWL5gkGhXj6RVHQYzG/E5BcQdF3Jb7QBDGU+t/WfdYVcZYwa1QV1IzPOBLT0y7O4EwchwN7+oyCHzgkTtMApYsQ9Jw4G1LyZ3bYQdju8QlXLuce3ZwsksMTGznbJsI/oJ28R143e+twp4MPjeHfCGt2reWD9gPKXkUJ+bsDWuHZSolTMo4l3nN3xjSl8s7jN22q5eDCBmziIIvGuX5p7viGgIxy8qdqlj14/wPBMm3fQUr3j1MzNIKBJO/HSRX6uYEISbHRpkNB5zagmWPLL1CgG42zGXUmLl+xrBb7MVPjV7f0xr2E/yl+RQIiMQgOddok26LxN2A/g/17M2IQIGZDXId1Ac4+t6mOa+7g6SwO2S5mfSpWqKhUVhf/NkG9+p3EEa+qZMv9+/etNn+nPSkRQnlYojTcRcKW/NIdixmM0uzpJJU0gugFsGJS+K1R3bdjrCXeNtEc21A9GyTVJmN5QZAnzxfNhWYY7VPVE2NoAz4JaztyI0KML+itvRMDA2zt482FhNeaUYG22tTYPpOUSjjgQqPAQyibgu/8CkheOnw+a/WOHHhr4QC44j5ovr/J96bRD6ej/AyxQPKwbrdzQxfOj7I4/pkIgHhiHeFRB+Nt3THUJBwDneJON2s29V76cjOWwzdCPLyfKhMpCL1BeX5XkHoyZQsAeqPo10skyA3fJQ75uzdBtiFILdJ5s X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Call folio_undo_large_rmappable() if needed. free_unref_page_prepare() destroys the ability to call folio_order(), so stash the order in folio->private for the benefit of the second loop. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ff8759a69221..aa7026d81d07 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2552,7 +2552,7 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a batch of 0-order pages + * Free a batch of folios */ void free_unref_folios(struct folio_batch *folios) { @@ -2565,19 +2565,25 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) + unsigned int order = folio_order(folio); + + if (order > 0 && folio_test_large_rmappable(folio)) + folio_undo_large_rmappable(folio); + if (!free_unref_page_prepare(&folio->page, pfn, order)) continue; /* - * Free isolated folios directly to the allocator, see - * comment in free_unref_page. + * Free isolated folios and orders not handled on the PCP + * directly to the allocator, see comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); - if (unlikely(is_migrate_isolate(migratetype))) { + if (!pcp_allowed_order(order) || + is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, - 0, migratetype, FPI_NONE); + order, migratetype, FPI_NONE); continue; } + folio->private = (void *)(unsigned long)order; if (j != i) folios->folios[j] = folio; j++; @@ -2587,7 +2593,9 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned int order = (unsigned long)folio->private; + folio->private = NULL; migratetype = get_pcppage_migratetype(&folio->page); /* Different zone requires a different pcp lock */ @@ -2606,7 +2614,7 @@ void free_unref_folios(struct folio_batch *folios) if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, - folio_pfn(folio), 0, + folio_pfn(folio), order, migratetype, FPI_NONE); locked_zone = NULL; continue; @@ -2622,7 +2630,8 @@ void free_unref_folios(struct folio_batch *folios) migratetype = MIGRATE_MOVABLE; trace_mm_page_free_batched(&folio->page); - free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, + order); } if (pcp) { From patchwork Tue Feb 27 17:42:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25F07C5478C for ; Tue, 27 Feb 2024 17:43:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 597686B0093; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A80F6B0085; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A0826B00E6; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EB5816B0093 for ; Tue, 27 Feb 2024 12:43:03 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AD6831A0BAD for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.06.2995677 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 3F92A2001C for ; Tue, 27 Feb 2024 17:43:02 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=guqNcAQk; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x+eQ503sIMKKAZzDxsNw3aS/Qi24OUnLRvTeEgt3A14=; b=OYXM69aKsGUhaUp3q86Vga3E0ZLgi52rB2Lx4xdYSl0cLN7GnTgjcNM+aiT9IiE7qBUhn2 1BLbGuTM6N1UqZOVZanRreCmIskA4kKaC7YIvXS/42Z06ZHK0sfwatAryGIQz3rIr9eMRg p3KNFHKeTrVzUmaBhl5pY91vTyJNr7w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=FVSki9zYT4MdcG7WytVslahf3XvKZ/2Pf9QMIBm6iv9F4Pe7DOaiyix6B6rH2QJSPRB7RG OsYkIsivB1TXlnPUq1+IBiVtw0KcFERvk1qcf7s7Ye/XSjdRFmKgFsot1KtWfKh+FNIiu4 hv3NgrqgcN5ar1rMZQWsWTRskJiSHw4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=guqNcAQk; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=x+eQ503sIMKKAZzDxsNw3aS/Qi24OUnLRvTeEgt3A14=; b=guqNcAQkawG8Ae7WxtinRyKPDU d/Rj8Xa4wMT/oDuR/4+tR06XbLCzg3jcG3/E5+zaZXybz+icPVeJEFJ1yzT8dbzszn9p9FQvwPvNj O94gu3teXW84eiPimajTIrleFrRRpglo5MrvyYXX/9dp0lTEP1VINgv4X3dqfTMg1yTJmbqV7mhG7 QVFKapK14xTHObrXDHtsshadU8cZSsL93O4xbaOCEfqQ4yAQU34JjFeT0fy9L0OTqQXpc6Vndisqt CNObFeJN/RiFBgID0ojLgFE76YC9VVCZuyMTGifkLMZNtG/M0tPlF0jNOY52R2cbHRdzCRqVkLKVo rCQZAcrQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yrY-3rKI; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 10/18] mm: Allow non-hugetlb large folios to be batch processed Date: Tue, 27 Feb 2024 17:42:44 +0000 Message-ID: <20240227174254.710559-11-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 7hifbdw64otg9nx5y9uptxsfagp3nn3d X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3F92A2001C X-Rspam-User: X-HE-Tag: 1709055782-739145 X-HE-Meta: U2FsdGVkX1/ah+tg1Vwzn3DvsQ49rUhufGWjHtGKW0tNe9cOG/fbcT9L1Znztje4fj4Yl7ps0AnfwL4Q98qjfNdppl0pklBHLrlt4zcfGEgWPlUzyx6x8kkqLXX4HpeJD40Sky98PzJS6TNn6RGVTbFR228P7/3h/uOn7MAfUNa5aIDsaUx2jCaknMR/3wtspZSCLwMvTDmN+pUskoMkzGYLHhNEzy01RiL77GbQdGtVb5zctipdAuK9U15snrCidBLPj5lZlVet5FR1zMLlybDEo7irLATBZFWYMM8YiQrPy40XNncODoVeKBhYZZrxKrR0QY/SRg+BccZ8Z6kmqeWzJnqnVW15kBd2oiVJ0wrCnidNLi5nc2D4hg8K0iQTLLaxPoMyxy8L/GjcGKP2NelrlbETTGMzkwcFqjKnAJGdnCN4teyggSDi6ONw0I5xSB85L19YpuX/fkgRI3ODx9Kj7j7XbTd9EJBjwQ+qmKOZTu9AF+NoqlOhlt+9OegygIawYx0XM0rl3Ov2fo/YNsbdxi8et9ixvbG1vClOXNM7zwnoeCdTNl4aAlXXO591zI6fJwhW5UB8zAPcf584HXEYJ8LlQ3VFRN5nPZBlHUOP4hQFYIeQSQ5DUFkqSrgn/qNBqKesG9EHRU/ucX6enfq9Pmo9Q8BFr0btRl5/WIxYB0NRkEMnKX1a7XBM+Gb6DhqAc6+CmreN4N+TghAJ3Q1xYnTa1sLPxjmsIiotgrg+KudiFfwiG/Rv1jWo8TON2+Bc7ZbkrLUy/m77iffc229kVhiQlIyNXOLmYG/bbgmcwi6z5DmjtIs+qjQcyqeltPKNfbiSABS7ZB7EeSDO5jaEmc4f7rWcVuUWMxGAVO3GwjgeXFA/yg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hugetlb folios still get special treatment, but normal large folios can now be freed by free_unref_folios(). This should have a reasonable performance impact, TBD. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/swap.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index dce5ea67ae05..6b697d33fa5b 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1003,12 +1003,13 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) if (!folio_ref_sub_and_test(folio, nr_refs)) continue; - if (folio_test_large(folio)) { + /* hugetlb has its own memcg */ + if (folio_test_hugetlb(folio)) { if (lruvec) { unlock_page_lruvec_irqrestore(lruvec, flags); lruvec = NULL; } - __folio_put_large(folio); + free_huge_folio(folio); continue; } From patchwork Tue Feb 27 17:42:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6146C54798 for ; Tue, 27 Feb 2024 17:43:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D3B66B00EB; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 11B316B00EC; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3FE96B00F0; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4BFA06B00E7 for ; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E4A731A0C97 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.04.28A133B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 65A00180028 for ; Tue, 27 Feb 2024 17:43:02 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ggDylXov; dmarc=none; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ga6KYsObfe/VGxSB5IYxXXQ8b3liJ4Nmvd2PEithDXA=; b=3nmFek8zT6QcvYNHTTp4fKHIkDNqCvd077dRYOpMAHje+P500U2aD7g65DUSaCwSstHHr3 NhyOzufomPJnEnrD5fmtlg0yzN7UE2Yz481KWh+fCJKxJKqpz3jDWxeNELEFyEfL7tM+wj cBlzj/WOPjydBb+szsVHFoxsC1aEDqM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ggDylXov; dmarc=none; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=bSnz4tpnI852FDYNVDPxAN352EkQoUpcB/+NUWFiXqQJVaQR6kpglphdOwNoUBpkFchzvY I0NG7nkh0MCYdtoWP0TgxrYqgQaDLp8dJEqWbVh6JcAOrWJtLdQGy6MVLuGPhXGWMmM7Es z7CyD90GhExYlkBY2QcaVXGjc7xpWNo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ga6KYsObfe/VGxSB5IYxXXQ8b3liJ4Nmvd2PEithDXA=; b=ggDylXovC1rEQMSlqMxuv1t8SK LHOHViTEyr3CFrNJ5AWSkIQ8/ZXkZ8FlquuktIsxdImpEgPj9q3OMfwVamYOGLuTLvXDiosDXB/xr 55gh5TmU3USZAb/3Wi26clzbmOFKR/G3bzNtKwPkKtiAB35gFDYidzqBVEEFcr/dgbhCq8LrZLwM2 4EcjwSg+9WyINRUg9CakMWUJv8Z0b8ZsYl+y5UqaIAijItBe28YK4H6tEX9urt3OBcpuRbjHuUoJA JKD/87LjxaFxcTx8izgUznW6iAk6quS9N54kYgKWo2K7/JMH8/T+b11Gejo9/f33BFfCyXzpyKbyw QBEBruqg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002yre-0A0L; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Mel Gorman Subject: [PATCH v3 11/18] mm: Free folios in a batch in shrink_folio_list() Date: Tue, 27 Feb 2024 17:42:45 +0000 Message-ID: <20240227174254.710559-12-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 65A00180028 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 5fbf3s7q418qmhudztx8x3t91thy4ue1 X-HE-Tag: 1709055782-592276 X-HE-Meta: U2FsdGVkX18Ka2hTIGO7z+9Dr0GfdDajJM+k12f5Hl0doXQVV7r3AvzSTdeVcSRSw20MVwyIPmbTzWMwFcG8abivLupjxpOqvXoZ8gIgiK7Iu6f1VLkN0c3x8/RyJME5S1sPRjR0BzzknS+zVhjCYEBg29BwJ5jP9OVz0MasY0l5/0TU9mJzrVZUqmNrYi/zD5ZVwbiLtL53Wyptxsm13upA3wLn9VOn9ANzh9xbBk3NqxBJG1yZr299Zw8AmrfAa2GbriWm8vkCFXmuoI6fa5FZnHHcfg7rvl9aDPfwsrFqdipjAreyVApFQECSFs4gRA1c1zkPlGlr7l04IoMqOyZ4AASE75X58xTey4ZYR3kY//vH0CmEpTw8H3ftvWMtxhpv4qUjfovvMXVPjhR8EVXTk+uC6RjOglVDtOi9mqHctbOOu8j1pp2gXDuw0P4zdn59G8h0jXgGohHHymKFQzeKOXbKHiteY81Ub5WIx/aSd2YV/6iSSkFlODFh2iA2Z/o4OOc6cUsIBEH0XJUb7svZ8Kvpf/BDqnCy2eNeUDuZfkaIV3F0jSoFZRRYLuArEasYU450NtfE8j+UlRQQCEIQ8BjaKQonCkm9SsEWArakLQM7BQK7aC2IKU9Ophv1dqoQtKJxd8jj65no78qzt9ABusdU5vEccIFB8ecF0PEMxIVNnR5FWM1D0Rcwyc/yf9xclA9dz3tbU+GdOvg3ud9REGQ5Hh8ZQ20dhaQobwS6/HiaJfgxKEPC3OhP/e27WJ/pEIyr8jPO8qu1JpSAIKEtMyQOzufhiOlRVAQ9LOGOWPLaEiq1w9+QU0Xk3x3KHegtEDsMTgXHXyqp07g4xcntn3hmMduUvcBIXqY9wqA2agN7OUBOzhbL0/LxUGa1/Jj56NcKYoQl/rP87XYrYnWlXYEdCLxG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use free_unref_page_batch() to free the folios. This may increase the number of IPIs from calling try_to_unmap_flush() more often, but that's going to be very workload-dependent. It may even reduce the number of IPIs as we now batch-free large folios instead of freeing them one at a time. Signed-off-by: Matthew Wilcox (Oracle) Cc: Mel Gorman --- mm/vmscan.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d3c6e84475b9..0c88cb23cc40 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1026,14 +1026,15 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct pglist_data *pgdat, struct scan_control *sc, struct reclaim_stat *stat, bool ignore_references) { + struct folio_batch free_folios; LIST_HEAD(ret_folios); - LIST_HEAD(free_folios); LIST_HEAD(demote_folios); unsigned int nr_reclaimed = 0; unsigned int pgactivate = 0; bool do_demote_pass; struct swap_iocb *plug = NULL; + folio_batch_init(&free_folios); memset(stat, 0, sizeof(*stat)); cond_resched(); do_demote_pass = can_demote(pgdat->node_id, sc); @@ -1432,14 +1433,11 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, */ nr_reclaimed += nr_pages; - /* - * Is there need to periodically free_folio_list? It would - * appear not as the counts should be low - */ - if (unlikely(folio_test_large(folio))) - destroy_large_folio(folio); - else - list_add(&folio->lru, &free_folios); + if (folio_batch_add(&free_folios, folio) == 0) { + mem_cgroup_uncharge_folios(&free_folios); + try_to_unmap_flush(); + free_unref_folios(&free_folios); + } continue; activate_locked_split: @@ -1503,9 +1501,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; - mem_cgroup_uncharge_list(&free_folios); + mem_cgroup_uncharge_folios(&free_folios); try_to_unmap_flush(); - free_unref_page_list(&free_folios); + free_unref_folios(&free_folios); list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate); From patchwork Tue Feb 27 17:42:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40822C54798 for ; Tue, 27 Feb 2024 17:43:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B3A56B0101; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C5B4940013; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 276806B00FA; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E16286B00F7 for ; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 48F99140962 for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) X-FDA: 81838304730.26.F14E2A6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id DC3F64000F for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hNus+nOY; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8ClF8cOCV2qWtW1lr6Cu4rAKz4XJquX7C1JC7chVGGM=; b=VDbJE9culqkt3CfU31gzQdF2rc4Lz4pVIBPiK+N5KfDzVcq6PGD9dKCqCu3+ZnvFUCOlbQ AuJ/vYnp2artpIBFaNC89CONou64M9qBBp9hy1hLRS+NWpvM3vsd2u/MuGWl/woFt3omkP ZFPkEzzMKkehLf23kZy+NJMM/E2SkhQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hNus+nOY; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055784; a=rsa-sha256; cv=none; b=U2VsNJWAekL4ObnaJHz+ehtXc1Yw4wDYH8pHsjj+o7JNMo09zh4x63x8irBBV6Viq9hS+J O7tNrxgkub1kiJN+Fa+OE7FtxDLCZlfqsA8yNqnDTbOQctcCCR4x7s036gkD0GuV+rSuq6 TQ8lB6K/CFYQCuvNxjUCoijXRCqQGoA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8ClF8cOCV2qWtW1lr6Cu4rAKz4XJquX7C1JC7chVGGM=; b=hNus+nOYLvgBoVAtnbFN9m/+eD c+YOj98yR+0TfCdEeJzbmwsJ8APrBrPuu1utJAJ8+ScYCqrQFEseONSWvwe1n12qK3QxvaDIjukDR JxTk1BW+P1yhps/2vzt1RuHp25NtdLuR1F0D/yYFTFPQYWcWrFSPt6qhti5G3xWsTpIrIcblO0xGz mGBAti+s+YwqBJkIU7Jxfabav0q7hW6dXO70prH/xHtVbLxnWmH5Jkt5Jf+KdPug8l7sNx7q3PW16 aDvuNdgUl/Z6MHsMqA9EQqj578cB06M+Epw9V9cJxO7YkZWJtKFxeCOhATa5WJ8gu92q97Rze66zC c3sAwmxg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002yrm-0j14; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 12/18] mm: Free folios directly in move_folios_to_lru() Date: Tue, 27 Feb 2024 17:42:46 +0000 Message-ID: <20240227174254.710559-13-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: DC3F64000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: b6jhamr8ykaexo6xo38jeqf11w8auwxu X-HE-Tag: 1709055783-519772 X-HE-Meta: U2FsdGVkX19tU4Z7UcmJ1iNYQ+lB6FugB4XgCfUWUXfeiuXRrhE5zF1DzyhchXznqgFGtYeJZ6C2e0+0NIyC1znZYXq6R4QDGuO+btPzpfyCWr4c+CME3v8UButiUiwmbcJdmCvMEAIi+jwjrDWYr99f/AWU9BiIpN84YsrCwg2ZwQph1MpF+i48GyU2+Znd18nNhOdTlvTDyVoIk4B8xjBdAOyRnxcxpjzo2NlnV27UVflOOcLvPdlhgudp++gYtq8/BLOxZm+wFcf6sy+jM+94cd4L/isWUl8zvXQayDmwugx72uf62OaB+mIuawq6RReCwOK11MtkjkW/AfT/tME1A48KOV2Pqjp9YW8EDFpqBm/TuK15+u55wKIDeZQTwgwYGlXveeRjkWz+BqcZ0h8CCBjh9MghP4H58ojF0dC73b2bt/UdlChjiCAJ91wiDiLEonf5PEjyeuhUNiO5KyplzTnmXM+2EZORG1sw75HNV4VJQZtCvjnG2yulxvbay1z6uKs8+RpWJLklKs+vWFPZ1xzICqDBatkI4lUGKvlwAZFQaaMx+Ld+mxC0dN7kdEvbl3kfxw+zcHx8rooSoEboFxVhi164A8JkH99zEbJV5RHgi8JJKXKoeNu5cHpTrtKEOIU1BtvHEDqjVmCEBeJ9k8Xfg+AJdfooEBHL0m2ywKLmuF4ayNffEDWgcZX+BIA+mgNvoAerrYuMMiL1la09UCqim9UuytZs8oUNDp/f3ouIydeuo2539wsT0jj9eSH2K3FLnOSndsA66j70gkjMqtugktEFCwYrPGcmuDuq7LHW+eQdqbDgLBGzuRQOF0nFiyxvVAU7D0pcxOY7ccsU0EiirXFIyF7rQOU0Ef9cWLTmF13zmjxI7HE0NHdVO/CfdH4AFkFIFSKeW2C0WRBx130mQ/0H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The few folios which can't be moved to the LRU list (because their refcount dropped to zero) used to be returned to the caller to dispose of. Make this simpler to call by freeing the folios directly through free_unref_folios(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0c88cb23cc40..c86c4694bcb1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1801,7 +1801,6 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file, /* * move_folios_to_lru() moves folios from private @list to appropriate LRU list. - * On return, @list is reused as a list of folios to be freed by the caller. * * Returns the number of pages moved to the given lruvec. */ @@ -1809,8 +1808,9 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, struct list_head *list) { int nr_pages, nr_moved = 0; - LIST_HEAD(folios_to_free); + struct folio_batch free_folios; + folio_batch_init(&free_folios); while (!list_empty(list)) { struct folio *folio = lru_to_folio(list); @@ -1839,12 +1839,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, if (unlikely(folio_put_testzero(folio))) { __folio_clear_lru_flags(folio); - if (unlikely(folio_test_large(folio))) { + if (folio_batch_add(&free_folios, folio) == 0) { spin_unlock_irq(&lruvec->lru_lock); - destroy_large_folio(folio); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); spin_lock_irq(&lruvec->lru_lock); - } else - list_add(&folio->lru, &folios_to_free); + } continue; } @@ -1861,10 +1861,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&folios_to_free, list); + if (free_folios.nr) { + spin_unlock_irq(&lruvec->lru_lock); + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; } @@ -1943,8 +1945,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - mem_cgroup_uncharge_list(&folio_list); - free_unref_page_list(&folio_list); /* * If dirty folios are scanned that are not queued for IO, it @@ -2085,8 +2085,6 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate = move_folios_to_lru(lruvec, &l_active); nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); - /* Keep all free folios in l_active list */ - list_splice(&l_inactive, &l_active); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); @@ -2096,8 +2094,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) lru_note_cost(lruvec, file, 0, nr_rotated); - mem_cgroup_uncharge_list(&l_active); - free_unref_page_list(&l_active); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -4601,10 +4597,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap spin_unlock_irq(&lruvec->lru_lock); - mem_cgroup_uncharge_list(&list); - free_unref_page_list(&list); - - INIT_LIST_HEAD(&list); list_splice_init(&clean, &list); if (!list_empty(&list)) { From patchwork Tue Feb 27 17:42:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8870C5478C for ; Tue, 27 Feb 2024 17:43:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B22B36B00F0; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E07F6B00F7; Tue, 27 Feb 2024 12:43:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1E3B6B00F6; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D24066B00E5 for ; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 77F3D1C0DCC for ; Tue, 27 Feb 2024 17:43:04 +0000 (UTC) X-FDA: 81838304688.03.2DF4790 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id A754E100012 for ; Tue, 27 Feb 2024 17:43:00 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SzY+XFkr; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=2UnI21JvN495mLUvOXUNZL8NSXgU6TXN6Rd16ydd/lwmmIXXSFBoCSz+a8DRShag2DiYVI 639x/q/0bjGl0qMf8Kjel5sFR8aToAHxzAgKLLiKpGvbICk1OSAhdLs4FC7DLU+FK+Blby 7ZvkjdcImxis2Qyc5LzJUTXi888x1oI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SzY+XFkr; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U2EQset2B1F0v5WlVxF1O6/RQfFktgSfImLNMKOn+UI=; b=Abq9r14wJk0ymGwSF1klpPyJZTL3lqpyL3r71fdBahcPlrQQE36FnrBgWkqInUaTCg9KyQ 2mERyftCw3waZVEJAimjdmo3sLBF59ao3BGZYAvI3duSnI6UMYK0m4lGrM4dx3HGfRRviB CaDiHqGVkzc7JH76kThADlUBbw2gqz4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=U2EQset2B1F0v5WlVxF1O6/RQfFktgSfImLNMKOn+UI=; b=SzY+XFkrbf1OmJxVqKyejogrQI Fa5PvlfPXr0eSyeB9H9YNOzgtRNCFkDt9Fz/ZLBgDYXCDMppG9L5+BEkzPFvB+wO6mqoOYON0Knw5 otrc8cMHXJI/8T6M2jcZQ0yWSTEdsor8A6HPSpzsZBcagoVj9x6twuWae5Gl3AuaGrlUPWAPNUeQy jl2hfRkkANfTWWp6lwYefQG//zzW+M4I1wTf2Q3LovcErS2+DwAmXtv5QMad9cZ0uhp/+W7eu3XYe f8QQ2D/X3Clgabhab/kzMJWd8a4CCMQ7zw2l6jkrlfql3Op0T9CHlMB+z6q17cd3vqZdgTCYuzgmd J57hyjnA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002yro-14tX; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 13/18] memcg: Remove mem_cgroup_uncharge_list() Date: Tue, 27 Feb 2024 17:42:47 +0000 Message-ID: <20240227174254.710559-14-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A754E100012 X-Stat-Signature: imn1kknibux68ynqcc6fkmaq3cm55m5o X-Rspam-User: X-HE-Tag: 1709055780-635306 X-HE-Meta: U2FsdGVkX1+GwkTzv5MWPdNWok6UOqPmkmbE1uhJYKx54+LKIEoInlZukr/mhPi9rh0vM+HYLQMB1owG2yvTTAUHP9nRruqhVPdstPDIaUOmWcsW7qHsmIsyFu8V+t4yKenFN+xETCHlkEsHTtYCJrwrNCwyNJNOi6ofUrbG9rcC4IJIyv/tHSJmxH3dS57+eBbSQsGnC79PjEl+maoj346eII7PFrFk1HSLskSZlrqBostEW7oukXKQ1J/uJYOGTTTspvKHNaV8nJmLND8tN2kHsNduJUWTc2IijsMflFh6/0f4BVMFysH6aNujBbv8kJt8/CRKuY/kS/y6WX797Mvi7soIvFf5YM+gCBjnwSLMkKxEM68eyXJSoM50XNfN8BO0sLdHpkvM4U/K+aADxv5SKwClXLMGA1vux5vEbOxKFZioqFhLrjirJu9QgVSs7o+7W2XXFgEULrtN0t2X+op5hhoBXlFYHwIAjBhsizQNche/BZL/K/ATqZnWMZGVMUGQm/fEnE4FaZqPbptmRgddX6dNLHpfPTKzr55nKcBLxA1jlkmEMxOK9J/6ngLgjAh4nIAZWLFkBJkrhGuSrj81XqTc/McE2hGYSwKcs28bTT4igLUDO0Xxp1da6eT+8zX15yT7kxAmRApDwz5/m6moNHW5jPfQNNyfx2Bj6XZ0vsH3wAmc6SOPg6+yTbUXkMC+r/kZN3IKuQGIyahtnCcTzvblZrnrl+Gb3aHZAlvRTVJzYVBzCrZ9E64uTneB7oqlPsdbNESWJhYumSS1fbcUlUV1QFDSBn29IdYcMqF1EDb5SH2ueCwFuk/pGw+fj3K15RryyyTM5zY19h4TuZz/TOOMgxYpKJoAPzdTRGj9NORVOrC5Go6Y3WrwG6Vs0DEVKfUcfXx7muKDNoZMdjg8m4bUyTfYiYSCGv5rZOxPz1dalGcnqrTYpdtIgWOHOuuXSuCTxbD9lO3b8Kp +V7JO5/2 1jGDV1xOScSoT/RQDOOYcMs5YwNxrQJxE53rH2mev0YMuibW1eancd9dPmw1Rxuc4r8nR1ChzrvSMRTq5thH3IWrw7QAvTsfo0Q042QuplLVo9QLuyJ/6YLC5mNEyMKwN2jzwehimxDPEQSZvg39PbAO7+I1wnj477MKlrtGwk9O6AR/aVE5okpWOrLtvzwMZu4Q3yu7RclE/oqZN6bWRDbka7zlvvboIHHyvgF/Oi/ZCL4nKGzeSahzrA1U1/HRIc8o/c9zSiIFqxEYRVFGHpScNBRdglV/rrZpE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users have been converted to mem_cgroup_uncharge_folios() so we can remove this API. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- include/linux/memcontrol.h | 12 ------------ mm/memcontrol.c | 19 ------------------- 2 files changed, 31 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8a0e8972a3d3..6ed0c54a3773 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -713,14 +713,6 @@ static inline void mem_cgroup_uncharge(struct folio *folio) __mem_cgroup_uncharge(folio); } -void __mem_cgroup_uncharge_list(struct list_head *page_list); -static inline void mem_cgroup_uncharge_list(struct list_head *page_list) -{ - if (mem_cgroup_disabled()) - return; - __mem_cgroup_uncharge_list(page_list); -} - void __mem_cgroup_uncharge_folios(struct folio_batch *folios); static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) { @@ -1301,10 +1293,6 @@ static inline void mem_cgroup_uncharge(struct folio *folio) { } -static inline void mem_cgroup_uncharge_list(struct list_head *page_list) -{ -} - static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4be37c9a0759..22db1760e9bb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7546,25 +7546,6 @@ void __mem_cgroup_uncharge(struct folio *folio) uncharge_batch(&ug); } -/** - * __mem_cgroup_uncharge_list - uncharge a list of page - * @page_list: list of pages to uncharge - * - * Uncharge a list of pages previously charged with - * __mem_cgroup_charge(). - */ -void __mem_cgroup_uncharge_list(struct list_head *page_list) -{ - struct uncharge_gather ug; - struct folio *folio; - - uncharge_gather_clear(&ug); - list_for_each_entry(folio, page_list, lru) - uncharge_folio(folio, &ug); - if (ug.memcg) - uncharge_batch(&ug); -} - void __mem_cgroup_uncharge_folios(struct folio_batch *folios) { struct uncharge_gather ug; From patchwork Tue Feb 27 17:42:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94D38C54798 for ; Tue, 27 Feb 2024 17:43:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AB566B0105; Tue, 27 Feb 2024 12:43:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED3F86B0108; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 840806B00FE; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 086066B00FE for ; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 88909140AC8 for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) X-FDA: 81838304730.19.6B63AE4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 036C818001F for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Z2r+qljn; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Py6+s19ldATppZRQG9f/uEiEQuQwyRI9PUiiL1ghTdQ=; b=eLtg2NMrzLgXOdeT3fxvEomjQ7Q3wSFTjhHOoy6Vn1W9JR1i0b5rcW+IN65J9G6gQMHr+U OX3NijFbctlbQtpG+l6c9l99/WkCLoInJq5oJY326XqIYT7Cf/kZi00lCtTbn5y0RFGaNN SYG9F6D/Wal+efolR5mljcvyb/+Ndhw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055784; a=rsa-sha256; cv=none; b=kNM03ZlFs4KBx3KMiCSrJKaiT4wvqnv2Jy+5/iqF74rQIvx+piPspMzkiSTOSY5shJI+T+ TSR5rR6bgFGqh4HmHbotCIwi4yIGmLfE5Iw8lBHBv8KuWvrGYGIZ754QJUzNXg/JnKXjQ9 ndXXpy4tcq6RyLWxKejIcbJD0mUjCGs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Z2r+qljn; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Py6+s19ldATppZRQG9f/uEiEQuQwyRI9PUiiL1ghTdQ=; b=Z2r+qljnJGeWWGMNsO6CycXZNe ahhEPG2WD7F4Nm6OUMGayGiLJCSl26yrgnH9t2HjgwTVvssfXKkDkGdIwudYriGXGWeKbNIvBk3g7 ZTZsu5adl+mP7QYgeTUo4cTcXuE/A98xtkx+Q0sytXcpHjZXNYLqcNGDpLll8MIBD9k7GFzzkcJlw 1NdUflVllPnyWVG0yYtiUWI8Nps6Pp01cq+o1Bkdg+S0xt/Su/7hlpRMu8DiA5YVsojg+gvGezMbj Na/xNIZcHtBz3W+JKcFCsybbu3ne7yLr2Ezwul/ktMnuCt7ej5tuTMqpBBbBxABfC7qnOxtydRH+1 e7zXtpww==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002yrx-1rLP; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts Subject: [PATCH v3 14/18] mm: Remove free_unref_page_list() Date: Tue, 27 Feb 2024 17:42:48 +0000 Message-ID: <20240227174254.710559-15-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 036C818001F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: kcdjtrz9hfwgioxqkgs77xjzjynypsr8 X-HE-Tag: 1709055783-707896 X-HE-Meta: U2FsdGVkX19dGv8IPmxRKc115kkoCQJGjIMXR0LBET9ypAyhVjEVM9TPQIJk6EgFJIITJtRAOb4zLgj47B387HaWjZ8VsQyw/Gm+LLkUGUZeij2DgtADZLUfxDKk4behOvbWh2SP93vJFHwv7bGy1vCX4WZoA6Y7hmXZO3BQ7v1qdgvN4ssdKpgut2SoJ017YP+pv2ofsZdaOjhVeyaegYPtsPB7cFiwQWbweF3APMhA2ZUZ/xuy9l1ghUabUBHjQ+ysipCwxGFKggF5nlUytZ+kyVWFeqFZCELBPXwQzrXWeyocO3ey2Du8yDOB3f7d7qrLtYuRnCn9U066qrtYqwoLiFMEZWe+Q7vML19mhHUL8qNvnRb65pCfB0t7wlnwExNP31NRc7E6vjrKYl5lI/85j+gxQs9EsiYAbCL7sWhhNQ8wrDX6VQ634uRglLTUSWvr1mwXjXEhDrEElRqzetb4WRy2mIC6dLzDAEZw5uuRc2bEj3im40sN4+enWZg7VNMy9Uvox7r6811S58d+u8Hz5ZV/r4fXcQLNwtdXST8VHqYua17rCoQLVxld3TeLvj9r4iGFqWWGyqgIqpgJJ4n8efXOzq6xQCWxf5BS1MY3E17Mzee1y7vws3UQRZE+QxLfC8Q8rgJjgh4FEhG1SDBlJaUEpDRjABZWboRFTFSGV479LJ9UZBzE+8j2ZOKPU0Z/yHQw8Dt3Eq7/4wdEVEvG/DEeiaXr8s4O4nqybP/kJB9KQ7EAAG5qELIkadVB8FLvOImtNKRkVnhZ78lBwSNmFgCo9eEWz0WNRoBOXH5xoOUEW57h10qocf1Fxxilqi4mjZ5NsBGVaHutjiDVt3wcFQNJB9ZdurZCtSpW2gr00L/h55ckSjfLjZICEeC3LyLgtaZ9e5ToCVau+fuoV4YWECZUlIjO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers now use free_unref_folios() so we can delete this function. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts --- mm/internal.h | 1 - mm/page_alloc.c | 18 ------------------ 2 files changed, 19 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 3ca7e9d45b33..cc91830f6eae 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -454,7 +454,6 @@ extern int user_min_free_kbytes; void free_unref_page(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); -void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index aa7026d81d07..01b60769726e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2641,24 +2641,6 @@ void free_unref_folios(struct folio_batch *folios) folio_batch_reinit(folios); } -void free_unref_page_list(struct list_head *list) -{ - struct folio_batch fbatch; - - folio_batch_init(&fbatch); - while (!list_empty(list)) { - struct folio *folio = list_first_entry(list, struct folio, lru); - - list_del(&folio->lru); - if (folio_batch_add(&fbatch, folio) > 0) - continue; - free_unref_folios(&fbatch); - } - - if (fbatch.nr) - free_unref_folios(&fbatch); -} - /* * split_page takes a non-compound higher-order page, and splits it into * n (1< X-Patchwork-Id: 13574206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4947AC5478C for ; Tue, 27 Feb 2024 17:43:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CA646B00F8; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 13F1C6B00F9; Tue, 27 Feb 2024 12:43:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D63026B00FE; Tue, 27 Feb 2024 12:43:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A6CF76B00F9 for ; Tue, 27 Feb 2024 12:43:05 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7449980D17 for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) X-FDA: 81838304730.21.3897726 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id E9397120011 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KSJfO8IB; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055784; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DZMuXWWK/77uXZcdaTpXjwM3G8/5uKO6ESnfF3LEyZk=; b=JDAjs8GP7JTRmp00uZJ1cdVmK1QBEXt4aLgA/abZ5xjNbhbss05Rz1gATVTTueGj7aKr8C /Gevb6RNLevwJAcBXeAiR9H83vyCHvpSjVGVXOCSxhab2FFXF2cS9wfdamdgoh6WtycyVr 1OTPe68NlJALnPhmMe59B22dyW3aGb0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KSJfO8IB; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055784; a=rsa-sha256; cv=none; b=y/EtXoTqPMihTKKk+1VinR3/7YbmsA3VYXj29w3lkLQUKCgwZDV9Svy29pKU5IFw0F8kor opoR6qADrcVeCOuyHBQrT096D267pfzYEbBMjj0RJKwVezHx5PAnAEMICcAm5pkaE4KFz0 B8sQW7VLtcycysG8S1955JMZOSwRsck= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=DZMuXWWK/77uXZcdaTpXjwM3G8/5uKO6ESnfF3LEyZk=; b=KSJfO8IBc9drl0rx8FR4CTs8Pt hl6H69oJW5Ybf71U7RJGe2M3czxHUSkNcjQCdG4yDs850VVypAlbm5YEPVA9vlbcoQBRaiEVGTb59 qRVAaXTDEmlPGiN4siesn56r2+MFsmrAP7umreKjyTIREensLN4spM91a1/pUhKir4xRKqzv0qvoM 6qABfE1zV/hgG+yMrnwNTyvFKT25RAIs6kDFP67O58x8iFKS/Gz83PjMAqXG6LzCc5ipHMiaNa4Bw 4OeMWIuKP+swnr8BXCIgjFS+XtoMFKMyc47uuIxT/lGiRIKT6ltW8CPAvUs1MD6TgXlCuIoQlQyFD u2ze/TsQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002ys7-2JVY; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 15/18] mm: Remove lru_to_page() Date: Tue, 27 Feb 2024 17:42:49 +0000 Message-ID: <20240227174254.710559-16-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E9397120011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: kqanpsg36bp6zk6pyj4zypk3rsyaw7b1 X-HE-Tag: 1709055783-90964 X-HE-Meta: U2FsdGVkX19qjZjkWaBQDcjELSYkGnr9BacE4ci5syVVnsoJ/cxxOh6+XXdJVIbz9obTDEcpq5n7cje478QWa+rSTuAe7K9q8MlMKoqHK0RgCEtnilLTXIjgGTMBvev9prLv9ohmg/F0oxjkYgLnSm5jksklGQ6K6d/AbN4QeJDleEZ1LQhOMFnNKdJQ9EhAoZk70PE8V7kugQMFT8nmHl8sGhIXWaRmOjzQVFqr2Qni1y9IWsVMp7tMwQmuQjmCdKNcTXWFfAISIUDfTExUknhfcFFhXbg87SLW/q9PBiApRo+fZuxRLG/yTxsyDjdviaqBJGpMKh3hTsEvrwSH6f9UwfchVFP57K/6WE4Jxjx6oaA7iSKZiAeJgiPObWbaNTo51vLbBTkrrMVK1GK0B1bF9Mb5j6envk6VJtzb5xLvqorYz5HP5UYbmv3RYkERMYsTmp7N+l2xIn4uk3q1g4NeUqNC/y6zk6oPtRWJtnSs1zGAhcMEfx6c1Ll9u6O0K6QfhYJgyxjyvPEXcLxUC4p/ulhTswMwdFF5oUq4juGWxk6BKnnNzR8FmrA8d6N0u6dVsXlGV1V+eaImr39rKSpZCDzEzx7mENI4DlfAWeGqKmYm55bRYLCxE48mS7KmLtxoyQU8DE3gJeY+JXNrrZhyAqi4c511UtucAptF0fcI94eOlPkJD+HskScg1yO2v+nbnctbNlVU4puYVjRA5wwsittg++xQKvpugKBH2mUWhJDoj90r1/3HfcVN8zHWAvaUCs/heX42H3QLXYKfZCvSf9YiyZh0d+4HaRq40S/IrWfXKhTTM/P1foCA+4SrMbgtBM6pMotwn3lU8jWF9qsrNPWKagpscCXBwdflhwVE/xhXj3bCXDaHuKWmZB8xckBfv32Zo28F37CtFpptaLqXSi0+Z1pn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The last user was removed over a year ago; remove the definition. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 07d950e63c30..c4a76520c967 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -227,7 +227,6 @@ int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *, /* test whether an address (unsigned long or pointer) is aligned to PAGE_SIZE */ #define PAGE_ALIGNED(addr) IS_ALIGNED((unsigned long)(addr), PAGE_SIZE) -#define lru_to_page(head) (list_entry((head)->prev, struct page, lru)) static inline struct folio *lru_to_folio(struct list_head *head) { return list_entry((head)->prev, struct folio, lru); From patchwork Tue Feb 27 17:42:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 201F6C54798 for ; Tue, 27 Feb 2024 17:43:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AA7A6B00EA; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 24CAC6B0093; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02B2C6B00E8; Tue, 27 Feb 2024 12:43:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DD6476B0085 for ; Tue, 27 Feb 2024 12:43:03 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9BCED1A0905 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.05.3AD6EC3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 3606E4000F for ; Tue, 27 Feb 2024 17:43:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ljcg3YSj; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R3goVY0LruHLbvMFj7bnimSm59kHXIfZY+DKSTKrvLI=; b=VeusWnwBDDeVfqu/9KvB3dLqr/RkbM2v36Z7MH6iX4gYrLBNqpnJBLs3IfGJCqXpGUPL+T +7XoBLKey6FZwsUcSObLdysETPlXU+S7mtA22wvgJxASPyb7NHsmNc1E6mq1SRJk0HpfCs 0i1x7xP8xaByV6CpjH08CedawJ4lLuo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ljcg3YSj; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=xerul7YoHMMBV62qHNYrIh3e0YlK9jfrQtB4unLyqOOAi3yjTjokz3Vi2K7Pl6o8/vTSj2 0r5PANhlRd2GrvZlNf8zyxXQsFZS5n2b5FV72vEqZpzX4zCBPtDvL7scGjU9JZ9wsovJCC lAKXKXhrAkEpP20apXaKWM1zKwbcgf8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=R3goVY0LruHLbvMFj7bnimSm59kHXIfZY+DKSTKrvLI=; b=Ljcg3YSjPx01EbKbyb1zd1vg3e GTWnm5lsNfgpU8I0kgy23UQ9eRiq4hjyT0HrkStJ1w7NdxxMFqCYz2dbyNVc7t/0Gmnhvo2UjNKlo NDGXofldXg8rs9Ks8MAXmsqdTcqGLd4HIIyUdAoYzcRw5lh/tLBsKP2uBDAnihGf/AL0fNdPgYxe7 tk7mYsLP7E50j3dpAdXqvS1ujAhes0ZxjQIf99rOPDUbddCuMrHzUQmMESL/F3UONH6ZHJ5/nHjU2 ewJl4TYn8uhibLqtt4eLDvXJuBiw7QxC92Cjhzkhy1Cmc6/9B11Q9IfwdiLADnk3YYLIZqX7lFc6v bKgfXPWQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002ysG-2sUt; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 16/18] mm: Convert free_pages_and_swap_cache() to use folios_put() Date: Tue, 27 Feb 2024 17:42:50 +0000 Message-ID: <20240227174254.710559-17-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3606E4000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: g6inx4r5m9k8rc6y4w3qqyeu5jds6r7d X-HE-Tag: 1709055780-123412 X-HE-Meta: U2FsdGVkX18bWMPmD1PmMEiaZ0LVY5GwKEmMjeaKRJfKqCT17rH21nzuBrHruMyb29QhpYcg8WqcmR79+UZbhDkpyAG5BW7uwipjw8JmBxBkAI022TG8akY1jJWnBkvUZV8zZHInQtW7yhVrgtvYCNjcl2W07+KXD7abQJasWwrjT19Ab4ULW/DMNuZOXW5DVuvbk9leZCNUBzgBfd4eyhuLHJj444z+4qukVs5x5uCotUPFqnQzP8+WdLHmtEZVx30yECCW2Tr4fV0TkKcfBpfcAbY1VD5/euTfZdHyqf2rkH2SpFuQ5qySUzTLBamElcPNlIsppxBxWF5tTWkxUHgSWyKshtH5USI7lcHF8OW4CmmomvKyy9x6e0XnreTJZCbN3UomwZYNETyVV9QuVeaASwCuV90DAV6c8p1bJWnJh88numBnBzegWB6leLaUa3Q1r85Fyrbt18MkmyF7TcY+PrB0VwgQOeGjFUSo2P1qdtd42bPepAie/blPr2ms0nnasYK6XFuDZQV4+9pHRtRfZ3B4IbRSlZXCbcT/vgExPf2hDXuAuT7inxYY2VAaFA2DuEEtXjwj4rgtdo7DH1sm1Ev0zy8DBlRonnUpA3CUR5yh0fxyp4cuXvXIDsdB49ogcBzYbL1sYMKytev+nTNcoMGZIOXaAPIq9g59JYMJ+vf70fbkF/LzXp/LkpJAuVgWPqc7Z7TA0hPmV5qvMIQrvG5WJPJ7+tBDmGgCG42lhHIv/quEzqIGWT8mSD+5MVktKXUGMLd+GvAMpZQaIEF815wLalfd89Iy9+H+ml6LIKsoAX6H3B5K2LoAN5LIDIngvV+hAOq0RtUI4SizzRszv183h3VHr8vtvCUUvNOVkH/h8/gt5r2bvEG4pXltUVL9KU2QixB7e2A8hZbuK1hovz2ujIY8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Process the pages in batch-sized quantities instead of all-at-once. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 2f540748f7c0..2a73d3bc5d48 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -310,21 +311,25 @@ void free_page_and_swap_cache(struct page *page) */ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) { + struct folio_batch folios; + unsigned int refs[PAGEVEC_SIZE]; + lru_add_drain(); + folio_batch_init(&folios); for (int i = 0; i < nr; i++) { - struct page *page = encoded_page_ptr(pages[i]); + struct folio *folio = page_folio(encoded_page_ptr(pages[i])); - /* - * Skip over the "nr_pages" entry. It's sufficient to call - * free_swap_cache() only once per folio. - */ + free_swap_cache(&folio->page); + refs[folios.nr] = 1; if (unlikely(encoded_page_flags(pages[i]) & ENCODED_PAGE_BIT_NR_PAGES_NEXT)) - i++; - - free_swap_cache(page); + refs[folios.nr] = encoded_nr_pages(pages[++i]); + + if (folio_batch_add(&folios, folio) == 0) + folios_put_refs(&folios, refs); } - release_pages(pages, nr); + if (folios.nr) + folios_put_refs(&folios, refs); } static inline bool swap_use_vma_readahead(void) From patchwork Tue Feb 27 17:42:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A540FC54798 for ; Tue, 27 Feb 2024 17:43:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B03996B00E4; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A8A1B6B00E5; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C6E06B00E6; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id ED8EB6B00E7 for ; Tue, 27 Feb 2024 12:43:03 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CB9114061F for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.06.04AE17F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 4F516120004 for ; Tue, 27 Feb 2024 17:43:01 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DWWvRhmX; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o3omPoxMLsk94BWO3XT1KRKpFrR6WgQB/H8HRcd+FzA=; b=w0xzkliL5KQ1bUZTDgtz7o+ZRPiwep1xkfFqhV1UZQcFRUM8zuVQfSvl+tcjmRs+/aD6P8 fKU8uW/0FH3Efx13kAGfTgndv/FgnmESl5i/n9trqOCGz0UJmztYZeOfT9Drg5wX68KFlF bTKD18ZtfwYxx39vj70EI8peRU30TFg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DWWvRhmX; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=yks94MIFf2CU1mHnqLuHUd+3zUXqM+uaDELYt8ipJ1T6chW0qcrw4+RIs8XOBOcqXZJV6L OsbeUebNjd64crU9TSVwfIEcYOrPC1jathyU2So6uCc8EzITGc4BTlic7W0qBk41qmh1LP 7DvOpdhI2zrdEWZCw3aMATRIgx+LMbA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=o3omPoxMLsk94BWO3XT1KRKpFrR6WgQB/H8HRcd+FzA=; b=DWWvRhmXp5wd0D5DDCBoTMJ3G/ 0kebLqw3WdBr3SFIllqZw1ijZlx8KUItP0v4iuaDs5hJ6yRAfooueQN11p0kg//Vlxy/J4Nc4jw9S acIiBrRKuxosiqP7nlCJKPpYKoxa0/6cQBi6Kfb0bSDdPIxdARG6wFFCzBFKOH8vtpJ7RucUKdiZE LZdzS0EhBvIf9tMyT3ou8Q9qPpYNR/dfpTDJ042Qgam1zlJfi48Zl0EJ13zAIpOY+08Yn4gRs4KWE n+9r1XYOjuoSQIeVyS5LhpFRQRSZffZ15xK8gpgBhYepIIJmwNSVhTRDEbXrt8IRTHmjARNr/cfF2 H8xZOSYA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002ysM-3NW2; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 17/18] mm: Use a folio in __collapse_huge_page_copy_succeeded() Date: Tue, 27 Feb 2024 17:42:51 +0000 Message-ID: <20240227174254.710559-18-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4F516120004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: q73ytyoqxywrxk9j93jizkfu3pj5pn3u X-HE-Tag: 1709055781-289822 X-HE-Meta: U2FsdGVkX1/5/BLrKdohfFw1Wrtf+1Jl9B28+t1wcdZMXHQ+6WlSguI+iisqtJxHKDoOqwRO8HYOQXgLL2I/5DKqroAlDSvEN5baJ+qQexrIKYUhDOmcFqLWQ/plQkE4m3I0Sp8wEuKHmI3sw04dsN5HTVz2wRqu9JO9ZYQxg2HGpWuT3qFhBmPN7iDLmNpksQeT7RCTrHvByUNHkYAv88zqfOYhUMAGn6Dm4SN8l7sUCyUDnIiBdcVB4vvpAMz6Vk4QpFz2E11wEV4TZ33Tjad6A85FwyXJKQTBFVWeXFK9qixbugm8j8XAwW+zKFCHYDSLTg1iAvKPtUF0pb7xRrgd/IZ3wPkC1jC3wPYL/Y1LwsphMTxq9UccSICRbEN7Xe4dQ4zT6lB+zMfUhY464aADleF6Owy4zA46GnEHtpXv6CvnblMN02AelOHg4cY50XNsUBoZa1ITG1oYtNs8QJMD5W9wWamOp/keXQ1KTJA4Ji9ISlnJwdlUhF6QN6VXndiysS4rwFt88DWTUaPon+hIKay5U9+mZkecb6AeLxf/TFiPm0TA3v6YUsyCtNCbkHcYxl1Jkm66IHRVLGIAVonz675UqnloqJ3FHXZN+mxZZnUc7HzC4FIvsu1RizGMan5xvGW+sxh6UKxpyX1wV1WGZyNssz+lA+3xB+hfmiDjfFfrN27j3jfqNEvt/BDkUiA3QQZjvRoGreUFbqPT0NTxPD6VJLr/obhtRGElcerWGIX5MlF3RzAwaoeKSpwkEgunM2zzpdwzbUt6pfN/UIKTkB/pIDf+FjR5kTuSf+0MugXhRWr+ekoxOFdtzw5LA+AxE+kTjkRafye8JVp2EsrsexieM0GFnGH2XYN9kv2qRUub15aN2cSJNmAvZK7SmRfXgl9lh5H4xhUR7iv+7H3a7jEDbV6+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These pages are all chained together through the lru list, so we know they're folios. Use the folio APIs to save three hidden calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2771fc043b3b..5cc39c3f3847 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -689,9 +689,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { - struct folio *src_folio; - struct page *src_page; - struct page *tmp; + struct folio *src, *tmp; pte_t *_pte; pte_t pteval; @@ -710,10 +708,11 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, ksm_might_unmap_zero_page(vma->vm_mm, pteval); } } else { - src_page = pte_page(pteval); - src_folio = page_folio(src_page); - if (!folio_test_large(src_folio)) - release_pte_folio(src_folio); + struct page *src_page = pte_page(pteval); + + src = page_folio(src_page); + if (!folio_test_large(src)) + release_pte_folio(src); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats @@ -721,20 +720,19 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src_folio, src_page, vma); + folio_remove_rmap_pte(src, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } } - list_for_each_entry_safe(src_page, tmp, compound_pagelist, lru) { - list_del(&src_page->lru); - mod_node_page_state(page_pgdat(src_page), - NR_ISOLATED_ANON + page_is_file_lru(src_page), - -compound_nr(src_page)); - unlock_page(src_page); - free_swap_cache(src_page); - putback_lru_page(src_page); + list_for_each_entry_safe(src, tmp, compound_pagelist, lru) { + list_del(&src->lru); + node_stat_sub_folio(src, NR_ISOLATED_ANON + + folio_is_file_lru(src)); + folio_unlock(src); + free_swap_cache(&src->page); + folio_putback_lru(src); } } From patchwork Tue Feb 27 17:42:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13574211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F09AC54798 for ; Tue, 27 Feb 2024 17:43:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22A4F6B00FE; Tue, 27 Feb 2024 12:43:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD6D06B0106; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CFBC6B00FA; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 20AF46B00F7 for ; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EF4731408BF for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.08.E8FD540 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 54B7718001E for ; Tue, 27 Feb 2024 17:43:01 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SXhmA1Ou; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vKMYoOiyUO81OcdzYFKguTw2dj3YSsxMWjG9EzdOdEY=; b=mgEfwdDdBHuxpB0MI0D0PRL01y1I2wy4JSuq16jxq/2CXyDiXmW4zmi8EjmZcKFipC+okW VRcxFhhuWKXitslYpLsLUW6EZJ6SzeJwVjOU6yz89GYuv6gZo2eadW4U6xWHaePV5p+4jN Oo3FMarsixCe8Xg4ySZzP2jpzDx0dD8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=8rJUgbDqi9q6p1guKs5zGoyInBvbBaPriKYuzHLrGVxzPV05SnMdIn6yl/UuTueLKdpIOS NCT95NDMuha3rOCIajxn12T/rn3eKzp3XpZJ6a9Y5Xqkk888X2oweN4Wwwpqq8QFE/7l5D vva+lKURxg4QjGh7UejQP/tVHUuTrjE= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SXhmA1Ou; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vKMYoOiyUO81OcdzYFKguTw2dj3YSsxMWjG9EzdOdEY=; b=SXhmA1OudjsbC85VTPejRTCAcV N3FH16PFA8rWEb2kLxok93WxBI4FKxpXRmRdggNkBLNpOUI+ew+lKF5lvlm8dXt7WgvlcHwuPYMYD QON8OiaN3wIlAqv2rpVLp+F9sFYIXS3AML2iBrMnAJIwD5tZHkP+SIJ0ZU+R/7dHkzDHUWYX8Bro1 QEYUEo1vZTUSTR/2FmMNRQhWPSlxkFaNZ33eGHeH2xdxPceCw/zO2I20Bt//Tq36ct/7KMkhpKpwz pZ35xBWRJn4kFHmAu1C515NGkxG+kibV9SF44pizk91ypjl8nTu+75/lMPejJvCIg7vYPFaqx9RKW bhd351eg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002ysT-41wl; Tue, 27 Feb 2024 17:43:00 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Ryan Roberts , David Hildenbrand Subject: [PATCH v3 18/18] mm: Convert free_swap_cache() to take a folio Date: Tue, 27 Feb 2024 17:42:52 +0000 Message-ID: <20240227174254.710559-19-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 54B7718001E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: czq8p79d5otzsjyf6wgu1izwsnxhf6n7 X-HE-Tag: 1709055781-116276 X-HE-Meta: U2FsdGVkX18nfuMStO0oOeM0hhnJFByOv73eVdJzvnaeboVQSJk369rC9LU/hK+SlBuTHvvlyBvAsFivoNd7RKBJ/YRApddysnItKPDiVlQB5UMx7AYTg7fNdGCsHk4yI4WXkB93PldBu3YnVYA2mGjXiXduUd3YIY23ietGfSoHh/J+oB5QksUBc7MG4f+BgGx5LqczABl7/ljg15Fr442ClPONdX6ZYEVFx0AdO8UN09BWu8WIIATqTXQuEjhvyStwSvBVRlCyC32/N3ZZbdTDb+igPrhbmx7giRtllZb+ZAF1NjJm4x/mIRy8y2CglBJhcZty0q+7gn7138jwYn8BU3N5L6dDH5Jp537vzSiCUE+MCZ6Fy5gUMZMSEHkrW78caBI58//V93yZ8qo4nV1jQdb1JA+dte91ujBAK+ykO9YxfCSrYCN9ZN0EFHCQ5vbhg6ZI/zSp26ho/saxiMFj+iJEfzxwoZ/J5gCY8ErzTll1g0+FVCbdSB687a0ah3UvwB2C83kuoKOQ6bfPHQ/rYyND2vv+yLeduefkoFk6O++2QoI4blkWdfLiOgM8DnZyXmCNZ5+Y+qXB2hH7XS3/NV/XzdDRxSvbrjHoplfdEhpmcDHQ+SDBsIhgYTSntTdegR/iS6Gd5BeTuLvE3+6IH581cWnwXp4pKupNumbOD+7G6iMZz3W/IHvBLW0keOEeYJComM2za8c6529cz+VpZSF5sCBMHiEIsxHHIFDv4BYLuEYdgc2/8Ff+KueJRbcdYczCfZgXW30LvOcZ6zl2KG6WeYpG3vHfvelP2Ywzuls1ahoHWBK4HWbPn4j5tRyWAjmo0toa5rPRz4TxKpxvUH+TyJy+dC6HQghdvYQafyWj1s7U7l1fWN0pzmxOpjqqjudUPZe59eImeU1CrdtvhfE8PTJz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All but one caller already has a folio, so convert free_page_and_swap_cache() to have a folio and remove the call to page_folio(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Ryan Roberts Reviewed-by: David Hildenbrand --- include/linux/swap.h | 8 ++++---- mm/khugepaged.c | 2 +- mm/memory.c | 2 +- mm/swap_state.c | 12 ++++++------ 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 3e2b038852bb..a211a0383425 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -440,9 +440,9 @@ static inline unsigned long total_swapcache_pages(void) return global_node_page_state(NR_SWAPCACHE); } -extern void free_swap_cache(struct page *page); -extern void free_page_and_swap_cache(struct page *); -extern void free_pages_and_swap_cache(struct encoded_page **, int); +void free_swap_cache(struct folio *folio); +void free_page_and_swap_cache(struct page *); +void free_pages_and_swap_cache(struct encoded_page **, int); /* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; extern long total_swap_pages; @@ -524,7 +524,7 @@ static inline void put_swap_device(struct swap_info_struct *si) /* used to sanity check ptes in zap_pte_range when CONFIG_SWAP=0 */ #define free_swap_and_cache(e) is_pfn_swap_entry(e) -static inline void free_swap_cache(struct page *page) +static inline void free_swap_cache(struct folio *folio) { } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5cc39c3f3847..d19fba3355a7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -731,7 +731,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, node_stat_sub_folio(src, NR_ISOLATED_ANON + folio_is_file_lru(src)); folio_unlock(src); - free_swap_cache(&src->page); + free_swap_cache(src); folio_putback_lru(src); } } diff --git a/mm/memory.c b/mm/memory.c index a4b9460d6ca5..f2bddabae199 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3452,7 +3452,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(new_folio); if (old_folio) { if (page_copied) - free_swap_cache(&old_folio->page); + free_swap_cache(old_folio); folio_put(old_folio); } diff --git a/mm/swap_state.c b/mm/swap_state.c index 2a73d3bc5d48..b194dcf49f01 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -283,10 +283,8 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, * folio_free_swap() _with_ the lock. * - Marcelo */ -void free_swap_cache(struct page *page) +void free_swap_cache(struct folio *folio) { - struct folio *folio = page_folio(page); - if (folio_test_swapcache(folio) && !folio_mapped(folio) && folio_trylock(folio)) { folio_free_swap(folio); @@ -300,9 +298,11 @@ void free_swap_cache(struct page *page) */ void free_page_and_swap_cache(struct page *page) { - free_swap_cache(page); + struct folio *folio = page_folio(page); + + free_swap_cache(folio); if (!is_huge_zero_page(page)) - put_page(page); + folio_put(folio); } /* @@ -319,7 +319,7 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) for (int i = 0; i < nr; i++) { struct folio *folio = page_folio(encoded_page_ptr(pages[i])); - free_swap_cache(&folio->page); + free_swap_cache(folio); refs[folios.nr] = 1; if (unlikely(encoded_page_flags(pages[i]) & ENCODED_PAGE_BIT_NR_PAGES_NEXT))