From patchwork Mon Aug 21 20:44:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF12FEE4996 for ; Mon, 21 Aug 2023 20:44:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A97E3940015; Mon, 21 Aug 2023 16:44:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A4702940013; Mon, 21 Aug 2023 16:44:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87237940015; Mon, 21 Aug 2023 16:44:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7013F940013 for ; Mon, 21 Aug 2023 16:44:43 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 504A2C07D7 for ; Mon, 21 Aug 2023 20:44:43 +0000 (UTC) X-FDA: 81149290446.14.5240C6E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 9CB054000B for ; Mon, 21 Aug 2023 20:44:41 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rzIBTvDK; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650681; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iQ6N/LjQx+6o+U2nCa8DzMMTkBWW+AcF6vFCml2CAOc=; b=CMbg9Rs4DnLTzz/N/yplIln66rMWcVgK+Al9KCjieNU8rmlClrLBkTcGUomrb8g+bj0oJk ynkSogq+hxV0M7//XrU7OG3Ze89HV9vSGjq/3jusNC+MlXYYOzZGD/Qs+LH95Zo+puudHF D2KAMr4SlGmzt9NA/F5ABVUqJ45L9EQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rzIBTvDK; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650681; a=rsa-sha256; cv=none; b=jF99Aqu343ypGkb95GkNsUEFYCWQwWZEePNZBuspDPNo3gL5SN8EGhBLtPZt8sE5Z/YMAu qC7V5DVA8gmrKhmqiBUwwyvyyK8Vs850z4k/upquKt98MT88mZG1a+pZFdsrjkhT6cmfx5 Zelapm5u1s02HPOgds8S6KLECwTNUSM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=iQ6N/LjQx+6o+U2nCa8DzMMTkBWW+AcF6vFCml2CAOc=; b=rzIBTvDK8Pk0RQceRRAl76K1f9 NGnTg9B59ZUz3WjY2SEsQN+m8Ucg8weTeTcBwjS+CUceeWIDduR1Uv06uxGl4vMDlBCMPZYE7ttcK I8/EnoTTyv2Yecui/H62M7nREHC7USK/BGSjwK1g0OmGOC8sY6BDa/DZDN/FXIRm2kWdH/2REfwlb 1GpgRBuCB96Y6mXA8kGAsI3rKNCe3ypQKbyqd/Hwnak8ALNLQLemvTNd0E89YYvNo2mI9KVVuI5AA wMp4XqKQc8qbpgEKwqe92Zi3pQCOXXB0/KOHbnviyd69XPXsDkpom+BUzOJW4t+5HrAk11gyRQE3m T1bYbKjg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlE-00CL04-22; Mon, 21 Aug 2023 20:44:28 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 1/7] mm: Make folios_put() the basis of release_pages() Date: Mon, 21 Aug 2023 21:44:19 +0100 Message-Id: <20230821204425.2940496-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 9CB054000B X-Rspam-User: X-Stat-Signature: niyok1bt8p6prb94ox8us7f6h67km6uk X-Rspamd-Server: rspam01 X-HE-Tag: 1692650681-743770 X-HE-Meta: U2FsdGVkX18Vyg/0Ox1QkyZWhgLP0ctZs26lBRbv1C/gedTGCL9lON9TVDhSysJ/H42+39JXpjt9R+deyy/K0QT5D2I13kH0Uracfb0QU4iyH7Xnrs7CJWs0MyVNk1BYNijlfEBYkU8esXVwpiJ5uR2gxIgBgVoPE/i5WHeeFPaHRKnW3MYo8BuS3k5C/dMbFUxKVIca5/zzfU/OW+nF0A7sCDbRxK0Fh6j5YpBiOR5v50rLlakbsDYuJ+cxgXgUl5fuFW3WXdJMm5MSl2TN4bkYaInBAoj7dGQUq0feLvdw7UMihh6B6M2ZEo7lBz8xEq0cN8v+Tttj4XpZYHl9Jay/Biv5tZck2qsIjkclbSDvEMoS/B8HzyZyMZb3Evki6zSW7PQ3JRrXNnPI0NH7LQyRtoqe5I2c0NxY0XMeULfHUewib4EuN2sYzO5jX0vsBTeyDqOzIHH94E+4p0tJzKXzS3z+gHaLuIMbFonqe8rjL6zC/H271xwO31nAtHmL6TzEbbAvNbYqfP4W84UYbbnhs3PpS5hlJUs+oyGkFoC5c/0TK33Uqu1PA8Lgfn804UtF3J6aZx37vfTOIEfXkzJK9Ifpbi4Zz1Ak4uI9RDz3qnd45Tipe5mGxN0Gw4PUVRrS+1SbKzQEsXsYxmnXvJ6KACjZycDF3YSh3SJllaeXqDOJ9+FffngFJapf4wERUMkYyW4+muXMPCqqQusCiu7p+SjZyJCJXuQb+N/QG+hodK6wmxj3/qUTY4K/E38rK9mChuY9Wr7HbxNUQ1kh1BNPRPkDy3f9JntIX0QIrmI+cQ4vxeib0tj/saC6IYLmWfMJj3DVLy0rizLBEOJT9cNwEUVN8SYfoKIDyZjvfTNiCx+onYVJ8qoxqqdW9vMuxYvNW7ruWhMy7A5uC/arsDu0lszZqCVHeLtxdDGO6tNNhJeZtqT7kbiWeV98fXkVpL6zli7paX1GXvRKMGR b3MPfz4B iAVMSUJRfpg5bT0mJVDcJUsXBQW15Z7PiMOoIRc/awog9fqCl2IaYtDXJKWuc9Gyt1kFeG3WTPlPIt1lRC+sjybMv2M0gHNvoIWtaY6qMcI7xbgdHYSfUp3UesMkTcWhhQO/ffs6XeTskOB1wFJxa77BwP1LTvvjT5d5jI8bpvfb5L0zzwLr4J879ISVO+RKfwMREEZI9gLBDKl38GfQLYGlkkX9+p5fAugRW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: By making release_pages() call folios_put(), we can get rid of the calls to compound_head() for the callers that already know they have folios. We can also get rid of the lock_batch tracking as we know the size of the batch is limited by folio_batch. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 19 ++---------- mm/mlock.c | 2 +- mm/swap.c | 77 +++++++++++++++++++++++++++------------------- 3 files changed, 48 insertions(+), 50 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 85568e2b2556..c1fc81da8dff 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -36,6 +36,7 @@ struct anon_vma; struct anon_vma_chain; struct user_struct; struct pt_regs; +struct folio_batch; extern int sysctl_page_lock_unfairness; @@ -1497,23 +1498,7 @@ typedef union { } release_pages_arg __attribute__ ((__transparent_union__)); void release_pages(release_pages_arg, int nr); - -/** - * folios_put - Decrement the reference count on an array of folios. - * @folios: The folios. - * @nr: How many folios there are. - * - * Like folio_put(), but for an array of folios. This is more efficient - * than writing the loop yourself as it will optimise the locks which - * need to be taken if the folios are freed. - * - * Context: May be called in process or interrupt context, but not in NMI - * context. May be called while holding a spinlock. - */ -static inline void folios_put(struct folio **folios, unsigned int nr) -{ - release_pages(folios, nr); -} +void folios_put(struct folio_batch *folios); static inline void put_page(struct page *page) { diff --git a/mm/mlock.c b/mm/mlock.c index 0a0c996c5c21..67ec840cf5f1 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -206,7 +206,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch) if (lruvec) unlock_page_lruvec_irq(lruvec); - folios_put(fbatch->folios, folio_batch_count(fbatch)); + folios_put(fbatch); folio_batch_reinit(fbatch); } diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..11ca25d4843f 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -221,7 +221,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); - folios_put(fbatch->folios, folio_batch_count(fbatch)); + folios_put(fbatch); folio_batch_reinit(fbatch); } @@ -946,41 +946,25 @@ void lru_cache_disable(void) } /** - * release_pages - batched put_page() - * @arg: array of pages to release - * @nr: number of pages + * folios_put - Decrement the reference count on a batch of folios. + * @folios: The folios. * - * Decrement the reference count on all the pages in @arg. If it - * fell to zero, remove the page from the LRU and free it. + * Like folio_put(), but for a batch of folios. This is more efficient + * than writing the loop yourself as it will optimise the locks which + * need to be taken if the folios are freed. * - * Note that the argument can be an array of pages, encoded pages, - * or folio pointers. We ignore any encoded bits, and turn any of - * them into just a folio that gets free'd. + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. */ -void release_pages(release_pages_arg arg, int nr) +void folios_put(struct folio_batch *folios) { int i; - struct encoded_page **encoded = arg.encoded_pages; LIST_HEAD(pages_to_free); struct lruvec *lruvec = NULL; unsigned long flags = 0; - unsigned int lock_batch; - for (i = 0; i < nr; i++) { - struct folio *folio; - - /* Turn any of the argument types into a folio */ - folio = page_folio(encoded_page_ptr(encoded[i])); - - /* - * Make sure the IRQ-safe lock-holding time does not get - * excessive with a continuous string of pages from the - * same lruvec. The lock is held only if lruvec != NULL. - */ - if (lruvec && ++lock_batch == SWAP_CLUSTER_MAX) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; if (is_huge_zero_page(&folio->page)) continue; @@ -1010,13 +994,8 @@ void release_pages(release_pages_arg arg, int nr) } if (folio_test_lru(folio)) { - struct lruvec *prev_lruvec = lruvec; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); - if (prev_lruvec != lruvec) - lock_batch = 0; - lruvec_del_folio(lruvec, folio); __folio_clear_lru_flags(folio); } @@ -1041,6 +1020,40 @@ void release_pages(release_pages_arg arg, int nr) mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); } +EXPORT_SYMBOL(folios_put); + +/** + * release_pages - batched put_page() + * @arg: array of pages to release + * @nr: number of pages + * + * Decrement the reference count on all the pages in @arg. If it + * fell to zero, remove the page from the LRU and free it. + * + * Note that the argument can be an array of pages, encoded pages, + * or folio pointers. We ignore any encoded bits, and turn any of + * them into just a folio that gets free'd. + */ +void release_pages(release_pages_arg arg, int nr) +{ + struct folio_batch fbatch; + struct encoded_page **encoded = arg.encoded_pages; + int i; + + folio_batch_init(&fbatch); + for (i = 0; i < nr; i++) { + /* Turn any of the argument types into a folio */ + struct folio *folio = page_folio(encoded_page_ptr(encoded[i])); + + if (folio_batch_add(&fbatch, folio) > 0) + continue; + folios_put(&fbatch); + fbatch.nr = 0; + } + + if (fbatch.nr) + folios_put(&fbatch); +} EXPORT_SYMBOL(release_pages); /* From patchwork Mon Aug 21 20:44:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56CE7EE49A6 for ; Mon, 21 Aug 2023 20:44:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B316E940011; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE1E4940010; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A997940011; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8664B8E0012 for ; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5ACB11C9A64 for ; Mon, 21 Aug 2023 20:44:39 +0000 (UTC) X-FDA: 81149290278.07.FFAF006 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id AF8D140010 for ; Mon, 21 Aug 2023 20:44:37 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rzJR+ALE; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650677; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R60f4MZlgl3FiJgLjXWtZXeM99QHm1vtfudYO1B8YHk=; b=KrjqQjUa31x2y6PkVJuxBysiff9OmyQ4a+IRnPfZs18NJ7mrO79Y6L0CS12v2oxbLNTGGZ ETw7S+gGSqWri5+RBGTIjEKrsIcrQomUepNOV5RslZ/LbKEPcUp8LV5/uhSM5e5VsarvKC ofJeGkmjFzhi/AgEvo/Z6ZTVnMKdUGU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rzJR+ALE; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650677; a=rsa-sha256; cv=none; b=paTW5ph3rCE1loiun7gkcVZNbIJ0OLuPuRb48bVRSqkOyU1JFeKESXReTz7ncdmY69dkMk Cc+YAKYmxywp+Vi72wolK68DDyxMoWHf1F2RqYL+p+1sxk3uY90+oTtrvZoa+ANNzWPwTA e70mwI2lCwj9sCGtbcJDAbqMGfi0sd4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=R60f4MZlgl3FiJgLjXWtZXeM99QHm1vtfudYO1B8YHk=; b=rzJR+ALEuWNqK9HrcFfa8MzDG/ /cTXZSUZqPrUIkirOzoygnEazjDTrCAdZl+jsxjLjiS8BjfWrpHcucFTaJlBhX/QbsHZ8dRnzUqP/ oLtAzcQQGv58zaD8LnjUXJZTBJ+vM9y82DUozXRCXVK6QbJ6ue+HqKSk/mZpOQSzdVQ88X2K25n9b awISxEAZ+9IUkRF3s1UKwSIzwilgwGFpmnBBRe6f5X9NHnmqlnpbd2iLaP4dcLZ5Dw4olzWomrOWe dbu3siwyIqVwOHqWvvxUBtF0cvvifgE+ZFikoNkWEC09VzXHCtEjK07Htq5t8bS9FInuFGrmpSYLm 9o43L/Vg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlE-00CL06-9Z; Mon, 21 Aug 2023 20:44:28 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 2/7] mm: Convert free_unref_page_list() to use folios Date: Mon, 21 Aug 2023 21:44:20 +0100 Message-Id: <20230821204425.2940496-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821204425.2940496-1-willy@infradead.org> References: <20230821204425.2940496-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: AF8D140010 X-Rspam-User: X-Stat-Signature: qwaxo1yihs5hxfpfnpzdhhqpjj16jae4 X-Rspamd-Server: rspam01 X-HE-Tag: 1692650677-840138 X-HE-Meta: U2FsdGVkX188aCuYDG9CI5nrebNQMUHdMPD9p5h//S2TCTGCIeWjV03S6QQ/IKkQaBw3B6ROlwd8NauC0RG+4mcp4G0qzI/Oy9IoGrguKwHtxtE92VJ8yrz/q6duftayhAoqNLgLg70Iq0Qev6oqSrYIL+g3zMkWheJUbsR0KEz1swo/ymMHPOBmrWv1UayQK1SVhUh5abx5UWvjA6g5kct+RFxakm61Gbr71XO32VOaKxeqqAfAWGROKxIrULX4OZwN64E9vyKUgbzn5CQYHmeKHNB5DB1Ax2jdTTUhqkjzU/hVaH6A+02kijgEr8J/a0anKLZfs74wB29hfaiAbCR/l3ZssIORXwm7G8LJtCIy+X3oUbCUrwS/EmpwutixCMjkFWlXMeMmlI3wCrxXpLgRub/Ba/jdXKyjl5800l3RZJmCogqZDo4XwBHjw0zhznMv6gISTilXi88CQrFTp7ChpFuCYgvtBjM01VNtbKaYvNnG8XB2wsZCcuvlCNoZ9n2X1FBqMIXVGLdocHgvEqzA0vYTuLIHkMqf4Wua5JnwWEeRmyDhuvZZXI7scw43sYoHjC3jIUQLFBr+UB0KWJ0UNpghggTRi2hhmaRPJfKKtpcGSQIwc3WgwgTIiQyUkg96kwgT7PYoZnn20AI9EUPuqMH06tnNdw/BtDNkTCg1F+wqzD+MVBFD3IWANGfLz8XAhMSrNaIszyJMIJG33dSZweWClJhcPbwsh9QelFDEaJ/W7ZIjJPA0Pd1G6h61+KEP0lr+yP0c5evPafLuxmAA4pNHgpIC7XwApn94pboBsIJIUkV3Dn/d9AizXz2YwOA/YI3jp5vQfpDE65pBhUxHyNigs7c8rDgv8iV++Rll9SzdwDgf3hOeDG/Q6H0Y2FMJcrgwntuo+/6I0kplivb1ZEEFhZqACuBr+39gC8yRKJ1u5SqSkS/+Za5+sd8RdxjiOT8sy4RGZ4VDpPi i516XEaE /5cmJA53xm0bOznSrg/AsqIKMjhVr2XC8lfs+brbLbOfuPlcWcf9YAI0JCoQYiHoS1yJp6MWxS2B0edgq04n1siwCgC7QUgnBETjOPXEELszhKjhELDolHZD4DZu3zreAwcSA3kgY1yytsYq77nyRf4Oc3OwM2UzaAD+Y+bfeEhDTy1KOqc6lKlPoGA00fpg8+4C5emxqNtUYweUQ7d4fXAeF4xeKdwhEzTzc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most of its callees are not yet ready to accept a folio, but we know all of the pages passed in are actually folios because they're linked through ->lru. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 81b1c7e3a28b..2f2185929fcb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2498,17 +2498,17 @@ void free_unref_page(struct page *page, unsigned int order) void free_unref_page_list(struct list_head *list) { unsigned long __maybe_unused UP_flags; - struct page *page, *next; + struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; int batch_count = 0; int migratetype; /* Prepare pages for freeing */ - list_for_each_entry_safe(page, next, list, lru) { - unsigned long pfn = page_to_pfn(page); - if (!free_unref_page_prepare(page, pfn, 0)) { - list_del(&page->lru); + list_for_each_entry_safe(folio, next, list, lru) { + unsigned long pfn = folio_pfn(folio); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) { + list_del(&folio->lru); continue; } @@ -2516,24 +2516,25 @@ void free_unref_page_list(struct list_head *list) * Free isolated pages directly to the allocator, see * comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(page); + migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&page->lru); - free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE); + list_del(&folio->lru); + free_one_page(folio_zone(folio), &folio->page, pfn, + 0, migratetype, FPI_NONE); continue; } } - list_for_each_entry_safe(page, next, list, lru) { - struct zone *zone = page_zone(page); + list_for_each_entry_safe(folio, next, list, lru) { + struct zone *zone = folio_zone(folio); - list_del(&page->lru); - migratetype = get_pcppage_migratetype(page); + list_del(&folio->lru); + migratetype = get_pcppage_migratetype(&folio->page); /* * Either different zone requiring a different pcp lock or * excessive lock hold times when freeing a large list of - * pages. + * folios. */ if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { if (pcp) { @@ -2544,15 +2545,16 @@ void free_unref_page_list(struct list_head *list) batch_count = 0; /* - * trylock is necessary as pages may be getting freed + * trylock is necessary as folios may be getting freed * from IRQ or SoftIRQ context after an IO completion. */ pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, page, page_to_pfn(page), - 0, migratetype, FPI_NONE); + free_one_page(zone, &folio->page, + folio_pfn(folio), 0, + migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2566,8 +2568,8 @@ void free_unref_page_list(struct list_head *list) if (unlikely(migratetype >= MIGRATE_PCPTYPES)) migratetype = MIGRATE_MOVABLE; - trace_mm_page_free_batched(page); - free_unref_page_commit(zone, pcp, page, migratetype, 0); + trace_mm_page_free_batched(&folio->page); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); batch_count++; } From patchwork Mon Aug 21 20:44:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C25BEE49A5 for ; Mon, 21 Aug 2023 20:44:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D679C8E0012; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CBC44940012; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1F8E8E0012; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8CF72940010 for ; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4D555140883 for ; Mon, 21 Aug 2023 20:44:39 +0000 (UTC) X-FDA: 81149290278.06.23D19B6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id A9449140017 for ; Mon, 21 Aug 2023 20:44:37 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cX48794h; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650677; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RORJVnZ1xjomMTeygi3JJ1E6/B1esghHOmLJuCDuVL8=; b=Xn0A4Dc0k6y9R+TNPAnjYoGZE2NSRCbNkA/1JYh9Khz2BNtrPs0NdDuIbw9Phuyju42Cga e0sBy/F21DpuWRXYf63yJGWgzQavqKT8v1GSI9RKpBG1kU+4Gi6VLt1zJVcFaO1DoH5vWk PQULN4EPIWIMVeJ08bd6aCPuZ+MH/fI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cX48794h; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650677; a=rsa-sha256; cv=none; b=ZzGxlns9MAnOXx86QPDL6dsmQZ/gbPbR07ZEBp7xvH+MElSXCLLHTXolgFutF9o4GdVRO5 qNJTztZ7nx0X1UyDYjj4lLI/Qi5JWRTlZc6IilEgQ+GbmukM/gtVVCJQBpEnuPK6wiXXiq rLfMqqhQNXhR6WMTzAOZsGEMOoAgsHA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=RORJVnZ1xjomMTeygi3JJ1E6/B1esghHOmLJuCDuVL8=; b=cX48794hHeiyAzu9fcQmsTqm/r aqdYpWMbkTSg7sdRenY7DY184Sjqx0L5TE1lRgR1Rv/aCmpiho/0J8NfdRRgDvgFHaQ9sh85P0saE Ey1GDns++lTylnBp8OVqiMfqhFMzLA0fpZGEodGibshcHZBSYp3j6aPLIDFwP1ocmxTHjfQXSr6px xr/QshlBPusGF0zTRlEZH8lHdsFiiY9YmUKnFUHdiJr3VPUBOpZfpiAhZnQ8FyqDrjkrlCHWc+dgm 7+hR/W6e6bBmn7K8tUqHbHnagtZH6cUOlrhdg4Auy7hyJ8bNsFbqaLdZ3mv0bZKbpg7B9GN6tG/LJ C4P1H2Sg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlE-00CL08-JJ; Mon, 21 Aug 2023 20:44:28 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 3/7] mm: Add free_unref_folios() Date: Mon, 21 Aug 2023 21:44:21 +0100 Message-Id: <20230821204425.2940496-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821204425.2940496-1-willy@infradead.org> References: <20230821204425.2940496-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 9waj7ewjnxjeq8wz1nakpabjhuwqy5wx X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A9449140017 X-HE-Tag: 1692650677-931918 X-HE-Meta: U2FsdGVkX1+GsoJjLvKuCW2X6CXAnp4QO9GgjaEE+fDEV2TSJ/vXoQwwp++n0kSOFneJe88TVc88K5H9F51JhmG8ciSWpYX7xcHN4sDzabLs41cyzYP7ZZr5tn1VkYGeFuoD/6B9uAn+LQeFHb3nFaLuKSW7keST9KXYQJl/5TkqWPoqRmIKH245FPcl9JYit19zcyG29TU+ggFFw2Oix8JO6vOijA1j7zwbiwLvJ7txhzQIt2TBF/SdmfQQPge3hc2A1KCZ+LcX92RH+ZEFrQ6KoriIVL/8B3HQam/2BA1AtkykyScNuFhu0BbOwVcHjMn0b9dGN1Rk9YGCdzNEziwGSN7AxPMJ2ZaIUHYOiLKHWYMUMg9ztQxDiDn+ILRKCYExKZ0g5vPVszi9JeO7bTRc/RVIwPSi+tDXpOosGOjYZEFmOJwWnQHaX56OKbd/JbRH52iBYHE5SQbPODzRcRF0JEaulHp4O4mLwBP0P8dvkannkl0wb1Jlk8ClW2cah0rVuF/4VRzGLO3ut0yyIJIFgRoo4hqd4LQpIll768st36MJJ/T3dm7TMR7d/ikfoIOy030ipAYpJkEeqh8jMqDrhfPTtoKNX8bob1xSvT/bJf8vGlXaaJT1A0mOitaabK5llsOSlr3y+xlDmZE5ANoXJoLW1TbyPrHtroMCvLxD69H0hByAZ83a2QOr1eiDidwcZ34EB0y1ECBUh/58AMCCWiZe+RDuaDe7HGbIrV/U9QrbXvoHWjF0HNz0luEC2OwjulvxWe2sm7cNrr+aQOBghUaWPontv+Mmfrt+WaCYIArAC0hGrUJuPJa18O8BkXgV6R57AYaHpDeMuo0/RDC7aNJHsawn0vycZNVMPxdI8tbn/tl6YXhbv/Gi+OMlSZkkXdvaE9GTLEpVHBhy58zzpBfrtOvarvfxTHAva+/3zghhreAJs1JWUHm/9UOBJ9sL4H6myDWU1nkzMaA SvQ6p4xq zWYdNdt/8S6XNb/7pSfH2FeeEGrYdAVxWU5gWEHCIZ1sAFPZuu82C6g1xtHHipo6EDoEXzJAfd9Kfn2bFAqgFnBU7xqFR7IY2Jz5YzgibIWkqeNEBHs6z3xj32dYkpZGvbMbpbNkesMlc7WDIPTyrvMEFiuILsNTEAsFzxmewJSM1CHsSLohYSmoP9Oj/Bsq9DMYVEsW7DfQZlnjaTtoRTclOa9v5l71vgBUf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Iterate over a folio_batch rather than a linked list. This is easier for the CPU to prefetch and has a batch count naturally built in so we don't need to track it. Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 5 +++-- mm/page_alloc.c | 59 ++++++++++++++++++++++++++++++------------------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 5c777b6779fa..3e6b448e7d63 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -440,8 +440,9 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern int user_min_free_kbytes; -extern void free_unref_page(struct page *page, unsigned int order); -extern void free_unref_page_list(struct list_head *list); +void free_unref_page(struct page *page, unsigned int order); +void free_unref_folios(struct folio_batch *fbatch); +void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2f2185929fcb..4354938ca3b0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2493,24 +2494,21 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a list of 0-order pages + * Free a batch of 0-order pages */ -void free_unref_page_list(struct list_head *list) +void free_unref_folios(struct folio_batch *folios) { unsigned long __maybe_unused UP_flags; - struct folio *folio, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int batch_count = 0; - int migratetype; + int i, j, migratetype; - /* Prepare pages for freeing */ - list_for_each_entry_safe(folio, next, list, lru) { + /* Prepare folios for freeing */ + for (i = 0, j = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) { - list_del(&folio->lru); + if (!free_unref_page_prepare(&folio->page, pfn, 0)) continue; - } /* * Free isolated pages directly to the allocator, see @@ -2518,34 +2516,31 @@ void free_unref_page_list(struct list_head *list) */ migratetype = get_pcppage_migratetype(&folio->page); if (unlikely(is_migrate_isolate(migratetype))) { - list_del(&folio->lru); free_one_page(folio_zone(folio), &folio->page, pfn, 0, migratetype, FPI_NONE); continue; } + if (j != i) + folios->folios[j] = folio; + j++; } + folios->nr = j; - list_for_each_entry_safe(folio, next, list, lru) { + for (i = 0; i < folios->nr; i++) { + struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); - list_del(&folio->lru); migratetype = get_pcppage_migratetype(&folio->page); - /* - * Either different zone requiring a different pcp lock or - * excessive lock hold times when freeing a large list of - * folios. - */ - if (zone != locked_zone || batch_count == SWAP_CLUSTER_MAX) { + /* Different zone requires a different pcp lock */ + if (zone != locked_zone) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); } - batch_count = 0; - /* - * trylock is necessary as folios may be getting freed + * trylock is necessary as pages may be getting freed * from IRQ or SoftIRQ context after an IO completion. */ pcp_trylock_prepare(UP_flags); @@ -2570,7 +2565,6 @@ void free_unref_page_list(struct list_head *list) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); - batch_count++; } if (pcp) { @@ -2579,6 +2573,25 @@ void free_unref_page_list(struct list_head *list) } } +void free_unref_page_list(struct list_head *list) +{ + struct folio_batch fbatch; + + folio_batch_init(&fbatch); + while (!list_empty(list)) { + struct folio *folio = list_first_entry(list, struct folio, lru); + + list_del(&folio->lru); + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); + fbatch.nr = 0; + } + + if (fbatch.nr) + free_unref_folios(&fbatch); +} + /* * split_page takes a non-compound higher-order page, and splits it into * n (1< X-Patchwork-Id: 13359800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A594EE49A6 for ; Mon, 21 Aug 2023 20:44:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FA78940012; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A9D9940010; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AD99940012; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6028F940010 for ; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 368B51A03E1 for ; Mon, 21 Aug 2023 20:44:40 +0000 (UTC) X-FDA: 81149290320.06.867E027 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 85C8F2000A for ; Mon, 21 Aug 2023 20:44:38 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=v0t2Syi3; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650678; a=rsa-sha256; cv=none; b=3tzsEjQHQJmXSX0KJMxg6oj1+mmwleEWXF+uUrX5XnW8OQd/xJ1J92fC4JIgDj2ncbnnbU lrVoqUSy98BcjIp3i51Q5Eb+fk5lO/OghAhpnN56bx5xz9o1UQsh2NwAsjLs4rPDC6y4dn TPvT+RaKXjvcixS2N2c+hW7RWurcGZY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=v0t2Syi3; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uZKmm6KWu5gIA8MZSDwDZfTtk9T5DcLVj5SgxVN7+4w=; b=6VxYqyD0sx0WY256dZyiLm/luqCQ03X49kZo4fOZR/59q0i6L1/V0koKxED198WfBG7gLl lnZhYmRNH1/os8qBJ6lejwMBfdYsih7MlO1QnoAnJrdI/P2xYpFcIGt4VMVbAYsA+Ppr/s m8LV3rdpoyOmMloAwTYWsQdwgyPDpDc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=uZKmm6KWu5gIA8MZSDwDZfTtk9T5DcLVj5SgxVN7+4w=; b=v0t2Syi3ZRpGhRgAu0OekJV8fg i1bZOE90UDMiASJIcuWOZSxmfOOePDQlvt8HvZfsbeZ6oq5e/e3hdmJa6CjWaFDpueGvhk5C4O7RG mM3iWkWtoMQIoEsJRcifKq6bTwVt/5uIPogb9Y4GHBmo/+CAqeS21plFM9t9F/hJ8SyqTLDKZw6ZJ fl/px1t9awQR5d266OlhyEh/q6fQ1QH7FNmMZYESIgL6huigzRGkKtCQwXvAWgD40tyLj/m54ZLp8 yj46PeNE7g2cbyCrU4r6JMxWVOD71/V7EEtmktVZCqtbvz3nLMg1bcnMJF1HOVhJs+M4eBiIC1uZs 1Z1cVBng==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlE-00CL0D-RA; Mon, 21 Aug 2023 20:44:28 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 4/7] mm: Use folios_put() in __folio_batch_release() Date: Mon, 21 Aug 2023 21:44:22 +0100 Message-Id: <20230821204425.2940496-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821204425.2940496-1-willy@infradead.org> References: <20230821204425.2940496-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 85C8F2000A X-Stat-Signature: pwros5auw5ac663w641hpifnngtkibt1 X-Rspam-User: X-HE-Tag: 1692650678-342975 X-HE-Meta: U2FsdGVkX1+L+gM8PteIG3TmL83iXu/UWLMFmq8KMIgDJl3ylSHNzx3aX1217dyZOMyLLN4oBFpsp5cpTfhj5niluaQZx97GAuTjrf9bmdmEDJKBVfyD69xDeEjpXXSk45AisvpkKibF2HT0Bq1Fl0eJcW/bv9+kcPruKSVjOA9VaU2TQiAVDpeENzkxdC8ZsVd66/Sb1/NOBzt8fBW9YHiC0UegFOzb0M3t3u0bDgNm5VO0sa7t9gC/0AnbJFq1JpYpOSAVOdP2s+dg9WfvXX4jzAogkVpywOygJNmZs7T38uT4k35rQG2HV1ShYln6jKsY144RAYDjiEblTai9r9SCUKFSp2k9Qfgq/mSIMkd/pQKNxTeVZT3qReSpmmWZ4iMOADXn0isdaa4j5xhtccUXgPWjEFrTY0Iqtqy/bVSq3NN+7cPbIs6fi+cB69ogmCzf05sGvWxT8zotGEWoSHeH8Xpx/MvcRxx5ZMOUW2bsEjZ322oylVBtRhSW3MkTKtn7gqRNeqRAzt6+z9m/FA1enH9e8jShdvkXsPF4+Cay4JH3EAgL874iz2Uq1mgPgkWplwL3YTch+aUGP77oUrhBlowHt5R7vzODDrX1l7MG23lXA6md7ukgKZvZW7e3h0MT0jl6eBx5K/fSK2MLFq0cMMJ5rDXItS+K66WENwGlsRo+huTVCDc2VPi4WoqMMieMvpgEnQ4M4VW4gNmSxUPUsNwazV5sRenz1LJifoSd+pkn17xLM62UUuh66GStnmnZfjlaEUqaGFja2Kndwz3XuM1rBZ+9gemdneuiAGuqoH2mRCwY4Gsmkq+gJiRPAY4Nnn4T9EAxJcaCgXSknuEXjFIC+fh1bXxxa6I4wfP8iuZ3kCY6X6ewOpeEFgBd0cZOwxPNRy1UQGv5eQ1YcKQRfnY+JvfMJGBpTGtuxqfuOHLZYqz1eQKyxYG+RQ+lt6A1+k27KeEuGCYipCF EVxvw37h 9U8LFD4m04OnvRX9WEyWSOUX1YGB2Blqins6fQ+u0ZZTNs3Sb6nAjwZSmLev+zxzpGa1uJWz+9cXJyihpVdxhz/d7g2mnZYgLqsDjDBSCN46NTUJTP3XxhFa+RAFw5Ty0q4MkWLMZanaaWyzfpKVobVpfrQOePBZkjmYarYTVp84PrML0vdxwOlhez2rt/C53KO9IUG3941txh8uFicgSS1zd/+0NyxiqORY0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There's no need to indirect through release_pages() and iterate over this batch of folios an extra time; we can just use the batch that we have. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swap.c b/mm/swap.c index 11ca25d4843f..9d31185dc27b 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1072,7 +1072,7 @@ void __folio_batch_release(struct folio_batch *fbatch) lru_add_drain(); fbatch->percpu_pvec_drained = true; } - release_pages(fbatch->folios, folio_batch_count(fbatch)); + folios_put(fbatch); folio_batch_reinit(fbatch); } EXPORT_SYMBOL(__folio_batch_release); From patchwork Mon Aug 21 20:44:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2B30EE49A8 for ; Mon, 21 Aug 2023 20:44:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27EDD8E0015; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 22FAF940010; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CE4B8E0016; Mon, 21 Aug 2023 16:44:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E964D8E0015 for ; Mon, 21 Aug 2023 16:44:39 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B804340D05 for ; Mon, 21 Aug 2023 20:44:39 +0000 (UTC) X-FDA: 81149290278.07.B2B24B3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 2ABE91C0015 for ; Mon, 21 Aug 2023 20:44:37 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aiiT88YK; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0TDRpjnt3h85NYff/3U+8TfIQhuK6SlrkWQVnJ9yr4o=; b=saY8PNPeFmkocy+yPDiwle/rDLJ4FBbrjmcZUsq66733jUeOZZpe8rt4h4Y5SNtOO+BHBp ZEpeV70AmGROkOCZUWqy26zVE7/9YUluGjcD2KTeibDL0OlDf/De2OWFWqHhJPnVicGJM9 JgwHceOBFiYaM4ZRSrxg8Lb357yPIsE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aiiT88YK; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650678; a=rsa-sha256; cv=none; b=rtX7QplNUt3dvdUHABsyMwmkJDIAJ++yHJ0nGixgrHAYTZVPEwfBnrIcVZFLg/qJkEO/Ny RbjtvlHSdtITMAeS9F68Q8q1nEIPCIkcn/q9zJwj+wcqwJKDTumy22UGCUIKNMTeHsr7mk zb4BOQXG11VJyhBgSlGuUnNEnwJrbJA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=0TDRpjnt3h85NYff/3U+8TfIQhuK6SlrkWQVnJ9yr4o=; b=aiiT88YK6Uq8pKYYh93LEy3hCo ajum08THddgPaTpbnBA3jfDAgrJJ5E79QHuouJ+U0j4Z/1E1CUN/BUnXceIybVt8GPL3Nu+B9PjzD AE0yomUGbAvwFIS2TFZPuFY1qBtBDW1m7tHXVozEnoS1/QH7ZAu+6nsWmQ59131mZAYmBC1PP1OnB gfEaUoDsI/W/CwDvz0kgSGufTriLT0IvyIVDV8y3oisW93lDCHIdxaFDKi1oyXkhEpZQxqzBXROWX ca6+SPI7sfqyPbDKo0gSkQ8ipUxXCD9KpFQkWChmtkVI406PQiDMBYZDV/twMf8te4IPV3GP4di19 6qq////Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlF-00CL0M-CQ; Mon, 21 Aug 2023 20:44:29 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 5/7] memcg: Add mem_cgroup_uncharge_batch() Date: Mon, 21 Aug 2023 21:44:23 +0100 Message-Id: <20230821204425.2940496-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821204425.2940496-1-willy@infradead.org> References: <20230821204425.2940496-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2ABE91C0015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: q9up78uddraoqz5qic6m9az1jsq9iu5z X-HE-Tag: 1692650677-94435 X-HE-Meta: U2FsdGVkX18S1nrpCjedLVt94tyviCYY+v2jZ2VfAsYYjeSpQGnFy6LxAUwdXou7lqzR8P27Cryj179NzTF4hWQwAVtEEOVOYcVBgvjL3oJzR7ag/M0FCIjjEv8QuP3w0Hr+0RpSUTEIqgMP/EB5Qi/t4Wwp4z1oOWKw3ZRXf3WGRdphXxsrLaeiK6ZWa8l5FhfQgiFTDwSWJgEW3TjO9t7OkPmO1o4Ar+FeHrHpfO1PMZdyDHC6f1LlMmsgmkxlNvTPm1HuqMSFBN/AcfPwD16cGzupmgt9CHcQpSrG+WTxc84h2SE/SHIyJlrCC7zndLu99vqZv/Rmo6aDmMnzl+UesHm0JODmG3N4Wd0sXd05s18uYPQ8XKSDSqw3e58ogtQ2e2W9Yd0sFk9nCYdfebTyGp8fEov05ipSL0xhtxuS3hbyOGwxhiMmzpMaPp2tV4IRk8+h5OF/vHEcuY7GqCs4roZRG4GVXgwgJ9VgjH9aaedsjDUTnNkfkp3PKbE3YJI5rCCFpQrG4sDGnIcMBG2H25hYEKtp37/eTiskMoU//nW5/Hcha+opwUuQaq24K38Oy9LvQg7bhtXlNwY53qA4MichBWnLWoIiYe7Gf2IGmfVjraD9cpjnqPt0/OoXOidbcBkKN9ejABpmvDsv00LzM0veK0f6W9Dryk7AaQYsdtr2dnK2cVnMNHPuEHXDi75LBk3vihO/SPdkWKkkqqfb35aMwfrGJ0T2k5BqvA6JPcy5B7lU2t9UScknBCz8O4DyGKpFBIyp9usQGei4p7wuwDhsJ409KYA1cRxFIbxz0tFqQfhE6eVNBnHZsptPTiHpAIiORvHJmAqMKMcRAO0tTTeBYXM/zKDrxoXa/lPBlWpYwjLfBr3E4XZJJecdRfF+jQemiylg9zs+FzVXwohogiB92WmM8i6CgVUrWL/eg+NkqUlZTYV2JSQzZmMb4rWkwWFGBv89z8/Vh75 yEeJOCP7 ewE1EZSucCBbSqSlnsSCmRe2feGlVRdYaImX3M41uR0kuH0bzRYwSxns//AJVS54rUB3B5BpgSI2uNsaxJc3TJg37fZ9RclxkOyJhU/TCzgHmnY4pS2i69G6pOFe0rIOjSj7+XpoUOMyQ5Wilq3m+hCQl2mUA0Vz8Me3bnuuwdM8VlF+G3iNAVkFUwZclib9hS6u8PC/K4KwqLn/KQXw93Ahb96HBX2zcMLiFs2knJjl3FPu8mdvCXnGG9gQTKqT95KqjJMxeY/2SKeg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Almost identical to mem_cgroup_uncharge_list(), except it takes a folio_batch instead of a list_head. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 12 ++++++++++++ mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3bd00f224224..4ea6f8399b05 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -708,6 +708,14 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) __mem_cgroup_uncharge_list(page_list); } +void __mem_cgroup_uncharge_batch(struct folio_batch *fbatch); +static inline void mem_cgroup_uncharge_batch(struct folio_batch *fbatch) +{ + if (mem_cgroup_disabled()) + return; + __mem_cgroup_uncharge_batch(fbatch); +} + void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1269,6 +1277,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) { } +static inline void mem_cgroup_uncharge_batch(struct folio_batch *fbatch) +{ +} + static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 67bda1ceedbe..205aa28c2672 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -7194,6 +7195,18 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) uncharge_batch(&ug); } +void __mem_cgroup_uncharge_batch(struct folio_batch *fbatch) +{ + struct uncharge_gather ug; + unsigned int i; + + uncharge_gather_clear(&ug); + for (i = 0; i < fbatch->nr; i++) + uncharge_folio(fbatch->folios[i], &ug); + if (ug.memcg) + uncharge_batch(&ug); +} + /** * mem_cgroup_migrate - Charge a folio's replacement. * @old: Currently circulating folio. From patchwork Mon Aug 21 20:44:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4523AEE49A5 for ; Mon, 21 Aug 2023 20:44:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD571940010; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C6F21940015; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D343940013; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 741F8940010 for ; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4E7301A06EF for ; Mon, 21 Aug 2023 20:44:41 +0000 (UTC) X-FDA: 81149290362.25.7FB6CEE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id A289840002 for ; Mon, 21 Aug 2023 20:44:39 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tLZ3qEXl; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650679; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XRi9y87QqXh4JZ1r/Njdw8M+xnng3qBCOABUAXt/gvk=; b=F9QZCe3IodspcV6gYezfHdNwgLZi2p0/rfLNH2GU+kRFqw1o99a7D7OkLwlSDmLJ043Bbq PS0zAGStUz8UxomNJ0ZHpm0BcX6l76ql5x0pZR3RmVdlzv5nd9LbYxW7aZEQimpKNZf6d8 9DQcWWPiX6lEpkBoDcGg8M/56vbCtFQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tLZ3qEXl; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650679; a=rsa-sha256; cv=none; b=Cm77Cw3MZoKOaKPcQkn98/9z4qE5qO8+4emSCZZGLcr9SrikxCxlKEQP669XQovR711diE n7NP5Hdq9oXk+XTxBxYk+QMRlAqLwL+TXDbvZn5GzB8g2GBgKKTs3CJN3Mk4FH1cqg9CaB uwGdSQA5qJDn/hobnTTxJ4w30DeORgk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=XRi9y87QqXh4JZ1r/Njdw8M+xnng3qBCOABUAXt/gvk=; b=tLZ3qEXllptACApUZPH235uOkb aThCwnrkMLDWUHcF9GtWTqx3jKpPdBX4/PfR8p0avHhtlsbSI7K0SOA87zZo19o0xFiisQoDR9zmy yukem8YeS7ZPMv1v/XH3oCQIO+Ch64oLqjVTBRjuL2ACD2r1fKNxBjovhH16h42r28DEl5pzvNwUR NcfxgknnO1iWC0NnDPkrK+5gqa6mFnQx5bN8HMY7Uu2ZNlNK6X4WjGNubRiac+NWyzcijOPUcV4eH NBphd5Z7PmwM1uyl/DYOTuxF+ZwSJDUI99VbMKZbbtJsX3fSJlZT8pINjV0KwTHd0mARG+4f0LapW dRaj01nw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlF-00CL0U-Mc; Mon, 21 Aug 2023 20:44:29 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 6/7] mm: Remove use of folio list from folios_put() Date: Mon, 21 Aug 2023 21:44:24 +0100 Message-Id: <20230821204425.2940496-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821204425.2940496-1-willy@infradead.org> References: <20230821204425.2940496-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A289840002 X-Rspam-User: X-Stat-Signature: msfbjfnz3tk4fzefokhioqhxsw9cma5r X-Rspamd-Server: rspam01 X-HE-Tag: 1692650679-724317 X-HE-Meta: U2FsdGVkX18MBQM9eaF8/45+hpxk1GicmJ+PWNnF+FVecyUNJ56XlosJln3dGQHTTppDJgXYD+4IT8CEpQAz8A+dINl76DTH8MK+Jwm0lgXRuF8He08/CExiTpGO2PsXPXclyttkxc3EpkULkZ5IGzKwvkgpVXHtDCqAg8JO2lg78/efb3W8BxVThjfwtk9MWoGnHgeO6W7zgUth63gVCytvhIeJJqFIRxIiFZuJYNRME3EZyWSiYd0enbdmlWJR47gqwDlNTZhqU7hG6tARGJFKfks7mZ2305P6G3NIsa/hrgWDTV5sKEMbm8GXvsB1rfXnDngFwW5digzg4aSYiAuGBFjP2HRsDdUYXy33efPaoRkZq1Oeg8JpFjeGKh0bD5TuV/P6MTVoZDdAlQohncpNJmg0zSnpD83EEDAiPo2zzKcSom3WZ87cyat7ryFTM3gPMIDmUMvsK/hQwSAyAg6bcwufKYicx/gTJC2rkJa+PlngVJodS33pqe5g3iKVBbAxZiJOz/nhcVDhP435z9dGFtUh01AM+d1akRExCOBn1yrYrwll6aFMKVjav6yTZFXU2XgA7s4btqXdzQRan7SfS50Q7sImQXFH7SomNC9cVkwBZ5sLiwdOMpDjEtUdveoyJJ//zQgr1DpyJniOcAR4zewoa1K3SpLxi6Audfr0RbUPUPXUY9I5mT5CrMgX12VZF3//a8vlryvoZj95Z6ag9RR2S9LPW9EkqrFPqbzArZG6FAmIO8KINgYWphqorR1fllmW6RDqdaBWrSEzgtm5A3rI7KMk9802gvUc0kTXpimxvSsLaU0vgHYMki8wX7pLsqUO+ctWFPh6Cj3XltKduIttsxx07W24C5wpmaa9fqmE/JPQ1kDTU3aiBAk1csHyLolTfe0LGd93wE7RN6Xrg955VUuMWwBkr9U8Z/qCoXN6tiAnWQsHQQitMpBvelN/7vfhsz+Lzv9k5XB xRUtlXBl p6NxuvfA+nhMXe+QkDfzIV5PFuRFFn5vaMpZpGEKuG92SNf3sWRh6G8M7qBG0RR2z2zgA9HVrxW7DuOtzBBAZRcAs7NGCK9Y8mHs39Xvb3L+V3ovtN6Y7wGOuCe0A1Qf4OBvPX8IdiA8P2gXHOvxairfUWo7NSERzUFjwXfhIexM3uXdrUX0/BA/n206o3NgpSfh+AeBZmMLGAnnxvqSgzdX3VvzrvunzwwXc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of putting the interesting folios on a list, delete the uninteresting one from the folio_batch. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 9d31185dc27b..8ebb10d44de7 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -958,12 +958,11 @@ void lru_cache_disable(void) */ void folios_put(struct folio_batch *folios) { - int i; - LIST_HEAD(pages_to_free); + int i, j; struct lruvec *lruvec = NULL; unsigned long flags = 0; - for (i = 0; i < folios->nr; i++) { + for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; if (is_huge_zero_page(&folio->page)) @@ -1012,13 +1011,18 @@ void folios_put(struct folio_batch *folios) count_vm_event(UNEVICTABLE_PGCLEARED); } - list_add(&folio->lru, &pages_to_free); + if (j != i) + folios->folios[j] = folio; + j++; } if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); + folios->nr = j; + if (!j) + return; - mem_cgroup_uncharge_list(&pages_to_free); - free_unref_page_list(&pages_to_free); + mem_cgroup_uncharge_batch(folios); + free_unref_folios(folios); } EXPORT_SYMBOL(folios_put); From patchwork Mon Aug 21 20:44:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13359801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DADACEE49AA for ; Mon, 21 Aug 2023 20:44:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA90E940014; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B0BDD940010; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9353E940014; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 82607940013 for ; Mon, 21 Aug 2023 16:44:41 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5852D1C9C33 for ; Mon, 21 Aug 2023 20:44:41 +0000 (UTC) X-FDA: 81149290362.26.A80ACC0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 949C214000C for ; Mon, 21 Aug 2023 20:44:39 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ceq6MGoy; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692650679; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pSrtzJTe3IU8p+1cAoPm0gIDtB8SqdaEhhREkvHFPcI=; b=xkgmLN+Lb4lM8WJgFVhUqVv4wCOJHJJDz9TvpMyQFrSsw5vlrVf1IsfylzFVTotjVCsEF0 v8F9AEbyH4AUtron5+jcPvH27vGv5KX0L8A+xLEEgfQwc0sfKDqpo4Ngr25+H8R4bugd0K UyPvfZ1ncctruou0Fek1ob2COASvyUk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ceq6MGoy; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692650679; a=rsa-sha256; cv=none; b=PsSb2np162xJ5bWfASvgWOF1cfKGJLU5uVvcLSYpI9J3OIEnrT7/ePS4KFC0QWz223EJ8M sZA9F4XhOyONP6zfCX6LPatPXxC7XHhCvOFUkpGYi6Zl8sbNayINDg1tfvQK4QWxTtwvZS F/4JNpXCsJxYV7hZ0r+MspI6RR1GhVo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pSrtzJTe3IU8p+1cAoPm0gIDtB8SqdaEhhREkvHFPcI=; b=Ceq6MGoywPDgt/xWZ3yRxd/WSL RkzO1yVGelML9XeqBKiR4+8SdApxY7F+p7ljHaoG0D6HLhXJ/sG+nrVXduVmXoAnrfZEpqL80lGco PoD2MLxNIaGzT2hN7NjKhPe5egBOm4TJfy5VI1r7rVR9CmQKcSySZHP3AUk2xjdkJ77tCaAyHS0vx ib8C07J9p0tmPvrdpd8PxYCPF2p/XZzUiI4BGZZXoZlZbjFfdiSg/7BiSvdoFBaKTt3KLueh+mhML 4uATzCFca0yv3rmigKvdTN8NCsXT3KZJSUPYy4NcdFmF/mR/U4BIKVyhYr4O78PyWou7n71PDyXbO kuyzOn5g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qYBlF-00CL0b-Tr; Mon, 21 Aug 2023 20:44:30 +0000 From: "Matthew Wilcox (Oracle)" To: Mateusz Guzik , linux-kernel@vger.kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 7/7] mm: Use free_unref_folios() in put_pages_list() Date: Mon, 21 Aug 2023 21:44:25 +0100 Message-Id: <20230821204425.2940496-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230821204425.2940496-1-willy@infradead.org> References: <20230821204425.2940496-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 3yo4ofz5y3uoqya78s4b5nziprwp1xmm X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 949C214000C X-HE-Tag: 1692650679-145649 X-HE-Meta: U2FsdGVkX18C3/aOLfeG8zaDUamFwoXov/qdvyGsbDAw1K9Y6pzure8hv/tLLjXu862NwNEorwX8sn6rUCTM+qNyaOFQp8gIlIOxxSYZDNBvFml1wK0nViyvKfHNo9DcRZtwEWO/Arg2PKzlDXX2rSwmIfUtR97XCYHiv1p1udM3qkjF1DSvt7OKbr2vakACi9IopOBKE69G26v3+7G/FevqBDhDy3O0+xMbtYi1q1VZop8+/W7ch1JoNvDqiqQ0QE67lyJUZhP3cCBbOqjo+2jo4LtKk0RxQApqS164LLVGAn7CilrlzLkYOmkUCzX3vzJcjKCzWvNzZ218wTmp9URN+Y8rRtpmx/vbnscgBGL+1WdcrgthA70DpT2A6DPyPOst2rwat8C9T4oxEFisSvu37d7vus8GrGZHOG2nRfNbYmREU2ZEunbUQue0ehB6p9rNYEn97BnBiUxIwUAUB5DOzTEDVVJ4KWO1cJSgSHo7wvA1HQnqp8D/MKgCxMH1uHodam5PUh3xqzrbL4tCgJY43Zg0hjajErtEFJ9Jq2qXP7Ph4R/f0TV1KtOljFVbQJebooxeyeZtVKpKAgB6BzI99KgO2pp+X7eaNclylEJMznhmKX79zISLzHjqPyyJLbj5pNRWHvJuIaQpcgPi/VvYKZERTrqojb4e1CA1L1wuL6iPHpQVO1JwzfZfyfXz7bkvXzthnxWqnXKks15bcJ7s+MdHsrSBjDiWuZpUZ29oalsI2DFcSLMgUIQ3dXmIaz3rPlbb0GD8MOwrGvqpTZ4MstDAas+8O4D9aST0PPsdyJxI8kNSjrV5pwPJIiFC4V90C2dpK8tcYAfTpP+7egxiExvsfpGPDGkr67vonQOvHVFk3SSfFgEeNg34VY5QjzZdAJOoK/de+ExPwqVpChCC56dPWBp1XN2ny/nDkZ96flC5mGViJq7pyBB1n8Qmvuu2gPCN0Kx9DCmQBNb V8/aMcWJ Mft2p7Sl3rEvqBlpGWyWmYrrtBiDASgjIgu+hMUH1BYOx590XJsuiiicW0GuUEX8aEoacTT3q51/y8nqk+DisNw5SDD1GlTOdzNLJnSEzpbzph9qZ4ATEFFErCv3vHh+pca0xD1vD2DVdThxUFwWzLr1SbWNNiPR63r+4NZ8RgG8kMeQWbA9zsAnYetdegcsuZ4mdegjFEkxZJPRy3O2iZDJltX8Ohzlk6ny6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Break up the list of folios into batches here so that the folios are more likely to be cache hot when doing the rest of the processing. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 8ebb10d44de7..51a613f477a9 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -138,22 +138,26 @@ EXPORT_SYMBOL(__folio_put); */ void put_pages_list(struct list_head *pages) { - struct folio *folio, *next; + struct folio_batch fbatch; + struct folio *folio; - list_for_each_entry_safe(folio, next, pages, lru) { - if (!folio_put_testzero(folio)) { - list_del(&folio->lru); + folio_batch_init(&fbatch); + list_for_each_entry(folio, pages, lru) { + if (!folio_put_testzero(folio)) continue; - } if (folio_test_large(folio)) { - list_del(&folio->lru); __folio_put_large(folio); continue; } /* LRU flag must be clear because it's passed using the lru */ + if (folio_batch_add(&fbatch, folio) > 0) + continue; + free_unref_folios(&fbatch); + fbatch.nr = 0; } - free_unref_page_list(pages); + if (fbatch.nr) + free_unref_folios(&fbatch); INIT_LIST_HEAD(pages); } EXPORT_SYMBOL(put_pages_list);