From patchwork Wed Dec 8 04:22:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19900C433F5 for ; Wed, 8 Dec 2021 06:22:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE3AD6B007B; Wed, 8 Dec 2021 01:19:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D92BA6B0081; Wed, 8 Dec 2021 01:19:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C81CF6B0082; Wed, 8 Dec 2021 01:19:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id B9E116B007B for ; Wed, 8 Dec 2021 01:19:05 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 3063020F71; Wed, 8 Dec 2021 06:17:52 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 1C0EE2074E; Wed, 8 Dec 2021 05:33:53 +0000 (UTC) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8C2682138A for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.20.2E731C3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 3EDE8600198A for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5xJN0Ke/RBge8yAC47u5+/emcH/Klxeoku8OPLSI/Ow=; b=ThqWMSkjkUDbbxstIeqTv1Ubuf YwM7nhdVERRrxUd7hMYHiDCuTMAe3Qkwf2qXobZiRQ8jl7+ZpNH/k3CwZCfUe7WbJ9gK2VREnO25I NEMUuF9ffXWnT7K1N1Q6sh4/PZ6stwtg7KkYVmCIQDAzqayz2Hu3kdembJY/vlMePJiWy0ddt9luT n1UUTM07y2l3x4YlQ28qFh++SVppLXo57Em+jMpxFyrPrCpWYf0CJcHG8Hvx/94hhJjwYMoyu7uI/ HOGQb5/sxBCFE0jBZ+zbWDgeGvmYzP29HmVskZDmCo0Sgx9yCJdwxngSSIX7QxjFtNwtdMguhN3sz GfHrv3Ow==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU1-0084Wh-Hg; Wed, 08 Dec 2021 04:23:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 01/48] filemap: Remove PageHWPoison check from next_uptodate_page() Date: Wed, 8 Dec 2021 04:22:09 +0000 Message-Id: <20211208042256.1923824-2-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3EDE8600198A X-Stat-Signature: 5bdzbod34ar13y6t3boa3s5qgegs8zpi Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ThqWMSkj; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937393-287393 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pages are individually marked as suffering from hardware poisoning. Checking that the head page is not hardware poisoned doesn't make sense; we might be after a subpage. We check each page individually before we use it, so this was an optimisation gone wrong. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index daa0e23a6ee6..39c4c46c6133 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3253,8 +3253,6 @@ static struct page *next_uptodate_page(struct page *page, goto skip; if (!PageUptodate(page) || PageReadahead(page)) goto skip; - if (PageHWPoison(page)) - goto skip; if (!trylock_page(page)) goto skip; if (page->mapping != mapping) From patchwork Wed Dec 8 04:22:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE3D8C433EF for ; Wed, 8 Dec 2021 07:08:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E22D6B0071; Wed, 8 Dec 2021 02:08:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 391296B0073; Wed, 8 Dec 2021 02:08:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 259F66B0075; Wed, 8 Dec 2021 02:08:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay037.a.hostedemail.com [64.99.140.37]) by kanga.kvack.org (Postfix) with ESMTP id 158256B0071 for ; Wed, 8 Dec 2021 02:08:37 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id AE18A801AE for ; Wed, 8 Dec 2021 04:48:17 +0000 (UTC) X-FDA: 78893395434.17.9897E31 Received: from relay.hostedemail.com (relay037.a.hostedemail.com [64.99.140.37]) by imf04.hostedemail.com (Postfix) with ESMTP id 5139340002 for ; Wed, 8 Dec 2021 04:48:17 +0000 (UTC) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E440F20785 for ; Wed, 8 Dec 2021 04:28:54 +0000 (UTC) X-FDA: 78893346588.24.6466BF0 Received: from relay.hostedemail.com (relay027.a.hostedemail.com [64.99.140.27]) by imf20.hostedemail.com (Postfix) with ESMTP id 7C99ED0000A6 for ; Wed, 8 Dec 2021 04:28:54 +0000 (UTC) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 8536A801BA for ; Wed, 8 Dec 2021 04:23:20 +0000 (UTC) X-FDA: 78893332560.08.701E62E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 00E48100006 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FUlAUp7w5xOWEu0PBG82mMLCrwUhe7kQ/xncNJzLm7U=; b=DMjJSFYmluQlnIkBFHPWvaW/RB QxqgFCqgI6c3NhA8TNGyRGJNIK6BrYGeKi0OVz5nC1mCMdAJB2ydtcTMCGw6qTqjhB4SAGc/v3tnH e7BkZc4soBQwtzDPUjnFJ1dq9KyClXXV8jfpKYwJ5SNhV9rguJfYhXiCmDZKARvyq+oAg42AeVUNb e6Tm2vjQR3eeaOLk5AWKuZa2AJA2Sy9wjt7LTyNbdTZkUVoJ9cIS+XoK9j07I0fpNhwpOYtbz9nPu oxbmMJbEz/SZAddCuat0VwV33ETMY40O/rP73Ft0E5u/PZ0hSOAhDLHOYE4FajiyoYylLcRXYrVF0 O4tYchDA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU1-0084Wj-Ki; Wed, 08 Dec 2021 04:23:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 02/48] fs/writeback: Convert inode_switch_wbs_work_fn to folios Date: Wed, 8 Dec 2021 04:22:10 +0000 Message-Id: <20211208042256.1923824-3-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937399-906384 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DMjJSFYm; dmarc=none; spf=none (imf04.hostedemail.com: domain of FILTER%FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.37) smtp.mailfrom=FILTER%FILTER%willy@infradead.org X-HE-Tag-Orig: 1638937734-86095 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5139340002 X-Stat-Signature: pffnomtw89utwua7ps5oznbo44kcc1mw X-HE-Tag: 1638938897-757844 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This gets the statistics correct by modifying the counters by the number of pages in the folio instead of by 1. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- fs/fs-writeback.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 67f0e88eed01..4f680f848c8b 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -372,7 +372,7 @@ static bool inode_do_switch_wbs(struct inode *inode, { struct address_space *mapping = inode->i_mapping; XA_STATE(xas, &mapping->i_pages, 0); - struct page *page; + struct folio *folio; bool switched = false; spin_lock(&inode->i_lock); @@ -389,21 +389,23 @@ static bool inode_do_switch_wbs(struct inode *inode, /* * Count and transfer stats. Note that PAGECACHE_TAG_DIRTY points - * to possibly dirty pages while PAGECACHE_TAG_WRITEBACK points to - * pages actually under writeback. + * to possibly dirty folios while PAGECACHE_TAG_WRITEBACK points to + * folios actually under writeback. */ - xas_for_each_marked(&xas, page, ULONG_MAX, PAGECACHE_TAG_DIRTY) { - if (PageDirty(page)) { - dec_wb_stat(old_wb, WB_RECLAIMABLE); - inc_wb_stat(new_wb, WB_RECLAIMABLE); + xas_for_each_marked(&xas, folio, ULONG_MAX, PAGECACHE_TAG_DIRTY) { + if (folio_test_dirty(folio)) { + long nr = folio_nr_pages(folio); + wb_stat_mod(old_wb, WB_RECLAIMABLE, -nr); + wb_stat_mod(new_wb, WB_RECLAIMABLE, nr); } } xas_set(&xas, 0); - xas_for_each_marked(&xas, page, ULONG_MAX, PAGECACHE_TAG_WRITEBACK) { - WARN_ON_ONCE(!PageWriteback(page)); - dec_wb_stat(old_wb, WB_WRITEBACK); - inc_wb_stat(new_wb, WB_WRITEBACK); + xas_for_each_marked(&xas, folio, ULONG_MAX, PAGECACHE_TAG_WRITEBACK) { + long nr = folio_nr_pages(folio); + WARN_ON_ONCE(!folio_test_writeback(folio)); + wb_stat_mod(old_wb, WB_WRITEBACK, -nr); + wb_stat_mod(new_wb, WB_WRITEBACK, nr); } if (mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK)) { From patchwork Wed Dec 8 04:22:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71025C433F5 for ; Wed, 8 Dec 2021 06:26:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B49876B008A; Wed, 8 Dec 2021 01:19:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF9256B008C; Wed, 8 Dec 2021 01:19:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C1F76B0092; Wed, 8 Dec 2021 01:19:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay032.a.hostedemail.com [64.99.140.32]) by kanga.kvack.org (Postfix) with ESMTP id 8CE8A6B008A for ; Wed, 8 Dec 2021 01:19:37 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 7C39A208F3; Wed, 8 Dec 2021 06:17:49 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 7920920B9D; Wed, 8 Dec 2021 05:33:54 +0000 (UTC) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9C2F6213C4 for ; Wed, 8 Dec 2021 04:28:38 +0000 (UTC) X-FDA: 78893345916.15.10F324F Received: from relay.hostedemail.com (relay034.a.hostedemail.com [64.99.140.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 3FDA4D0369CF for ; Wed, 8 Dec 2021 04:28:38 +0000 (UTC) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1750B20A06 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332266.19.9E807DB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id BE314B000185 for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gbbxmu+s+53RY1++9cSXBb5q+fBsNrDNEdjEeiwvuA8=; b=KHT2l7KWrwSvVWmmvX7PpKnxK1 L8+ad2z1XwLEB+iBzc92Z4LmFXM67tn867+cijQQFdXI2SK/5jA1QXQcTW3hSxM+C45hpOrESrQ4B ro/4VwGFTzcFUFK9qjyUvnndvI0NbL1QWBL61Cc1mlcJM4zG65o9vRwvTFSreAiukTBewdO9W5/uY 34/Mqzb/sUX0MXY2bRZDtEv3xnMmFLrwa+EHLFimUpN749LatBzS/MS3mNypJ+VsHFuYiXCgdgb3r CE5hzDxfh+gbc9pCSY90zXL7rZWAGHbxL8DbfnTxAtioZJKQ3pPPeXgPy9vbU9+djpyw5MKd07zXO fvJoOcWQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU1-0084Wl-Nj; Wed, 08 Dec 2021 04:23:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 03/48] mm/doc: Add documentation for folio_test_uptodate Date: Wed, 8 Dec 2021 04:22:11 +0000 Message-Id: <20211208042256.1923824-4-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KHT2l7KW; dmarc=none; spf=none (imf21.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.34) smtp.mailfrom=FILTER%willy@infradead.org X-HE-Tag-Orig: 1638937391-689692 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3FDA4D0369CF X-Stat-Signature: fizzq649g1an8nhwp5f1gc3mk9ow667w X-HE-Tag: 1638937718-227484 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the PG_uptodate documentation to be documentation for folio_test_uptodate() and expand on it a little. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/page-flags.h | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index b5f14d581113..b3d353d537e2 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -68,9 +68,6 @@ * might lose their PG_swapbacked flag when they simply can be dropped (e.g. as * a result of MADV_FREE). * - * PG_uptodate tells whether the page's contents is valid. When a read - * completes, the page becomes uptodate, unless a disk I/O error happened. - * * PG_referenced, PG_reclaim are used for page reclaim for anonymous and * file-backed pagecache (see mm/vmscan.c). * @@ -615,6 +612,16 @@ TESTPAGEFLAG_FALSE(Ksm, ksm) u64 stable_page_flags(struct page *page); +/** + * folio_test_uptodate - Is this folio up to date? + * @folio: The folio. + * + * The uptodate flag is set on a folio when every byte in the folio is + * at least as new as the corresponding bytes on storage. Anonymous + * and CoW folios are always uptodate. If the folio is not uptodate, + * some of the bytes in it may be; see the is_partially_uptodate() + * address_space operation. + */ static inline bool folio_test_uptodate(struct folio *folio) { bool ret = test_bit(PG_uptodate, folio_flags(folio, 0)); From patchwork Wed Dec 8 04:22:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D811C433EF for ; Wed, 8 Dec 2021 06:28:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57D166B0092; Wed, 8 Dec 2021 01:19:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 52CEB6B0093; Wed, 8 Dec 2021 01:19:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F42C6B0095; Wed, 8 Dec 2021 01:19:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id 301E06B0092 for ; Wed, 8 Dec 2021 01:19:53 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 203B62076A; Wed, 8 Dec 2021 06:17:48 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 82CA721083; Wed, 8 Dec 2021 05:33:55 +0000 (UTC) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4A25B2139B for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.02.DEABB4B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 05B4E20002 for ; Wed, 8 Dec 2021 04:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gVZ3rj3pgutoF/5OlXrYdrDfuJpr8xgCvEFXIoJz1Jc=; b=KP3Iav7r26bF8QCQ/kUMUsXXI6 woSAaYBE1KSsaNOSmtMKTCEhQo4V/eRqoImEZHjipA7BcI3ovCiggyoV5Ix8hHvZDDijEa1srBoPu j+smk4ADhhGCUqFY1Y6BDwoRQVbiqkHCX9P61oPxx5ky/cXfHyP67fEAoibxE1pUQ9HNG/SJFo8+Y jyVVMlv3IGMbCXmv9eE2CB3zI6gcTTkdbd/QiYED02LEVxlI5seeRbx5BWEgtuVEBxoctZzMidyaV hHjREjrP1zvIg+zFSOSuhN4gufcrEn+L9H9tWWbDfrlbqbBO+lMeobm2nzDRdIZG9gIsHb4C8tSZD 7bBYYDxw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU1-0084Wn-QP; Wed, 08 Dec 2021 04:23:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 04/48] mm/writeback: Improve __folio_mark_dirty() comment Date: Wed, 8 Dec 2021 04:22:12 +0000 Message-Id: <20211208042256.1923824-5-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 05B4E20002 X-Stat-Signature: ajfee1h7edinto3mbppuk9rgdhz88d64 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KP3Iav7r; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937392-989626 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add some notes about how this function needs to be called. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/page-writeback.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/page-writeback.c b/mm/page-writeback.c index a613f8ef6a02..91d163f8d36b 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2496,7 +2496,11 @@ void folio_account_cleaned(struct folio *folio, struct address_space *mapping, * If warn is true, then emit a warning if the folio is not uptodate and has * not been truncated. * - * The caller must hold lock_page_memcg(). + * The caller must hold lock_page_memcg(). Most callers have the folio + * locked. A few have the folio blocked from truncation through other + * means (eg zap_page_range() has it mapped and is holding the page table + * lock). This can also be called from mark_buffer_dirty(), which I + * cannot prove is always protected against truncate. */ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping, int warn) From patchwork Wed Dec 8 04:22:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E963C433F5 for ; Wed, 8 Dec 2021 06:23:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D8FC6B0074; Wed, 8 Dec 2021 01:19:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 861BA6B0083; Wed, 8 Dec 2021 01:19:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 729146B0085; Wed, 8 Dec 2021 01:19:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay033.a.hostedemail.com [64.99.140.33]) by kanga.kvack.org (Postfix) with ESMTP id 63D2D6B0074 for ; Wed, 8 Dec 2021 01:19:18 -0500 (EST) Received: by unirelay08.hostedemail.com (Postfix, from userid 108) id 8CD7320454; Wed, 8 Dec 2021 06:17:34 +0000 (UTC) Received: by unirelay08.hostedemail.com (Postfix, from userid 108) id 535FC20657; Wed, 8 Dec 2021 05:33:41 +0000 (UTC) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 94FE92099C for ; Wed, 8 Dec 2021 04:29:12 +0000 (UTC) X-FDA: 78893347302.15.69B0FCB Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by imf31.hostedemail.com (Postfix) with ESMTP id 36DDD1046300 for ; Wed, 8 Dec 2021 04:29:12 +0000 (UTC) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 12F9F21399 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.18.767FDD7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id BB48F600198A for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=XdgFvvcU0VY2BBoR+sckcbNFdIGXaT2EcoM1SoJJkrY=; b=XUM0qSdXd+arG+Gbf5tXz5RcLI uTsXxfSG0ZXk3kWT0GKNdL6llEhBpsiirmxwqZYaB9NjH0yzaLxegDy0DjjBKL9nbDGKV7muJ4fMq VD735KMHN+1bNXdxntF+LkWEBtCxI/lnVX78sp+1b4/8ZF20ajf66yMrjsokMgIThxgL8ZkXdm6RV BYKDBDGraSbUO2YVz0f3nCatfiRMj4T1h0mJ0ttsw/inYwtpHNhmw2HMCapihtuqRQHfPOqai0CCo tMKljSFTdQ0TjngQrc39JrlrNtaQ8x5zTh2hLrWZj6DLpBd4bnSmpfS96cl1+tmcT8yVrEvSkA23N Sh8voY+A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU1-0084Wt-UN; Wed, 08 Dec 2021 04:23:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 05/48] pagevec: Add folio_batch Date: Wed, 8 Dec 2021 04:22:13 +0000 Message-Id: <20211208042256.1923824-6-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 36DDD1046300 X-Stat-Signature: fnrgbkozrttzzebk395mswtzeecrk6ki X-HE-Tag-Orig: 1638937391-250391 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XUM0qSdX; dmarc=none; spf=none (imf31.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.25) smtp.mailfrom=FILTER%willy@infradead.org X-HE-Tag: 1638937752-371728 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The folio_batch is the same as the pagevec, except that it is typed to contain folios and not pages. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagevec.h | 63 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h index 7f3f19065a9f..4483e6ad7607 100644 --- a/include/linux/pagevec.h +++ b/include/linux/pagevec.h @@ -15,6 +15,7 @@ #define PAGEVEC_SIZE 15 struct page; +struct folio; struct address_space; struct pagevec { @@ -81,4 +82,66 @@ static inline void pagevec_release(struct pagevec *pvec) __pagevec_release(pvec); } +/** + * struct folio_batch - A collection of folios. + * + * The folio_batch is used to amortise the cost of retrieving and + * operating on a set of folios. The order of folios in the batch is + * not considered important. Some users of the folio_batch store + * "exceptional" entries in it which can be removed by calling + * folio_batch_remove_exceptionals(). + */ +struct folio_batch { + unsigned char nr; + unsigned char aux[3]; + struct folio *folios[PAGEVEC_SIZE]; +}; + +/** + * folio_batch_init() - Initialise a batch of folios + * @fbatch: The folio batch. + * + * A freshly initialised folio_batch contains zero folios. + */ +static inline void folio_batch_init(struct folio_batch *fbatch) +{ + fbatch->nr = 0; +} + +static inline unsigned int folio_batch_count(struct folio_batch *fbatch) +{ + return fbatch->nr; +} + +static inline unsigned int fbatch_space(struct folio_batch *fbatch) +{ + return PAGEVEC_SIZE - fbatch->nr; +} + +/** + * folio_batch_add() - Add a folio to a batch. + * @fbatch: The folio batch. + * @folio: The folio to add. + * + * The folio is added to the end of the batch. + * The batch must have previously been initialised using folio_batch_init(). + * + * Return: The number of slots still available. + */ +static inline unsigned folio_batch_add(struct folio_batch *fbatch, + struct folio *folio) +{ + fbatch->folios[fbatch->nr++] = folio; + return fbatch_space(fbatch); +} + +static inline void folio_batch_release(struct folio_batch *fbatch) +{ + pagevec_release((struct pagevec *)fbatch); +} + +static inline void folio_batch_remove_exceptionals(struct folio_batch *fbatch) +{ + pagevec_remove_exceptionals((struct pagevec *)fbatch); +} #endif /* _LINUX_PAGEVEC_H */ From patchwork Wed Dec 8 04:22:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4285C433EF for ; Wed, 8 Dec 2021 06:44:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C38666B0071; Wed, 8 Dec 2021 01:43:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B02176B0073; Wed, 8 Dec 2021 01:43:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8929F6B0075; Wed, 8 Dec 2021 01:43:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay037.a.hostedemail.com [64.99.140.37]) by kanga.kvack.org (Postfix) with ESMTP id 6F7A76B0074 for ; Wed, 8 Dec 2021 01:43:35 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 8F490601A9 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332308.14.A5FBA89 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 36AEC100005 for ; Wed, 8 Dec 2021 04:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FDCiLoAhGLqsIRLI+os8SCB3HaHnYoT1HRwaxkznZEo=; b=SUwz0gekB4isoCrRAe5K1ZGvSk OYuTW31exgXmtNmi9vW4IUW0++ptqAaP9fZkdGZfYkixDXZ3tcDh7/OtGSqzMGiYGRaxEm24M3Xrj DfHG/0gJuHAasLwTjvHs+pAZq+0CKLDIDPIZD0wHXxiNH1vqNzVX2X//wsBKygCgKOxUe0i4qU5Rg X5IInjCg+2jJ58TtlY2fknclLxAuoKJJ2vC0eaK6o3u+3ViB7SvpsbLUbjeQWzN+D0fVrkGd640lP Fup2dHKcWSEcw7WgvF1epiGeXjVmlE9opOyZEYFSBq1V6INxNpc/18dfwTL1GsCK0haFOre3HnzLy zHJaJbfw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084X2-1I; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 06/48] iov_iter: Add copy_folio_to_iter() Date: Wed, 8 Dec 2021 04:22:14 +0000 Message-Id: <20211208042256.1923824-7-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 36AEC100005 X-Stat-Signature: saewjfu81qdgeejsyadggin6c6waxkij Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SUwz0gek; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937392-16706 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This wrapper around copy_page_to_iter() works because copy_page_to_iter() handles compound pages correctly. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/uio.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 6350354f97e9..8479cf46b5b1 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -10,6 +10,7 @@ #include struct page; +struct folio; struct pipe_inode_info; struct kvec { @@ -146,6 +147,12 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i); size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i); +static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset, + size_t bytes, struct iov_iter *i) +{ + return copy_page_to_iter((struct page *)folio, offset, bytes, i); +} + static __always_inline __must_check size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { From patchwork Wed Dec 8 04:22:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DF07C433F5 for ; Wed, 8 Dec 2021 06:34:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60D0E6B0074; Wed, 8 Dec 2021 01:28:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BD146B0081; Wed, 8 Dec 2021 01:28:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 484BC6B0082; Wed, 8 Dec 2021 01:28:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay028.a.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 3897E6B0074 for ; Wed, 8 Dec 2021 01:28:32 -0500 (EST) Received: by unirelay02.hostedemail.com (Postfix, from userid 108) id CAB6720F65; Wed, 8 Dec 2021 05:33:01 +0000 (UTC) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E7424212D5 for ; Wed, 8 Dec 2021 04:28:54 +0000 (UTC) X-FDA: 78893346588.22.8C54F0D Received: from forward.hostedemail.com (ofarelay.outbound [64.99.140.10]) by imf28.hostedemail.com (Postfix) with ESMTP id 8E0A490000A2 for ; Wed, 8 Dec 2021 04:28:54 +0000 (UTC) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 4E0A6801B5 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332266.04.A9F8826 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id C8BC590000A0 for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=DYYtBGhjj8cEREZ7h3ka1eCGfC7joF81SMsCjzMNVEU=; b=q76KeqFugBaLOBGAos+7cxdUoj OCg4sRWPuVIfOGIY9eII2Xc0HoUKM3cElnqNxLs7hfC0s+PbdVd6+3Mj3ca3X7YDmM2VoikhlSbDO TXWChAhXEWe9Hi2Qmdtu9YGRqQGDAmiAG751LLRunV1ID27E1Xkk36vD75c4MWBhn6cMtzeu/NomU pIayQ/cFmnvYLcxxDiuRYDTmzc6relojgK4ddwovztMvwf5e+E9N/Nve6LaPmlNHV1slIHPHadIQr keeBu95LYdAsTr97uxzUzeZTbTxnO7qDrzaw8MycGq/SZ5I+lTne7c/N0+uxpZk/feMaw96SXJAxv jjYn41JQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084X8-6e; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 07/48] iov_iter: Convert iter_xarray to use folios Date: Wed, 8 Dec 2021 04:22:15 +0000 Message-Id: <20211208042256.1923824-8-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8E0A490000A2 X-Stat-Signature: i4ixai9x6mkkufbtt3cqrybphhfxptst X-HE-Tag-Orig: 1638937391-488654 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=q76KeqFu; spf=none (imf28.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.10) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638937734-366125 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Take advantage of how kmap_local_folio() works to simplify the loop. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- lib/iov_iter.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 66a740e6e153..03cf43b003a0 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -69,42 +69,39 @@ #define iterate_xarray(i, n, base, len, __off, STEP) { \ __label__ __out; \ size_t __off = 0; \ - struct page *head = NULL; \ + struct folio *folio; \ loff_t start = i->xarray_start + i->iov_offset; \ - unsigned offset = start % PAGE_SIZE; \ pgoff_t index = start / PAGE_SIZE; \ - int j; \ - \ XA_STATE(xas, i->xarray, index); \ \ + len = PAGE_SIZE - offset_in_page(start); \ rcu_read_lock(); \ - xas_for_each(&xas, head, ULONG_MAX) { \ + xas_for_each(&xas, folio, ULONG_MAX) { \ unsigned left; \ - if (xas_retry(&xas, head)) \ + size_t offset = offset_in_folio(folio, start + __off); \ + if (xas_retry(&xas, folio)) \ continue; \ - if (WARN_ON(xa_is_value(head))) \ + if (WARN_ON(xa_is_value(folio))) \ break; \ - if (WARN_ON(PageHuge(head))) \ + if (WARN_ON(folio_test_hugetlb(folio))) \ break; \ - for (j = (head->index < index) ? index - head->index : 0; \ - j < thp_nr_pages(head); j++) { \ - void *kaddr = kmap_local_page(head + j); \ - base = kaddr + offset; \ - len = PAGE_SIZE - offset; \ + while (offset < folio_size(folio)) { \ + base = kmap_local_folio(folio, offset); \ len = min(n, len); \ left = (STEP); \ - kunmap_local(kaddr); \ + kunmap_local(base); \ len -= left; \ __off += len; \ n -= len; \ if (left || n == 0) \ goto __out; \ - offset = 0; \ + offset += len; \ + len = PAGE_SIZE; \ } \ } \ __out: \ rcu_read_unlock(); \ - i->iov_offset += __off; \ + i->iov_offset += __off; \ n = __off; \ } From patchwork Wed Dec 8 04:22:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51DD3C433EF for ; Wed, 8 Dec 2021 06:32:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6AD06B007D; Wed, 8 Dec 2021 01:23:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD72C6B007E; Wed, 8 Dec 2021 01:23:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96CD56B0080; Wed, 8 Dec 2021 01:23:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay034.a.hostedemail.com [64.99.140.34]) by kanga.kvack.org (Postfix) with ESMTP id 8621C6B007D for ; Wed, 8 Dec 2021 01:23:55 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 760B3604E3; Wed, 8 Dec 2021 05:38:44 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 0503B607CE; Wed, 8 Dec 2021 05:32:07 +0000 (UTC) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 47D2960A09 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332350.24.8AEC029 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id B89F5E0016B0 for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Yp6HjXN/gjhhCCnIC8blGjRGkTqUSfJsyt3heH3EC8Q=; b=THifieJp3/uon84a99kjw6IH+n MN7o5exK542sKTyQ+ZuUo1jf4ak7AKw1lsrkntvRfE1J26t+NVueZpBxWeYoPsAIggZpbhJcBGTqa B7u6JazNWOSyrftVJTgDYXQXNQiKc/cqaP+IGI5tl1yX1zOY9a7blWPijuUWqrLr/W595Da2q+a+0 1gFnSTUD6XgbKUx/95U6Wi6pgv9JBRpoTlM1zFoYGDCMM6JrKuJdD8wnDRbOITOo0yjAlBq5CA5i6 B33Z0kIm4CQOfU9KaYt7FocF83GzZ+CY+81kpGuS3ZCfkkBFdSjFAOgHKa8S33nkyHFTHonwW59xZ TfEoC5Bg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084XE-BD; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 08/48] mm: Add folio_test_pmd_mappable() Date: Wed, 8 Dec 2021 04:22:16 +0000 Message-Id: <20211208042256.1923824-9-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 8y6pd5s3kfkab8c48kqjticok51fkcpr Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=THifieJp; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B89F5E0016B0 X-HE-Tag: 1638937391-11926 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a predicate to determine if the folio might be mapped by a PMD entry. If CONFIG_TRANSPARENT_HUGEPAGE is disabled, we know it can't be, even if it's large enough. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/huge_mm.h | 14 ++++++++++++++ include/linux/mm.h | 42 ++++++++++++++++++++--------------------- 2 files changed, 35 insertions(+), 21 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index f280f33ff223..e4c18ba8d3bf 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -274,6 +274,15 @@ static inline int thp_nr_pages(struct page *page) return 1; } +/** + * folio_test_pmd_mappable - Can we map this folio with a PMD? + * @folio: The folio to test + */ +static inline bool folio_test_pmd_mappable(struct folio *folio) +{ + return folio_order(folio) >= HPAGE_PMD_ORDER; +} + struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, @@ -339,6 +348,11 @@ static inline int thp_nr_pages(struct page *page) return 1; } +static inline bool folio_test_pmd_mappable(struct folio *folio) +{ + return false; +} + static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma) { return false; diff --git a/include/linux/mm.h b/include/linux/mm.h index a7e4a9e7d807..72ca04f16711 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -714,6 +714,27 @@ int vma_is_stack_for_current(struct vm_area_struct *vma); struct mmu_gather; struct inode; +static inline unsigned int compound_order(struct page *page) +{ + if (!PageHead(page)) + return 0; + return page[1].compound_order; +} + +/** + * folio_order - The allocation order of a folio. + * @folio: The folio. + * + * A folio is composed of 2^order pages. See get_order() for the definition + * of order. + * + * Return: The order of the folio. + */ +static inline unsigned int folio_order(struct folio *folio) +{ + return compound_order(&folio->page); +} + #include /* @@ -906,27 +927,6 @@ static inline void destroy_compound_page(struct page *page) compound_page_dtors[page[1].compound_dtor](page); } -static inline unsigned int compound_order(struct page *page) -{ - if (!PageHead(page)) - return 0; - return page[1].compound_order; -} - -/** - * folio_order - The allocation order of a folio. - * @folio: The folio. - * - * A folio is composed of 2^order pages. See get_order() for the definition - * of order. - * - * Return: The order of the folio. - */ -static inline unsigned int folio_order(struct folio *folio) -{ - return compound_order(&folio->page); -} - static inline bool hpage_pincount_available(struct page *page) { /* From patchwork Wed Dec 8 04:22:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7666C433EF for ; Wed, 8 Dec 2021 06:47:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94F846B007D; Wed, 8 Dec 2021 01:43:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DD9E6B0082; Wed, 8 Dec 2021 01:43:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6471F6B007E; Wed, 8 Dec 2021 01:43:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by kanga.kvack.org (Postfix) with ESMTP id 458526B0080 for ; Wed, 8 Dec 2021 01:43:37 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 786F81201C3 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332308.21.8FD2B27 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id DCA54F00008F for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=q7qEbqnJ5RUpHWaUhfXKj7MwXRK65W02Wk/Lkz0SF+4=; b=GancLpAeAyY7nvDVkXACF9qJa4 krZ1QlZRrR/yK0Z/87T2oirxcg/QArqFxyD1oAn0iGaXHE5vt0OZI0XRGVMuoXiJw+vGV1wQISIci M07sLdYrt2d3KDOG3vFHK7Wi04vVQo3TC8WJPvYXxNhlkrQcP3KZop4f4GDZI6WjvXw0LXb+x+C1Q 2gOZcWPDB6ffn9HrwipFsnuqgCqg3p8rJERS9XuC8sSz9FDNn+cTSWMRBswUnWJbYOD6f6+A+GTgP 3kx5sKRYeJh8EYOgYI7LXuZj039zdqG3zWLCrsnMV8fiBJ/XEX5VuesinO3uvbup0woeO9qKklpAe v3ad9+NA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084XK-F3; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 09/48] filemap: Add folio_put_wait_locked() Date: Wed, 8 Dec 2021 04:22:17 +0000 Message-Id: <20211208042256.1923824-10-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DCA54F00008F X-Stat-Signature: 9ngryn3r153jsqp61uuxeibpdb1jt35z Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GancLpAe; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937391-550305 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert all three callers of put_and_wait_on_page_locked() to folio_put_wait_locked(). This shrinks the kernel overall by 19 bytes. filemap_update_page() shrinks by 19 bytes while __migration_entry_wait() is unchanged. folio_put_wait_locked() is 14 bytes smaller than put_and_wait_on_page_locked(), but pmd_migration_entry_wait() grows by 14 bytes. It removes the assumption from pmd_migration_entry_wait() that pages cannot be larger than a PMD (which is true today, but may be interesting to explore in the future). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/pagemap.h | 2 +- mm/filemap.c | 27 +++++++++++++++------------ mm/migrate.c | 21 ++++++++++----------- 3 files changed, 26 insertions(+), 24 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 605246452305..841f7ba62d7d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -868,7 +868,7 @@ static inline int wait_on_page_locked_killable(struct page *page) return folio_wait_locked_killable(page_folio(page)); } -int put_and_wait_on_page_locked(struct page *page, int state); +int folio_put_wait_locked(struct folio *folio, int state); void wait_on_page_writeback(struct page *page); void folio_wait_writeback(struct folio *folio); int folio_wait_writeback_killable(struct folio *folio); diff --git a/mm/filemap.c b/mm/filemap.c index 39c4c46c6133..5dd3c6e39c9f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1259,10 +1259,10 @@ enum behavior { * __folio_lock() waiting on then setting PG_locked. */ SHARED, /* Hold ref to page and check the bit when woken, like - * wait_on_page_writeback() waiting on PG_writeback. + * folio_wait_writeback() waiting on PG_writeback. */ DROP, /* Drop ref to page before wait, no check when woken, - * like put_and_wait_on_page_locked() on PG_locked. + * like folio_put_wait_locked() on PG_locked. */ }; @@ -1439,22 +1439,21 @@ int folio_wait_bit_killable(struct folio *folio, int bit_nr) EXPORT_SYMBOL(folio_wait_bit_killable); /** - * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked - * @page: The page to wait for. + * folio_put_wait_locked - Drop a reference and wait for it to be unlocked + * @folio: The folio to wait for. * @state: The sleep state (TASK_KILLABLE, TASK_UNINTERRUPTIBLE, etc). * - * The caller should hold a reference on @page. They expect the page to + * The caller should hold a reference on @folio. They expect the page to * become unlocked relatively soon, but do not wish to hold up migration - * (for example) by holding the reference while waiting for the page to + * (for example) by holding the reference while waiting for the folio to * come unlocked. After this function returns, the caller should not - * dereference @page. + * dereference @folio. * - * Return: 0 if the page was unlocked or -EINTR if interrupted by a signal. + * Return: 0 if the folio was unlocked or -EINTR if interrupted by a signal. */ -int put_and_wait_on_page_locked(struct page *page, int state) +int folio_put_wait_locked(struct folio *folio, int state) { - return folio_wait_bit_common(page_folio(page), PG_locked, state, - DROP); + return folio_wait_bit_common(folio, PG_locked, state, DROP); } /** @@ -2447,7 +2446,11 @@ static int filemap_update_page(struct kiocb *iocb, goto unlock_mapping; if (!(iocb->ki_flags & IOCB_WAITQ)) { filemap_invalidate_unlock_shared(mapping); - put_and_wait_on_page_locked(&folio->page, TASK_KILLABLE); + /* + * This is where we usually end up waiting for a + * previously submitted readahead to finish. + */ + folio_put_wait_locked(folio, TASK_KILLABLE); return AOP_TRUNCATED_PAGE; } error = __folio_lock_async(folio, iocb->ki_waitq); diff --git a/mm/migrate.c b/mm/migrate.c index cf25b00f03c8..311638177536 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -291,7 +291,7 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, { pte_t pte; swp_entry_t entry; - struct page *page; + struct folio *folio; spin_lock(ptl); pte = *ptep; @@ -302,18 +302,17 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, if (!is_migration_entry(entry)) goto out; - page = pfn_swap_entry_to_page(entry); - page = compound_head(page); + folio = page_folio(pfn_swap_entry_to_page(entry)); /* * Once page cache replacement of page migration started, page_count - * is zero; but we must not call put_and_wait_on_page_locked() without - * a ref. Use get_page_unless_zero(), and just fault again if it fails. + * is zero; but we must not call folio_put_wait_locked() without + * a ref. Use folio_try_get(), and just fault again if it fails. */ - if (!get_page_unless_zero(page)) + if (!folio_try_get(folio)) goto out; pte_unmap_unlock(ptep, ptl); - put_and_wait_on_page_locked(page, TASK_UNINTERRUPTIBLE); + folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE); return; out: pte_unmap_unlock(ptep, ptl); @@ -338,16 +337,16 @@ void migration_entry_wait_huge(struct vm_area_struct *vma, void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd) { spinlock_t *ptl; - struct page *page; + struct folio *folio; ptl = pmd_lock(mm, pmd); if (!is_pmd_migration_entry(*pmd)) goto unlock; - page = pfn_swap_entry_to_page(pmd_to_swp_entry(*pmd)); - if (!get_page_unless_zero(page)) + folio = page_folio(pfn_swap_entry_to_page(pmd_to_swp_entry(*pmd))); + if (!folio_try_get(folio)) goto unlock; spin_unlock(ptl); - put_and_wait_on_page_locked(page, TASK_UNINTERRUPTIBLE); + folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE); return; unlock: spin_unlock(ptl); From patchwork Wed Dec 8 04:22:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6239C433F5 for ; Wed, 8 Dec 2021 07:53:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24F5B6B0071; Wed, 8 Dec 2021 02:53:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FF686B0073; Wed, 8 Dec 2021 02:53:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A0D96B0074; Wed, 8 Dec 2021 02:53:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id EB5BF6B0071 for ; Wed, 8 Dec 2021 02:53:35 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 5697360184 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332350.23.BF0D943 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id F161DD0000A6 for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rF8B5Bv6xTxudXTmrHELyzSGUCagkpzrcf/781BtVxg=; b=WC2Sw82qHt3NL9eKhvdbIcJ7jZ VkibfBA7ok4kr0WMAK06JK0P1a2F8FqP19vz3guLQBJUB1RWoiyI6lRNPfLfXXMxNmE9/0c+qUDyQ S7vW5ke/npS6XKfy9zpubgmniy69pd/3IRAviL6y1q27zJgGdVpFggZkhUc8l2ak1WtJ80ybjtUS0 2Z4tUgVFWzIgIRH917dlx6/G3OyAvoOJqkW9Bj1fe5RpduWzkcwD71I/RJG+nIaJga8RtPgyqmdw4 6RCfB4FFrZCmojNhNFzY2PCm+mxalOacqoUdLltDCVc1IWG+QmfAnNexZg0VOH1vTwsmfhnW50Jqw UrIXsEXw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084XQ-KT; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 10/48] filemap: Convert page_cache_delete to take a folio Date: Wed, 8 Dec 2021 04:22:18 +0000 Message-Id: <20211208042256.1923824-11-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: F161DD0000A6 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WC2Sw82q; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam04 X-Stat-Signature: bq4ekh3cjaij74a3wq5f535dm68h4art X-HE-Tag: 1638937391-469903 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It was already assuming a head page, so this is a straightforward conversion. Convert the one caller to call page_folio(), even though it must currently be passing in a head page. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 5dd3c6e39c9f..38fb26e16b85 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -121,27 +121,26 @@ */ static void page_cache_delete(struct address_space *mapping, - struct page *page, void *shadow) + struct folio *folio, void *shadow) { - XA_STATE(xas, &mapping->i_pages, page->index); - unsigned int nr = 1; + XA_STATE(xas, &mapping->i_pages, folio->index); + long nr = 1; mapping_set_update(&xas, mapping); /* hugetlb pages are represented by a single entry in the xarray */ - if (!PageHuge(page)) { - xas_set_order(&xas, page->index, compound_order(page)); - nr = compound_nr(page); + if (!folio_test_hugetlb(folio)) { + xas_set_order(&xas, folio->index, folio_order(folio)); + nr = folio_nr_pages(folio); } - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageTail(page), page); - VM_BUG_ON_PAGE(nr != 1 && shadow, page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(nr != 1 && shadow, folio); xas_store(&xas, shadow); xas_init_marks(&xas); - page->mapping = NULL; + folio->mapping = NULL; /* Leave page->index set: truncation lookup relies upon it */ mapping->nrpages -= nr; } @@ -223,12 +222,13 @@ static void unaccount_page_cache_page(struct address_space *mapping, */ void __delete_from_page_cache(struct page *page, void *shadow) { + struct folio *folio = page_folio(page); struct address_space *mapping = page->mapping; trace_mm_filemap_delete_from_page_cache(page); unaccount_page_cache_page(mapping, page); - page_cache_delete(mapping, page, shadow); + page_cache_delete(mapping, folio, shadow); } static void page_cache_free_page(struct address_space *mapping, From patchwork Wed Dec 8 04:22:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 001DBC433EF for ; Wed, 8 Dec 2021 06:27:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6422C6B008C; Wed, 8 Dec 2021 01:19:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F0436B0092; Wed, 8 Dec 2021 01:19:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B7706B0093; Wed, 8 Dec 2021 01:19:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay027.a.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 3C57C6B008C for ; Wed, 8 Dec 2021 01:19:50 -0500 (EST) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 0F0A54CD; Wed, 8 Dec 2021 06:18:01 +0000 (UTC) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 111064BE; Wed, 8 Dec 2021 05:34:08 +0000 (UTC) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D68632C0 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332308.27.A118561 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 2B7901044801 for ; Wed, 8 Dec 2021 04:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=4psUQu6RuoXUhrQ0CbS9UHTUdKcuRu/NAS8iw81NOmc=; b=uvvHK4RKM9GjSjDbGwABrt6EAG aO/qS04z5yWmrttjUhTXicFRC3TEHCj0FSTWlvYM+n4LYgJEyvQGOwHCnX2jtcdkCNb8NyqIYD4Fp u/2P8DKGgOxtLxlBNZD6gUCkOB1KcVQZEMBjVwqV2uKi1pBmXlcWiMZ5bWErhQVy4aaO7YC8k7SyY rXR6GXmAaVkexfRsFb2EX30H3mkiJ6vOb+z4Prazxvla4K13aaJsqAwAahgF7/lNIRw9vHODSOm5i m2b9HPv9pn/6dGgnZT02QUlCTvchrKYQTPedFRuhHFUuU2A0DsPjNl7BcfzQ8ZnR1B3RSEaBYS1Zp Ke25fDLQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084XW-OH; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 11/48] filemap: Add filemap_unaccount_folio() Date: Wed, 8 Dec 2021 04:22:19 +0000 Message-Id: <20211208042256.1923824-12-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2B7901044801 X-Stat-Signature: mw34sw76uwe74uo8iz3tfo7gasez8e31 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uvvHK4RK; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937391-831611 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace unaccount_page_cache_page() with filemap_unaccount_folio(). The bug handling path could be a bit more robust (eg taking into account the mapcounts of tail pages), but it's really never supposed to happen. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/pagemap.h | 5 --- mm/filemap.c | 70 ++++++++++++++++++++--------------------- 2 files changed, 35 insertions(+), 40 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 841f7ba62d7d..077b6f378666 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -884,11 +884,6 @@ static inline void __set_page_dirty(struct page *page, } void folio_account_cleaned(struct folio *folio, struct address_space *mapping, struct bdi_writeback *wb); -static inline void account_page_cleaned(struct page *page, - struct address_space *mapping, struct bdi_writeback *wb) -{ - return folio_account_cleaned(page_folio(page), mapping, wb); -} void __folio_cancel_dirty(struct folio *folio); static inline void folio_cancel_dirty(struct folio *folio) { diff --git a/mm/filemap.c b/mm/filemap.c index 38fb26e16b85..600b8c921a67 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -145,74 +145,74 @@ static void page_cache_delete(struct address_space *mapping, mapping->nrpages -= nr; } -static void unaccount_page_cache_page(struct address_space *mapping, - struct page *page) +static void filemap_unaccount_folio(struct address_space *mapping, + struct folio *folio) { - int nr; + long nr; /* * if we're uptodate, flush out into the cleancache, otherwise * invalidate any existing cleancache entries. We can't leave * stale data around in the cleancache once our page is gone */ - if (PageUptodate(page) && PageMappedToDisk(page)) - cleancache_put_page(page); + if (folio_test_uptodate(folio) && folio_test_mappedtodisk(folio)) + cleancache_put_page(&folio->page); else - cleancache_invalidate_page(mapping, page); + cleancache_invalidate_page(mapping, &folio->page); - VM_BUG_ON_PAGE(PageTail(page), page); - VM_BUG_ON_PAGE(page_mapped(page), page); - if (!IS_ENABLED(CONFIG_DEBUG_VM) && unlikely(page_mapped(page))) { + VM_BUG_ON_FOLIO(folio_mapped(folio), folio); + if (!IS_ENABLED(CONFIG_DEBUG_VM) && unlikely(folio_mapped(folio))) { int mapcount; pr_alert("BUG: Bad page cache in process %s pfn:%05lx\n", - current->comm, page_to_pfn(page)); - dump_page(page, "still mapped when deleted"); + current->comm, folio_pfn(folio)); + dump_page(&folio->page, "still mapped when deleted"); dump_stack(); add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); - mapcount = page_mapcount(page); + mapcount = page_mapcount(&folio->page); if (mapping_exiting(mapping) && - page_count(page) >= mapcount + 2) { + folio_ref_count(folio) >= mapcount + 2) { /* * All vmas have already been torn down, so it's - * a good bet that actually the page is unmapped, + * a good bet that actually the folio is unmapped, * and we'd prefer not to leak it: if we're wrong, * some other bad page check should catch it later. */ - page_mapcount_reset(page); - page_ref_sub(page, mapcount); + page_mapcount_reset(&folio->page); + folio_ref_sub(folio, mapcount); } } - /* hugetlb pages do not participate in page cache accounting. */ - if (PageHuge(page)) + /* hugetlb folios do not participate in page cache accounting. */ + if (folio_test_hugetlb(folio)) return; - nr = thp_nr_pages(page); + nr = folio_nr_pages(folio); - __mod_lruvec_page_state(page, NR_FILE_PAGES, -nr); - if (PageSwapBacked(page)) { - __mod_lruvec_page_state(page, NR_SHMEM, -nr); - if (PageTransHuge(page)) - __mod_lruvec_page_state(page, NR_SHMEM_THPS, -nr); - } else if (PageTransHuge(page)) { - __mod_lruvec_page_state(page, NR_FILE_THPS, -nr); + __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, -nr); + if (folio_test_swapbacked(folio)) { + __lruvec_stat_mod_folio(folio, NR_SHMEM, -nr); + if (folio_test_pmd_mappable(folio)) + __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr); + } else if (folio_test_pmd_mappable(folio)) { + __lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); } /* - * At this point page must be either written or cleaned by - * truncate. Dirty page here signals a bug and loss of + * At this point folio must be either written or cleaned by + * truncate. Dirty folio here signals a bug and loss of * unwritten data. * - * This fixes dirty accounting after removing the page entirely - * but leaves PageDirty set: it has no effect for truncated - * page and anyway will be cleared before returning page into + * This fixes dirty accounting after removing the folio entirely + * but leaves the dirty flag set: it has no effect for truncated + * folio and anyway will be cleared before returning folio to * buddy allocator. */ - if (WARN_ON_ONCE(PageDirty(page))) - account_page_cleaned(page, mapping, inode_to_wb(mapping->host)); + if (WARN_ON_ONCE(folio_test_dirty(folio))) + folio_account_cleaned(folio, mapping, + inode_to_wb(mapping->host)); } /* @@ -227,7 +227,7 @@ void __delete_from_page_cache(struct page *page, void *shadow) trace_mm_filemap_delete_from_page_cache(page); - unaccount_page_cache_page(mapping, page); + filemap_unaccount_folio(mapping, folio); page_cache_delete(mapping, folio, shadow); } @@ -348,7 +348,7 @@ void delete_from_page_cache_batch(struct address_space *mapping, for (i = 0; i < pagevec_count(pvec); i++) { trace_mm_filemap_delete_from_page_cache(pvec->pages[i]); - unaccount_page_cache_page(mapping, pvec->pages[i]); + filemap_unaccount_folio(mapping, page_folio(pvec->pages[i])); } page_cache_delete_batch(mapping, pvec); xa_unlock_irq(&mapping->i_pages); From patchwork Wed Dec 8 04:22:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09FB7C433F5 for ; Wed, 8 Dec 2021 06:18:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B73506B0071; Wed, 8 Dec 2021 01:17:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B23126B0075; Wed, 8 Dec 2021 01:17:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EB046B007B; Wed, 8 Dec 2021 01:17:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id 8FBE86B0071 for ; Wed, 8 Dec 2021 01:17:58 -0500 (EST) Received: by unirelay07.hostedemail.com (Postfix, from userid 108) id 3EE822049F; Wed, 8 Dec 2021 06:17:20 +0000 (UTC) Received: by unirelay07.hostedemail.com (Postfix, from userid 108) id 9F6BE206D3; Wed, 8 Dec 2021 05:33:29 +0000 (UTC) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A60D920A20 for ; Wed, 8 Dec 2021 04:29:09 +0000 (UTC) X-FDA: 78893347176.04.BA31A3C Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by imf28.hostedemail.com (Postfix) with ESMTP id 4D717900009E for ; Wed, 8 Dec 2021 04:29:09 +0000 (UTC) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 84C832139D for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332350.16.BEAF10B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 329224002087 for ; Wed, 8 Dec 2021 04:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cP2Ivi6zK+Qa/y07ReyASR7gp8ZaFUr3Vz/PfLjsdlA=; b=oaDJ4ZchXakogTE0J6WfBOAIid pMhFmwpKVVIjfyF9Sc3lGv7TJOLM3KfjA1wgiTaLX8NqFTJVv87mm7Lj3qa7jHpRqmIyRLkGMHsD+ FyhXyy9/aBUM9GHEDjBsG1vKYklAYUjB6d80YvTX/VnmIfztlJfElaQxmmaADFYhge0+OBEBalgBw fNGuE92hsgAnb/9/RkOjCEcV9aUwVQ6hlxpK2/6J/6x/rv1UGQVVuchJizZz39UrsO9N26qPGlnY4 7w2V5EcQYjl2ohAUET50Nzwvi4TkaBUiEQpDhuICGeA+qT+obZgNCDiMkbz5Ce5jCNFaY2UF+aNjq fgrABO4Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084Xc-Rn; Wed, 08 Dec 2021 04:23:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 12/48] filemap: Convert tracing of page cache operations to folio Date: Wed, 8 Dec 2021 04:22:20 +0000 Message-Id: <20211208042256.1923824-13-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oaDJ4Zch; dmarc=none; spf=none (imf28.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.25) smtp.mailfrom=FILTER%willy@infradead.org X-HE-Tag-Orig: 1638937392-851981 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4D717900009E X-Stat-Signature: dbrqbofob6bzcim185ct65yatdnkrnpj X-HE-Tag: 1638937749-452358 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pass the folio instead of a page. The page was already implicitly a folio as it accessed page->mapping directly. Add the order of the folio to the tracepoint, as this is important information. Also drop printing the address of the struct page as the pfn provides better information than the struct page address. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/trace/events/filemap.h | 32 +++++++++++++++++--------------- mm/filemap.c | 9 +++++---- 2 files changed, 22 insertions(+), 19 deletions(-) diff --git a/include/trace/events/filemap.h b/include/trace/events/filemap.h index c47b63db124e..46c89c1e460c 100644 --- a/include/trace/events/filemap.h +++ b/include/trace/events/filemap.h @@ -15,43 +15,45 @@ DECLARE_EVENT_CLASS(mm_filemap_op_page_cache, - TP_PROTO(struct page *page), + TP_PROTO(struct folio *folio), - TP_ARGS(page), + TP_ARGS(folio), TP_STRUCT__entry( __field(unsigned long, pfn) __field(unsigned long, i_ino) __field(unsigned long, index) __field(dev_t, s_dev) + __field(unsigned char, order) ), TP_fast_assign( - __entry->pfn = page_to_pfn(page); - __entry->i_ino = page->mapping->host->i_ino; - __entry->index = page->index; - if (page->mapping->host->i_sb) - __entry->s_dev = page->mapping->host->i_sb->s_dev; + __entry->pfn = folio_pfn(folio); + __entry->i_ino = folio->mapping->host->i_ino; + __entry->index = folio->index; + if (folio->mapping->host->i_sb) + __entry->s_dev = folio->mapping->host->i_sb->s_dev; else - __entry->s_dev = page->mapping->host->i_rdev; + __entry->s_dev = folio->mapping->host->i_rdev; + __entry->order = folio_order(folio); ), - TP_printk("dev %d:%d ino %lx page=%p pfn=0x%lx ofs=%lu", + TP_printk("dev %d:%d ino %lx pfn=0x%lx ofs=%lu order=%u", MAJOR(__entry->s_dev), MINOR(__entry->s_dev), __entry->i_ino, - pfn_to_page(__entry->pfn), __entry->pfn, - __entry->index << PAGE_SHIFT) + __entry->index << PAGE_SHIFT, + __entry->order) ); DEFINE_EVENT(mm_filemap_op_page_cache, mm_filemap_delete_from_page_cache, - TP_PROTO(struct page *page), - TP_ARGS(page) + TP_PROTO(struct folio *folio), + TP_ARGS(folio) ); DEFINE_EVENT(mm_filemap_op_page_cache, mm_filemap_add_to_page_cache, - TP_PROTO(struct page *page), - TP_ARGS(page) + TP_PROTO(struct folio *folio), + TP_ARGS(folio) ); TRACE_EVENT(filemap_set_wb_err, diff --git a/mm/filemap.c b/mm/filemap.c index 600b8c921a67..bcdc8bb4d2c8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -225,7 +225,7 @@ void __delete_from_page_cache(struct page *page, void *shadow) struct folio *folio = page_folio(page); struct address_space *mapping = page->mapping; - trace_mm_filemap_delete_from_page_cache(page); + trace_mm_filemap_delete_from_page_cache(folio); filemap_unaccount_folio(mapping, folio); page_cache_delete(mapping, folio, shadow); @@ -346,9 +346,10 @@ void delete_from_page_cache_batch(struct address_space *mapping, spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); for (i = 0; i < pagevec_count(pvec); i++) { - trace_mm_filemap_delete_from_page_cache(pvec->pages[i]); + struct folio *folio = page_folio(pvec->pages[i]); - filemap_unaccount_folio(mapping, page_folio(pvec->pages[i])); + trace_mm_filemap_delete_from_page_cache(folio); + filemap_unaccount_folio(mapping, folio); } page_cache_delete_batch(mapping, pvec); xa_unlock_irq(&mapping->i_pages); @@ -959,7 +960,7 @@ noinline int __filemap_add_folio(struct address_space *mapping, goto error; } - trace_mm_filemap_add_to_page_cache(&folio->page); + trace_mm_filemap_add_to_page_cache(folio); return 0; error: folio->mapping = NULL; From patchwork Wed Dec 8 04:22:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27100C433F5 for ; Wed, 8 Dec 2021 06:25:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBA206B0089; Wed, 8 Dec 2021 01:19:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D69676B008A; Wed, 8 Dec 2021 01:19:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C31B46B008C; Wed, 8 Dec 2021 01:19:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay040.a.hostedemail.com [64.99.140.40]) by kanga.kvack.org (Postfix) with ESMTP id B409D6B0089 for ; Wed, 8 Dec 2021 01:19:28 -0500 (EST) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 640E757B; Wed, 8 Dec 2021 06:18:03 +0000 (UTC) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 3DE373D9; Wed, 8 Dec 2021 05:34:12 +0000 (UTC) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id ADF96729 for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) X-FDA: 78893332266.01.FDD0DA8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 29A3220019C3 for ; Wed, 8 Dec 2021 04:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bWi9+2uQzBTaSMB6h0/ecOVlvYNdI+v4mVQ2AUCcldM=; b=sZ4pemu2GBittKqLPFjpBwXiqd OhZjGKi13MmC9rulQFxb3e/xlmgKhuwZ8hokT9AAcdQPZImlDOOddhwrLv4f2IW61eeRpgBNbkiXD FDN19/rmysP9uaoQ3oi8OvU0UYT2A5zR6uV5Axm8yrrdmRWrlU4gFcUtpUez38P/ufGa7K38Dnif0 DywutKJs8MmUxC2UqU8W69U6o08ycbeHafs6GMNN7Cz4AMSHm/vhgJAXp27l73vA3OvvGIqrIJOsn 9siiMw7OgD4pSQhU5/CVse4CgK19mFnvNWvHUwcKSE95t76+faAITjJ4Gpx2fnCH69KY/2T5mepp2 uzUAewzw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU2-0084Xi-VK; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 13/48] filemap: Add filemap_remove_folio and __filemap_remove_folio Date: Wed, 8 Dec 2021 04:22:21 +0000 Message-Id: <20211208042256.1923824-14-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 29A3220019C3 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sZ4pemu2; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam04 X-Stat-Signature: z6readubawbwja8zw97jor5394nwyjea X-HE-Tag: 1638937392-579273 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reimplement __delete_from_page_cache() as a wrapper around __filemap_remove_folio() and delete_from_page_cache() as a wrapper around filemap_remove_folio(). Remove the EXPORT_SYMBOL as delete_from_page_cache() was not used by any in-tree modules. Convert page_cache_free_page() into filemap_free_folio(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/pagemap.h | 9 +++++++-- mm/filemap.c | 43 +++++++++++++++++++---------------------- mm/folio-compat.c | 5 +++++ 3 files changed, 32 insertions(+), 25 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 077b6f378666..3f26b191ede3 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -930,8 +930,13 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping, pgoff_t index, gfp_t gfp); int filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp); -extern void delete_from_page_cache(struct page *page); -extern void __delete_from_page_cache(struct page *page, void *shadow); +void filemap_remove_folio(struct folio *folio); +void delete_from_page_cache(struct page *page); +void __filemap_remove_folio(struct folio *folio, void *shadow); +static inline void __delete_from_page_cache(struct page *page, void *shadow) +{ + __filemap_remove_folio(page_folio(page), shadow); +} void replace_page_cache_page(struct page *old, struct page *new); void delete_from_page_cache_batch(struct address_space *mapping, struct pagevec *pvec); diff --git a/mm/filemap.c b/mm/filemap.c index bcdc8bb4d2c8..4fe845b30f33 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -220,58 +220,55 @@ static void filemap_unaccount_folio(struct address_space *mapping, * sure the page is locked and that nobody else uses it - or that usage * is safe. The caller must hold the i_pages lock. */ -void __delete_from_page_cache(struct page *page, void *shadow) +void __filemap_remove_folio(struct folio *folio, void *shadow) { - struct folio *folio = page_folio(page); - struct address_space *mapping = page->mapping; + struct address_space *mapping = folio->mapping; trace_mm_filemap_delete_from_page_cache(folio); - filemap_unaccount_folio(mapping, folio); page_cache_delete(mapping, folio, shadow); } -static void page_cache_free_page(struct address_space *mapping, - struct page *page) +static void filemap_free_folio(struct address_space *mapping, + struct folio *folio) { void (*freepage)(struct page *); freepage = mapping->a_ops->freepage; if (freepage) - freepage(page); + freepage(&folio->page); - if (PageTransHuge(page) && !PageHuge(page)) { - page_ref_sub(page, thp_nr_pages(page)); - VM_BUG_ON_PAGE(page_count(page) <= 0, page); + if (folio_test_large(folio) && !folio_test_hugetlb(folio)) { + folio_ref_sub(folio, folio_nr_pages(folio)); + VM_BUG_ON_FOLIO(folio_ref_count(folio) <= 0, folio); } else { - put_page(page); + folio_put(folio); } } /** - * delete_from_page_cache - delete page from page cache - * @page: the page which the kernel is trying to remove from page cache + * filemap_remove_folio - Remove folio from page cache. + * @folio: The folio. * - * This must be called only on pages that have been verified to be in the page - * cache and locked. It will never put the page into the free list, the caller - * has a reference on the page. + * This must be called only on folios that are locked and have been + * verified to be in the page cache. It will never put the folio into + * the free list because the caller has a reference on the page. */ -void delete_from_page_cache(struct page *page) +void filemap_remove_folio(struct folio *folio) { - struct address_space *mapping = page_mapping(page); + struct address_space *mapping = folio->mapping; - BUG_ON(!PageLocked(page)); + BUG_ON(!folio_test_locked(folio)); spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); - __delete_from_page_cache(page, NULL); + __filemap_remove_folio(folio, NULL); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock); - page_cache_free_page(mapping, page); + filemap_free_folio(mapping, folio); } -EXPORT_SYMBOL(delete_from_page_cache); /* * page_cache_delete_batch - delete several pages from page cache @@ -358,7 +355,7 @@ void delete_from_page_cache_batch(struct address_space *mapping, spin_unlock(&mapping->host->i_lock); for (i = 0; i < pagevec_count(pvec); i++) - page_cache_free_page(mapping, pvec->pages[i]); + filemap_free_folio(mapping, page_folio(pvec->pages[i])); } int filemap_check_errors(struct address_space *mapping) diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 5b6ae1da314e..749a695b4217 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -140,3 +140,8 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, mapping_gfp_mask(mapping)); } EXPORT_SYMBOL(grab_cache_page_write_begin); + +void delete_from_page_cache(struct page *page) +{ + return filemap_remove_folio(page_folio(page)); +} From patchwork Wed Dec 8 04:22:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 374F3C433EF for ; Wed, 8 Dec 2021 06:31:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4665F6B007B; Wed, 8 Dec 2021 01:23:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EF056B007D; Wed, 8 Dec 2021 01:23:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 268DC6B007E; Wed, 8 Dec 2021 01:23:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay028.a.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 15A686B007B for ; Wed, 8 Dec 2021 01:23:55 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 7C7F460372; Wed, 8 Dec 2021 05:38:46 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id BD190605BE; Wed, 8 Dec 2021 05:32:09 +0000 (UTC) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 698DA60A06 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.30.4980560 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id EC132900024B for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gW/w9LUlR0JAZR+iSci5RbcImjtS3JPKQVeQeZjtBvI=; b=qmJDICws1QAL8J1sTd6QL6qzYI SCX8SLmn6qkuqbZ4yxc7aED8oSWtiR47EQRuEzsb6DEzqKN42jhVFD/fFizFs896Qf1EhQMJB8MAp 9E4Hp027ri0nLEPAUuURkLqHsmfJ8Hv0zexLyQV32uKmSIVgxVBqgenyGJT2ECVZnr3OrUwy5zax9 9zqtPVLd/fVOYS9kSWT+gpj6SIL7jgcDVccF41XseFGvib8YXZMiM+fkLeRX1JclTn6lO2NoCp6pb uFOg1W0cHpERnCiimr+u4QhgYhs8lsJfINJITceo4Xzzomi3lgSA7rFdQ3bqL8eJ3++lZqUmitpDV hj4fJfzw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084Xo-2v; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 14/48] filemap: Convert find_get_entry to return a folio Date: Wed, 8 Dec 2021 04:22:22 +0000 Message-Id: <20211208042256.1923824-15-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 8neiea5cyxuqecb4anwihhyxf3ao3mj6 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qmJDICws; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EC132900024B X-HE-Tag: 1638937393-841891 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert callers to cope. Saves 580 bytes of kernel text; all five callers are reduced in size. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 146 +++++++++++++++++++++++++-------------------------- 1 file changed, 72 insertions(+), 74 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 4fe845b30f33..2a51ec720e9e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1976,37 +1976,36 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, } EXPORT_SYMBOL(__filemap_get_folio); -static inline struct page *find_get_entry(struct xa_state *xas, pgoff_t max, +static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max, xa_mark_t mark) { - struct page *page; + struct folio *folio; retry: if (mark == XA_PRESENT) - page = xas_find(xas, max); + folio = xas_find(xas, max); else - page = xas_find_marked(xas, max, mark); + folio = xas_find_marked(xas, max, mark); - if (xas_retry(xas, page)) + if (xas_retry(xas, folio)) goto retry; /* * A shadow entry of a recently evicted page, a swap * entry from shmem/tmpfs or a DAX entry. Return it * without attempting to raise page count. */ - if (!page || xa_is_value(page)) - return page; + if (!folio || xa_is_value(folio)) + return folio; - if (!page_cache_get_speculative(page)) + if (!folio_try_get_rcu(folio)) goto reset; - /* Has the page moved or been split? */ - if (unlikely(page != xas_reload(xas))) { - put_page(page); + if (unlikely(folio != xas_reload(xas))) { + folio_put(folio); goto reset; } - return page; + return folio; reset: xas_reset(xas); goto retry; @@ -2042,19 +2041,20 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct pagevec *pvec, pgoff_t *indices) { XA_STATE(xas, &mapping->i_pages, start); - struct page *page; + struct folio *folio; unsigned int ret = 0; unsigned nr_entries = PAGEVEC_SIZE; rcu_read_lock(); - while ((page = find_get_entry(&xas, end, XA_PRESENT))) { + while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) { + struct page *page = &folio->page; /* * Terminate early on finding a THP, to allow the caller to * handle it all at once; but continue if this is hugetlbfs. */ - if (!xa_is_value(page) && PageTransHuge(page) && - !PageHuge(page)) { - page = find_subpage(page, xas.xa_index); + if (!xa_is_value(folio) && folio_test_large(folio) && + !folio_test_hugetlb(folio)) { + page = folio_file_page(folio, xas.xa_index); nr_entries = ret + 1; } @@ -2078,15 +2078,14 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start, * @indices: The cache indices of the entries in @pvec. * * find_lock_entries() will return a batch of entries from @mapping. - * Swap, shadow and DAX entries are included. Pages are returned - * locked and with an incremented refcount. Pages which are locked by - * somebody else or under writeback are skipped. Only the head page of - * a THP is returned. Pages which are partially outside the range are - * not returned. + * Swap, shadow and DAX entries are included. Folios are returned + * locked and with an incremented refcount. Folios which are locked + * by somebody else or under writeback are skipped. Folios which are + * partially outside the range are not returned. * * The entries have ascending indexes. The indices may not be consecutive - * due to not-present entries, THP pages, pages which could not be locked - * or pages under writeback. + * due to not-present entries, large folios, folios which could not be + * locked or folios under writeback. * * Return: The number of entries which were found. */ @@ -2094,37 +2093,36 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct pagevec *pvec, pgoff_t *indices) { XA_STATE(xas, &mapping->i_pages, start); - struct page *page; + struct folio *folio; rcu_read_lock(); - while ((page = find_get_entry(&xas, end, XA_PRESENT))) { - if (!xa_is_value(page)) { - if (page->index < start) + while ((folio = find_get_entry(&xas, end, XA_PRESENT))) { + if (!xa_is_value(folio)) { + if (folio->index < start) goto put; - if (page->index + thp_nr_pages(page) - 1 > end) + if (folio->index + folio_nr_pages(folio) - 1 > end) goto put; - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto put; - if (page->mapping != mapping || PageWriteback(page)) + if (folio->mapping != mapping || + folio_test_writeback(folio)) goto unlock; - VM_BUG_ON_PAGE(!thp_contains(page, xas.xa_index), - page); + VM_BUG_ON_FOLIO(!folio_contains(folio, xas.xa_index), + folio); } indices[pvec->nr] = xas.xa_index; - if (!pagevec_add(pvec, page)) + if (!pagevec_add(pvec, &folio->page)) break; goto next; unlock: - unlock_page(page); + folio_unlock(folio); put: - put_page(page); + folio_put(folio); next: - if (!xa_is_value(page) && PageTransHuge(page)) { - unsigned int nr_pages = thp_nr_pages(page); - - /* Final THP may cross MAX_LFS_FILESIZE on 32-bit */ - xas_set(&xas, page->index + nr_pages); - if (xas.xa_index < nr_pages) + if (!xa_is_value(folio) && folio_test_large(folio)) { + xas_set(&xas, folio->index + folio_nr_pages(folio)); + /* Did we wrap on 32-bit? */ + if (!xas.xa_index) break; } } @@ -2159,19 +2157,19 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start, struct page **pages) { XA_STATE(xas, &mapping->i_pages, *start); - struct page *page; + struct folio *folio; unsigned ret = 0; if (unlikely(!nr_pages)) return 0; rcu_read_lock(); - while ((page = find_get_entry(&xas, end, XA_PRESENT))) { + while ((folio = find_get_entry(&xas, end, XA_PRESENT))) { /* Skip over shadow, swap and DAX entries */ - if (xa_is_value(page)) + if (xa_is_value(folio)) continue; - pages[ret] = find_subpage(page, xas.xa_index); + pages[ret] = folio_file_page(folio, xas.xa_index); if (++ret == nr_pages) { *start = xas.xa_index + 1; goto out; @@ -2268,25 +2266,25 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index, struct page **pages) { XA_STATE(xas, &mapping->i_pages, *index); - struct page *page; + struct folio *folio; unsigned ret = 0; if (unlikely(!nr_pages)) return 0; rcu_read_lock(); - while ((page = find_get_entry(&xas, end, tag))) { + while ((folio = find_get_entry(&xas, end, tag))) { /* * Shadow entries should never be tagged, but this iteration * is lockless so there is a window for page reclaim to evict * a page we saw tagged. Skip over it. */ - if (xa_is_value(page)) + if (xa_is_value(folio)) continue; - pages[ret] = page; + pages[ret] = &folio->page; if (++ret == nr_pages) { - *index = page->index + thp_nr_pages(page); + *index = folio->index + folio_nr_pages(folio); goto out; } } @@ -2794,44 +2792,44 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) } EXPORT_SYMBOL(generic_file_read_iter); -static inline loff_t page_seek_hole_data(struct xa_state *xas, - struct address_space *mapping, struct page *page, +static inline loff_t folio_seek_hole_data(struct xa_state *xas, + struct address_space *mapping, struct folio *folio, loff_t start, loff_t end, bool seek_data) { const struct address_space_operations *ops = mapping->a_ops; size_t offset, bsz = i_blocksize(mapping->host); - if (xa_is_value(page) || PageUptodate(page)) + if (xa_is_value(folio) || folio_test_uptodate(folio)) return seek_data ? start : end; if (!ops->is_partially_uptodate) return seek_data ? end : start; xas_pause(xas); rcu_read_unlock(); - lock_page(page); - if (unlikely(page->mapping != mapping)) + folio_lock(folio); + if (unlikely(folio->mapping != mapping)) goto unlock; - offset = offset_in_thp(page, start) & ~(bsz - 1); + offset = offset_in_folio(folio, start) & ~(bsz - 1); do { - if (ops->is_partially_uptodate(page, offset, bsz) == seek_data) + if (ops->is_partially_uptodate(&folio->page, offset, bsz) == + seek_data) break; start = (start + bsz) & ~(bsz - 1); offset += bsz; - } while (offset < thp_size(page)); + } while (offset < folio_size(folio)); unlock: - unlock_page(page); + folio_unlock(folio); rcu_read_lock(); return start; } -static inline -unsigned int seek_page_size(struct xa_state *xas, struct page *page) +static inline size_t seek_folio_size(struct xa_state *xas, struct folio *folio) { - if (xa_is_value(page)) + if (xa_is_value(folio)) return PAGE_SIZE << xa_get_order(xas->xa, xas->xa_index); - return thp_size(page); + return folio_size(folio); } /** @@ -2858,15 +2856,15 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start, XA_STATE(xas, &mapping->i_pages, start >> PAGE_SHIFT); pgoff_t max = (end - 1) >> PAGE_SHIFT; bool seek_data = (whence == SEEK_DATA); - struct page *page; + struct folio *folio; if (end <= start) return -ENXIO; rcu_read_lock(); - while ((page = find_get_entry(&xas, max, XA_PRESENT))) { + while ((folio = find_get_entry(&xas, max, XA_PRESENT))) { loff_t pos = (u64)xas.xa_index << PAGE_SHIFT; - unsigned int seek_size; + size_t seek_size; if (start < pos) { if (!seek_data) @@ -2874,9 +2872,9 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start, start = pos; } - seek_size = seek_page_size(&xas, page); - pos = round_up(pos + 1, seek_size); - start = page_seek_hole_data(&xas, mapping, page, start, pos, + seek_size = seek_folio_size(&xas, folio); + pos = round_up((u64)pos + 1, seek_size); + start = folio_seek_hole_data(&xas, mapping, folio, start, pos, seek_data); if (start < pos) goto unlock; @@ -2884,15 +2882,15 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start, break; if (seek_size > PAGE_SIZE) xas_set(&xas, pos >> PAGE_SHIFT); - if (!xa_is_value(page)) - put_page(page); + if (!xa_is_value(folio)) + folio_put(folio); } if (seek_data) start = -ENXIO; unlock: rcu_read_unlock(); - if (page && !xa_is_value(page)) - put_page(page); + if (folio && !xa_is_value(folio)) + folio_put(folio); if (start > end) return end; return start; From patchwork Wed Dec 8 04:22:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78598C433EF for ; Wed, 8 Dec 2021 06:20:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 334D46B007D; Wed, 8 Dec 2021 01:18:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E40B6B007E; Wed, 8 Dec 2021 01:18:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1ACC46B0080; Wed, 8 Dec 2021 01:18:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay033.a.hostedemail.com [64.99.140.33]) by kanga.kvack.org (Postfix) with ESMTP id 0BA776B007D for ; Wed, 8 Dec 2021 01:18:55 -0500 (EST) Received: by unirelay08.hostedemail.com (Postfix, from userid 108) id DCD2F205D5; Wed, 8 Dec 2021 06:17:36 +0000 (UTC) Received: by unirelay08.hostedemail.com (Postfix, from userid 108) id 3CFB5202B4; Wed, 8 Dec 2021 05:33:39 +0000 (UTC) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 061FE20285 for ; Wed, 8 Dec 2021 04:58:40 +0000 (UTC) X-FDA: 78893421642.24.9C12A8F Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by imf02.hostedemail.com (Postfix) with ESMTP id 996BA80002 for ; Wed, 8 Dec 2021 04:58:40 +0000 (UTC) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 474E920A07 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332266.05.0B1330E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id B01E920019C3 for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=qEOJB5zqrjmIjIMgFkQEnCrDBHeSM6KGmwYonkVVccg=; b=vj99tWu1MyReP+jzY/YAJWhNVE PHZjCc6Pivhz/Q9ENpqol0+nR60b2WugjfAeSEhB1LWNcfSnxdFswkB6r+fesy6E1jRQ9q6kJ8+zo xhelbKHgobmdMqubTl/xCBSm71n4RQGUBglLQi0z35Fe1a90ZQYhlU76sZM4/ZrTNgcyBRJGUE3FU FhE+wb16q6iolgmxvQVW0is3Xre7PW03hp9nmtbYVoKWnSpiWkYD/MZYm4tYbZsFSyzcftzp36ii8 EkbgRiDndw3cqyMhQdq+PV3GMC6dUkxG+GZLyGEdNBwaVtKoqe8UPstfH8xdNvgrs0iRCGCX8PB52 pLaup+Qw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084Xu-8H; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 15/48] filemap: Remove thp_contains() Date: Wed, 8 Dec 2021 04:22:23 +0000 Message-Id: <20211208042256.1923824-16-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 996BA80002 X-Stat-Signature: yd1o4hmnyykkqf1scqdnrszmnnrn5rhc X-HE-Tag-Orig: 1638937393-802871 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vj99tWu1; spf=none (imf02.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.35) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638939520-347107 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This function is now unused, so delete it. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/pagemap.h | 9 --------- 1 file changed, 9 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 3f26b191ede3..8c2cad7f0c36 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -512,15 +512,6 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping, mapping_gfp_mask(mapping)); } -/* Does this page contain this index? */ -static inline bool thp_contains(struct page *head, pgoff_t index) -{ - /* HugeTLBfs indexes the page cache in units of hpage_size */ - if (PageHuge(head)) - return head->index == index; - return page_index(head) == (index & ~(thp_nr_pages(head) - 1UL)); -} - #define swapcache_index(folio) __page_file_index(&(folio)->page) /** From patchwork Wed Dec 8 04:22:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E367EC433F5 for ; Wed, 8 Dec 2021 06:32:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5ECFB6B007E; Wed, 8 Dec 2021 01:23:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 575766B0080; Wed, 8 Dec 2021 01:23:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 416AA6B0085; Wed, 8 Dec 2021 01:23:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay034.a.hostedemail.com [64.99.140.34]) by kanga.kvack.org (Postfix) with ESMTP id 3236D6B007E for ; Wed, 8 Dec 2021 01:23:56 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 73033606BB; Wed, 8 Dec 2021 05:38:45 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id DB7C0605D7; Wed, 8 Dec 2021 05:32:08 +0000 (UTC) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2605E609EE for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.16.8C75373 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id BB7B91900 for ; Wed, 8 Dec 2021 04:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=uCaFGZPNrDIvJ/RaktaqTtu3XrQ4Je459+upnkc0xsU=; b=LxpvwjxBZz00iz3SIC9Sr0xVQb HWICEjt/jzZFnq4UlSgWK0ni+rgkCEjwXhMPSET7+g9KCSu/VUM6ascOeq+5z8PbVBARezxPFFVSB 5xyf5RH/2Pkp3A7wsaWnPKeM2tYNOKoRvPijVzxsDMUSRYJt+x3bgYxA/QskF9yZkienHEYq4ROcn p8sKN3ImlEHijdffLogipscgB0ZrDeGI6oxtch2JrSxKW1yLB/4HmM0CueApx0uSsAMJQarySKyfu FsQ0qL7bdcd90I2zlnAQlTg/Yj53IxMEr5Y6Gv18dAa1mM2ng9iGer7wbnrxOP2S5JqbZSjff079w 65K3EHJg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084Y0-Cw; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 16/48] filemap: Convert filemap_get_read_batch to use folios Date: Wed, 8 Dec 2021 04:22:24 +0000 Message-Id: <20211208042256.1923824-17-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LxpvwjxB; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BB7B91900 X-Stat-Signature: nw5ogstnni3p84us4h6y8jc1ftauwup9 X-HE-Tag: 1638937392-449905 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page cache only stores folios, never tail pages. Saves 29 bytes due to removing calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 2a51ec720e9e..120df74f3c7c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2339,32 +2339,31 @@ static void filemap_get_read_batch(struct address_space *mapping, pgoff_t index, pgoff_t max, struct pagevec *pvec) { XA_STATE(xas, &mapping->i_pages, index); - struct page *head; + struct folio *folio; rcu_read_lock(); - for (head = xas_load(&xas); head; head = xas_next(&xas)) { - if (xas_retry(&xas, head)) + for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) { + if (xas_retry(&xas, folio)) continue; - if (xas.xa_index > max || xa_is_value(head)) + if (xas.xa_index > max || xa_is_value(folio)) break; - if (!page_cache_get_speculative(head)) + if (!folio_try_get_rcu(folio)) goto retry; - /* Has the page moved or been split? */ - if (unlikely(head != xas_reload(&xas))) + if (unlikely(folio != xas_reload(&xas))) goto put_page; - if (!pagevec_add(pvec, head)) + if (!pagevec_add(pvec, &folio->page)) break; - if (!PageUptodate(head)) + if (!folio_test_uptodate(folio)) break; - if (PageReadahead(head)) + if (folio_test_readahead(folio)) break; - xas.xa_index = head->index + thp_nr_pages(head) - 1; + xas.xa_index = folio->index + folio_nr_pages(folio) - 1; xas.xa_offset = (xas.xa_index >> xas.xa_shift) & XA_CHUNK_MASK; continue; put_page: - put_page(head); + folio_put(folio); retry: xas_reset(&xas); } From patchwork Wed Dec 8 04:22:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 771AAC433F5 for ; Wed, 8 Dec 2021 07:03:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B3206B0071; Wed, 8 Dec 2021 02:03:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 063586B0073; Wed, 8 Dec 2021 02:03:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6C776B0074; Wed, 8 Dec 2021 02:03:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay032.a.hostedemail.com [64.99.140.32]) by kanga.kvack.org (Postfix) with ESMTP id D820C6B0071 for ; Wed, 8 Dec 2021 02:03:35 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 010CE601B8 for ; Wed, 8 Dec 2021 04:34:53 +0000 (UTC) X-FDA: 78893361708.20.89E511F Received: from relay.hostedemail.com (unknown [216.40.35.100]) by imf04.hostedemail.com (Postfix) with ESMTP id A428440004 for ; Wed, 8 Dec 2021 04:34:53 +0000 (UTC) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4E9E26040E for ; Wed, 8 Dec 2021 04:34:53 +0000 (UTC) X-FDA: 78893361624.14.4B7D2C3 Received: from relay.hostedemail.com (relay031.a.hostedemail.com [64.99.140.31]) by imf23.hostedemail.com (Postfix) with ESMTP id E2E419000093 for ; Wed, 8 Dec 2021 04:34:52 +0000 (UTC) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2263D60A27 for ; Wed, 8 Dec 2021 04:28:40 +0000 (UTC) X-FDA: 78893346000.16.3F3A2BA Received: from relay.hostedemail.com (relay028.a.hostedemail.com [64.99.140.28]) by imf21.hostedemail.com (Postfix) with ESMTP id C2C7AD0369C2 for ; Wed, 8 Dec 2021 04:28:39 +0000 (UTC) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9155320A09 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.19.A08D191 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 41579B000181 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/Jrg5K2lm0b8mctZUfSr5Z1cNwHiyfe9P0jRf7cbjh8=; b=EUeKL+/oZub5WS+fcIk5TDXrNv b/hrZwkF/9xfkhxX1ONt53rcg6HvZ+Wez9MHjE4HWnpz5AIJeyn2cPHRJ+mJRP65IyfWNKKPoj4m1 YDWG9bcS9duVzCCrf1rQcugMsCXXvgb7KV6ubJvkQAKubjI9DSuzCJPB4CVys3CyszJQaRVjEltoV 5vpEvMy9vUhUSI9/Hlrri3UKEj6HgIBi4xLGXPnxZBpwEOXqY+wgZo8pqJeDfeny8hQlAoW5sq5v6 ClPlMoyS+NgLHcmNTZDGqMaKDfeb0Do6In0fqHYoQ2FB+gruQdONAnkdfQSGUOpzBlm3nAYW5O2iQ +0hhvB0g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084Y6-HU; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 17/48] filemap: Convert find_get_pages_contig to folios Date: Wed, 8 Dec 2021 04:22:25 +0000 Message-Id: <20211208042256.1923824-18-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937394-578674 X-HE-Tag-Orig: 1638937719-203566 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A428440004 X-Stat-Signature: idrftft8oyc1zsgb38i1zwgx8td6ioma X-HE-Tag-Orig: 1638938092-723514 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="EUeKL+/o"; spf=none (imf04.hostedemail.com: domain of FILTER%FILTER%FILTER%willy@infradead.org has no SPF policy when checking 216.40.35.100) smtp.mailfrom=FILTER%FILTER%FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638938093-805267 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: None of the callers of find_get_pages_contig() want tail pages. They all use order-0 pages today, but if they were converted, they'd want folios. So just remove the call to find_subpage() instead of replacing it with folio_page(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 120df74f3c7c..33e638f1ca34 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2208,36 +2208,35 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index, unsigned int nr_pages, struct page **pages) { XA_STATE(xas, &mapping->i_pages, index); - struct page *page; + struct folio *folio; unsigned int ret = 0; if (unlikely(!nr_pages)) return 0; rcu_read_lock(); - for (page = xas_load(&xas); page; page = xas_next(&xas)) { - if (xas_retry(&xas, page)) + for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) { + if (xas_retry(&xas, folio)) continue; /* * If the entry has been swapped out, we can stop looking. * No current caller is looking for DAX entries. */ - if (xa_is_value(page)) + if (xa_is_value(folio)) break; - if (!page_cache_get_speculative(page)) + if (!folio_try_get_rcu(folio)) goto retry; - /* Has the page moved or been split? */ - if (unlikely(page != xas_reload(&xas))) + if (unlikely(folio != xas_reload(&xas))) goto put_page; - pages[ret] = find_subpage(page, xas.xa_index); + pages[ret] = &folio->page; if (++ret == nr_pages) break; continue; put_page: - put_page(page); + folio_put(folio); retry: xas_reset(&xas); } From patchwork Wed Dec 8 04:22:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB948C433EF for ; Wed, 8 Dec 2021 06:31:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 419776B0075; Wed, 8 Dec 2021 01:23:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CA7A6B007D; Wed, 8 Dec 2021 01:23:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2698E6B007E; Wed, 8 Dec 2021 01:23:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 14FFA6B0075 for ; Wed, 8 Dec 2021 01:23:46 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 700236044F; Wed, 8 Dec 2021 05:38:45 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 8118B604B4; Wed, 8 Dec 2021 05:32:09 +0000 (UTC) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9E10F609FF for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.18.658103B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 356226001983 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Ffq/cQVPcFt5LoElUBw8aA2bmdFv10SCY9OdwQ5yJQA=; b=ZoCCrfBFGomvt8SJ8oYXyTew2p pDTXUDYEZDZQGTw0CSnLSGvksOSX4S9AYEAh/nNrd7DnRZ+FLb0kWO1KxyChPrtabjtzCd9ZCAcbR wNG90x7cVYy000WVHvgV1SdORhzBmPT3bpVy/BnPRuYJCkqnE3cuOdb98lpKcOQANraMI34RVVoH6 o4y5qvFFLPRKi8n3PwaI0XkqqW4zsLOv4NXx/lAvkeeIUkwx1duj/83q+e/bRFLMwSeVkD9PRLZ+8 iO4yBbRfol1nHLB037r7RlXrN1ddM0xm8/IEy3Jq/MpY6ta65Q88Nsnne18F4SseSNEdyZPf4kpmV ZD3gyquQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084YC-KU; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 18/48] filemap: Convert filemap_read_page to take a folio Date: Wed, 8 Dec 2021 04:22:26 +0000 Message-Id: <20211208042256.1923824-19-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 356226001983 X-Stat-Signature: t5bjk6amm65u71tgx85gge3gjtzbpwth Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ZoCCrfBF; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937394-765226 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: One of the callers already had a folio; the other two grow by a few bytes, but filemap_read_page() shrinks by 50 bytes for a net reduction of 27 bytes. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 33e638f1ca34..581f9fdb3406 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2369,8 +2369,8 @@ static void filemap_get_read_batch(struct address_space *mapping, rcu_read_unlock(); } -static int filemap_read_page(struct file *file, struct address_space *mapping, - struct page *page) +static int filemap_read_folio(struct file *file, struct address_space *mapping, + struct folio *folio) { int error; @@ -2379,16 +2379,16 @@ static int filemap_read_page(struct file *file, struct address_space *mapping, * eg. multipath errors. PG_error will be set again if readpage * fails. */ - ClearPageError(page); + folio_clear_error(folio); /* Start the actual read. The read will unlock the page. */ - error = mapping->a_ops->readpage(file, page); + error = mapping->a_ops->readpage(file, &folio->page); if (error) return error; - error = wait_on_page_locked_killable(page); + error = folio_wait_locked_killable(folio); if (error) return error; - if (PageUptodate(page)) + if (folio_test_uptodate(folio)) return 0; shrink_readahead_size_eio(&file->f_ra); return -EIO; @@ -2464,7 +2464,7 @@ static int filemap_update_page(struct kiocb *iocb, if (iocb->ki_flags & (IOCB_NOIO | IOCB_NOWAIT | IOCB_WAITQ)) goto unlock; - error = filemap_read_page(iocb->ki_filp, mapping, &folio->page); + error = filemap_read_folio(iocb->ki_filp, mapping, folio); goto unlock_mapping; unlock: folio_unlock(folio); @@ -2506,7 +2506,7 @@ static int filemap_create_page(struct file *file, if (error) goto error; - error = filemap_read_page(file, mapping, page); + error = filemap_read_folio(file, mapping, page_folio(page)); if (error) goto error; @@ -3168,7 +3168,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * and we need to check for errors. */ fpin = maybe_unlock_mmap_for_io(vmf, fpin); - error = filemap_read_page(file, mapping, page); + error = filemap_read_folio(file, mapping, page_folio(page)); if (fpin) goto out_retry; put_page(page); From patchwork Wed Dec 8 04:22:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB186C433EF for ; Wed, 8 Dec 2021 07:09:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 603BD6B0074; Wed, 8 Dec 2021 02:08:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D0EE6B0073; Wed, 8 Dec 2021 02:08:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CFCF6B0074; Wed, 8 Dec 2021 02:08:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 1B97B6B0073 for ; Wed, 8 Dec 2021 02:08:37 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 7E374801DF for ; Wed, 8 Dec 2021 04:48:21 +0000 (UTC) X-FDA: 78893395602.17.4285097 Received: from relay.hostedemail.com (unknown [216.40.35.100]) by imf21.hostedemail.com (Postfix) with ESMTP id 0F6B5D0369C2 for ; Wed, 8 Dec 2021 04:48:20 +0000 (UTC) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5739420A95 for ; Wed, 8 Dec 2021 04:39:00 +0000 (UTC) X-FDA: 78893372040.24.9213F32 Received: from relay.hostedemail.com (relay028.a.hostedemail.com [64.99.140.28]) by imf26.hostedemail.com (Postfix) with ESMTP id CB09120019C3 for ; Wed, 8 Dec 2021 04:38:59 +0000 (UTC) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 71553601A7 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332266.27.95FA1B2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 252A140002 for ; Wed, 8 Dec 2021 04:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=0HiMfEBapSaJ76Nsadm/Hl6t4X5OTaxDIeW3md1qPS8=; b=F9HU2Jd0g8x9PI6I8QPzUDhc6q MFdRSl5wmj+dWbRQX83gKwBUJcqLj0+JbSjfqOyvaQYwso082bAt9wYOObgr6fUk7E7XDcg8rSLrP /qGpTchwgqAl2ozG4ogVUAApd8Sochh7JYtfh/F/eh/cWWa4uAZeGfmDJgUAwlbQTDbvsme77Th6y fRUKi/nsWA0FoqAljbhP63J/F5jz3mm5WAJAMa40UCRQhdMzxqPjGZGfkSds7em4kQjWjOFaR4fW0 jSO9YNOcc5HFxf5vdt4toY1onIICo1bNUNe5GxaAknlAs65hYEp8x3891te/f2RTSG4J9Z8eifbDS tQC1gR9w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084YJ-O1; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 19/48] filemap: Convert filemap_create_page to folio Date: Wed, 8 Dec 2021 04:22:27 +0000 Message-Id: <20211208042256.1923824-20-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937392-354665 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0F6B5D0369C2 X-Stat-Signature: ar58rqs4qmhucahohz3hj1efhyzhk1oz X-HE-Tag-Orig: 1638938339-148845 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=F9HU2Jd0; spf=none (imf21.hostedemail.com: domain of FILTER%FILTER%willy@infradead.org has no SPF policy when checking 216.40.35.100) smtp.mailfrom=FILTER%FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638938900-573585 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is all internal to filemap and saves 100 bytes of text. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 581f9fdb3406..b044afef78ef 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2475,47 +2475,48 @@ static int filemap_update_page(struct kiocb *iocb, return error; } -static int filemap_create_page(struct file *file, +static int filemap_create_folio(struct file *file, struct address_space *mapping, pgoff_t index, struct pagevec *pvec) { - struct page *page; + struct folio *folio; int error; - page = page_cache_alloc(mapping); - if (!page) + folio = filemap_alloc_folio(mapping_gfp_mask(mapping), 0); + if (!folio) return -ENOMEM; /* - * Protect against truncate / hole punch. Grabbing invalidate_lock here - * assures we cannot instantiate and bring uptodate new pagecache pages - * after evicting page cache during truncate and before actually - * freeing blocks. Note that we could release invalidate_lock after - * inserting the page into page cache as the locked page would then be - * enough to synchronize with hole punching. But there are code paths - * such as filemap_update_page() filling in partially uptodate pages or - * ->readpages() that need to hold invalidate_lock while mapping blocks - * for IO so let's hold the lock here as well to keep locking rules - * simple. + * Protect against truncate / hole punch. Grabbing invalidate_lock + * here assures we cannot instantiate and bring uptodate new + * pagecache folios after evicting page cache during truncate + * and before actually freeing blocks. Note that we could + * release invalidate_lock after inserting the folio into + * the page cache as the locked folio would then be enough to + * synchronize with hole punching. But there are code paths + * such as filemap_update_page() filling in partially uptodate + * pages or ->readpages() that need to hold invalidate_lock + * while mapping blocks for IO so let's hold the lock here as + * well to keep locking rules simple. */ filemap_invalidate_lock_shared(mapping); - error = add_to_page_cache_lru(page, mapping, index, + error = filemap_add_folio(mapping, folio, index, mapping_gfp_constraint(mapping, GFP_KERNEL)); if (error == -EEXIST) error = AOP_TRUNCATED_PAGE; if (error) goto error; - error = filemap_read_folio(file, mapping, page_folio(page)); + error = filemap_read_folio(file, mapping, folio); if (error) goto error; filemap_invalidate_unlock_shared(mapping); - pagevec_add(pvec, page); + pagevec_add(pvec, &folio->page); return 0; error: filemap_invalidate_unlock_shared(mapping); - put_page(page); + folio_put(folio); return error; } @@ -2557,7 +2558,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, if (!pagevec_count(pvec)) { if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) return -EAGAIN; - err = filemap_create_page(filp, mapping, + err = filemap_create_folio(filp, mapping, iocb->ki_pos >> PAGE_SHIFT, pvec); if (err == AOP_TRUNCATED_PAGE) goto retry; From patchwork Wed Dec 8 04:22:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0748AC433EF for ; Wed, 8 Dec 2021 07:00:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96BD76B0075; Wed, 8 Dec 2021 01:58:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91BBB6B0078; Wed, 8 Dec 2021 01:58:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80A526B007B; Wed, 8 Dec 2021 01:58:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by kanga.kvack.org (Postfix) with ESMTP id 70A6E6B0075 for ; Wed, 8 Dec 2021 01:58:45 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 2FEEA801B5 for ; Wed, 8 Dec 2021 04:29:10 +0000 (UTC) X-FDA: 78893347218.11.FD900FF Received: from relay.hostedemail.com (relay034.a.hostedemail.com [64.99.140.34]) by imf14.hostedemail.com (Postfix) with ESMTP id C8F416001983 for ; Wed, 8 Dec 2021 04:29:09 +0000 (UTC) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C053621396 for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) X-FDA: 78893332266.28.38CF984 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 4BF91900024B for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zEgXkSFSyMhCDJOHlKNghSVrMcqn165U8NTQAl+dWqc=; b=bz12BuBcZTfHqa5dTwh8rOHjDm vc7TDqj5res0EDmYcwsR9xgH2Ney/EsiNS2mJY3zrc/MIkPBWDWvOZaFd0Rz8ra2P2GEEX6WcoJBb c0qtkWiokhfGoVJWW35NXYaUOD/OXEThVuW/fWJQSwHiYJ2K309g89G7iWK6n0fz6CoV5gCLOy4mR aC3faxLeAfmbx3ckFxm06CBguVnkFlg7C8xCmUqq5dj94ohonL9SrLOfExGnF2d4DTqW5NJA+GFu1 Dxnuo71keuwoJsYfZa/54mxTnVykZI6En91fvb5Jt/Il2WQ46O11qs4KZ0+sETe8Bptnf0f/DUetz f9yr+Cdw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU3-0084YR-S5; Wed, 08 Dec 2021 04:23:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 20/48] filemap: Convert filemap_range_uptodate to folios Date: Wed, 8 Dec 2021 04:22:28 +0000 Message-Id: <20211208042256.1923824-21-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C8F416001983 X-HE-Tag-Orig: 1638937393-414941 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bz12BuBc; spf=none (imf14.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.34) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-Rspamd-Server: rspam04 X-Stat-Signature: 91t41jnu18pzuhzo3jnhk6x85qycc9s1 X-HE-Tag: 1638937749-940347 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The only caller was already passing a head page, so this simply avoids a call to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index b044afef78ef..c4f887c277d0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2395,29 +2395,29 @@ static int filemap_read_folio(struct file *file, struct address_space *mapping, } static bool filemap_range_uptodate(struct address_space *mapping, - loff_t pos, struct iov_iter *iter, struct page *page) + loff_t pos, struct iov_iter *iter, struct folio *folio) { int count; - if (PageUptodate(page)) + if (folio_test_uptodate(folio)) return true; /* pipes can't handle partially uptodate pages */ if (iov_iter_is_pipe(iter)) return false; if (!mapping->a_ops->is_partially_uptodate) return false; - if (mapping->host->i_blkbits >= (PAGE_SHIFT + thp_order(page))) + if (mapping->host->i_blkbits >= folio_shift(folio)) return false; count = iter->count; - if (page_offset(page) > pos) { - count -= page_offset(page) - pos; + if (folio_pos(folio) > pos) { + count -= folio_pos(folio) - pos; pos = 0; } else { - pos -= page_offset(page); + pos -= folio_pos(folio); } - return mapping->a_ops->is_partially_uptodate(page, pos, count); + return mapping->a_ops->is_partially_uptodate(&folio->page, pos, count); } static int filemap_update_page(struct kiocb *iocb, @@ -2457,7 +2457,7 @@ static int filemap_update_page(struct kiocb *iocb, goto unlock; error = 0; - if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, &folio->page)) + if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, folio)) goto unlock; error = -EAGAIN; From patchwork Wed Dec 8 04:22:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8154C433F5 for ; Wed, 8 Dec 2021 06:59:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 302926B0074; Wed, 8 Dec 2021 01:58:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B1E36B0075; Wed, 8 Dec 2021 01:58:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A1936B0078; Wed, 8 Dec 2021 01:58:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay031.a.hostedemail.com [64.99.140.31]) by kanga.kvack.org (Postfix) with ESMTP id 0B05B6B0074 for ; Wed, 8 Dec 2021 01:58:36 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 4A3881201CA for ; Wed, 8 Dec 2021 04:29:13 +0000 (UTC) X-FDA: 78893347386.09.ADA7689 Received: from relay.hostedemail.com (relay038.a.hostedemail.com [64.99.140.38]) by imf27.hostedemail.com (Postfix) with ESMTP id CE8BB7000081 for ; Wed, 8 Dec 2021 04:29:12 +0000 (UTC) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3A02021330 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332350.30.4CB1A48 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id AAEA0900024B for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9Lt8nc8uCMiQ8EPI8aySjPso9UmiJ/85df+p8gpiD3U=; b=AHmJb0PrliFHPV1qjvwMrHEgRi tlvRWFkKM0IIz4vb7L6CaAPw2xP4EOqY+wxsUDTsWGc7DVkTncS3O0D4pG1ZOvmGxspAOukzy8lVn iubvMLqDUsr/OSSNmjYBmkk++4JIax4FU2qUEVSkXrb+wf/+JOBHAha7El/eiVcRIlmlHixtgv2KE IOQaDpM1V+3ZQhiMHHtGBRnkXnrhQF5cwocnlzcCGs48XU90nYNl5o/c1jqwidWYckvfEeRLSmySM 51lrPiLtunEKoIElPXZ3iPbLNOhZo6MyJLhqWlUse4N/T8r2yUfi/gDel8+W1/9ar7Kd12jSCkgOY NO9nNVtQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084YZ-0R; Wed, 08 Dec 2021 04:23:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 21/48] readahead: Convert page_cache_async_ra() to take a folio Date: Wed, 8 Dec 2021 04:22:29 +0000 Message-Id: <20211208042256.1923824-22-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: kkowz5owy8dp6dhzc1whdxy37sj8f47g Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AHmJb0Pr; dmarc=none; spf=none (imf27.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.38) smtp.mailfrom=FILTER%willy@infradead.org X-HE-Tag-Orig: 1638937394-898505 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CE8BB7000081 X-HE-Tag: 1638937752-421126 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using the folio here avoids checking whether it's a tail page. This patch mostly just enables some of the following patches. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/pagemap.h | 4 ++-- mm/readahead.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8c2cad7f0c36..30302be6977f 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -993,7 +993,7 @@ struct readahead_control { void page_cache_ra_unbounded(struct readahead_control *, unsigned long nr_to_read, unsigned long lookahead_count); void page_cache_sync_ra(struct readahead_control *, unsigned long req_count); -void page_cache_async_ra(struct readahead_control *, struct page *, +void page_cache_async_ra(struct readahead_control *, struct folio *, unsigned long req_count); void readahead_expand(struct readahead_control *ractl, loff_t new_start, size_t new_len); @@ -1040,7 +1040,7 @@ void page_cache_async_readahead(struct address_space *mapping, struct page *page, pgoff_t index, unsigned long req_count) { DEFINE_READAHEAD(ractl, file, ra, mapping, index); - page_cache_async_ra(&ractl, page, req_count); + page_cache_async_ra(&ractl, page_folio(page), req_count); } static inline struct folio *__readahead_folio(struct readahead_control *ractl) diff --git a/mm/readahead.c b/mm/readahead.c index 6ae5693de28c..e48e78641772 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -581,7 +581,7 @@ void page_cache_sync_ra(struct readahead_control *ractl, EXPORT_SYMBOL_GPL(page_cache_sync_ra); void page_cache_async_ra(struct readahead_control *ractl, - struct page *page, unsigned long req_count) + struct folio *folio, unsigned long req_count) { /* no read-ahead */ if (!ractl->ra->ra_pages) @@ -590,10 +590,10 @@ void page_cache_async_ra(struct readahead_control *ractl, /* * Same bit is used for PG_readahead and PG_reclaim. */ - if (PageWriteback(page)) + if (folio_test_writeback(folio)) return; - ClearPageReadahead(page); + folio_clear_readahead(folio); /* * Defer asynchronous read-ahead on IO congestion. From patchwork Wed Dec 8 04:22:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8E09C433F5 for ; Wed, 8 Dec 2021 06:29:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 554E96B0095; Wed, 8 Dec 2021 01:20:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5062C6B0096; Wed, 8 Dec 2021 01:20:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CD5F6B0098; Wed, 8 Dec 2021 01:20:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay038.a.hostedemail.com [64.99.140.38]) by kanga.kvack.org (Postfix) with ESMTP id 2DC526B0095 for ; Wed, 8 Dec 2021 01:20:00 -0500 (EST) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id A76814F5; Wed, 8 Dec 2021 06:18:01 +0000 (UTC) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id C511945F; Wed, 8 Dec 2021 05:34:09 +0000 (UTC) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3D02A244 for ; Wed, 8 Dec 2021 04:39:00 +0000 (UTC) X-FDA: 78893371998.27.144CCCE Received: from relay.hostedemail.com (unknown [216.40.35.100]) by imf17.hostedemail.com (Postfix) with ESMTP id CCDECF0001C0 for ; Wed, 8 Dec 2021 04:38:59 +0000 (UTC) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 04DD060165 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332350.08.BFEEEED Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 9AF7240002 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=t4pwlbtmjecL2V1Nz4HoWruFyvYom5U3vixCYsyX2S8=; b=EOFAdrF9B0cYq4AWYNj0NL4Oka 6cEb9SbPxApoqG/263g/BskuoT/olUn8FAQqUbnMGLiplE9Ekj+NVGb09QaF/1f6GaCwS5SRSFzN7 f91kISzGCXDOnc1oqctDBkdf0njk4VfnvGdvPDP2rCPDB+JsADIeyvR4PcXP9xleJcCBE+JW6UhJD H2SZ03zyxCKA0pbSQvd49klJvNiONagFNHKFCDEkiMabQMw30eP7MvEIDtvjUyuBw/gH0Gcr1HY+A oomCJ1z5wCSpNtAZo7vgB6/QHm1CADFYEmhsleHTVY67AIprbs4HKgp2kAv8SrtSjyR4cDzaH9tjM Jvw+rrtQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084Yh-52; Wed, 08 Dec 2021 04:23:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 22/48] readahead: Convert page_cache_ra_unbounded to folios Date: Wed, 8 Dec 2021 04:22:30 +0000 Message-Id: <20211208042256.1923824-23-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CCDECF0001C0 X-Stat-Signature: e6d6qu6mu6tcbji883mqm1crjr6o6qam X-HE-Tag-Orig: 1638937394-910988 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EOFAdrF9; spf=none (imf17.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 216.40.35.100) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638938339-573731 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This saves 99 bytes of kernel text. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/readahead.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index e48e78641772..cf0dcf89eb69 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -196,9 +196,9 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, * Preallocate as many pages as we will need. */ for (i = 0; i < nr_to_read; i++) { - struct page *page = xa_load(&mapping->i_pages, index + i); + struct folio *folio = xa_load(&mapping->i_pages, index + i); - if (page && !xa_is_value(page)) { + if (folio && !xa_is_value(folio)) { /* * Page already present? Kick off the current batch * of contiguous pages before continuing with the @@ -212,21 +212,21 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, continue; } - page = __page_cache_alloc(gfp_mask); - if (!page) + folio = filemap_alloc_folio(gfp_mask, 0); + if (!folio) break; if (mapping->a_ops->readpages) { - page->index = index + i; - list_add(&page->lru, &page_pool); - } else if (add_to_page_cache_lru(page, mapping, index + i, + folio->index = index + i; + list_add(&folio->lru, &page_pool); + } else if (filemap_add_folio(mapping, folio, index + i, gfp_mask) < 0) { - put_page(page); + folio_put(folio); read_pages(ractl, &page_pool, true); i = ractl->_index + ractl->_nr_pages - index - 1; continue; } if (i == nr_to_read - lookahead_size) - SetPageReadahead(page); + folio_set_readahead(folio); ractl->_nr_pages++; } From patchwork Wed Dec 8 04:22:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97552C433FE for ; Wed, 8 Dec 2021 06:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13BCF6B0071; Wed, 8 Dec 2021 01:16:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C52A6B0073; Wed, 8 Dec 2021 01:16:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA77A6B0074; Wed, 8 Dec 2021 01:16:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id DA5136B0071 for ; Wed, 8 Dec 2021 01:16:13 -0500 (EST) Received: by unirelay06.hostedemail.com (Postfix, from userid 108) id 609A6205C1; Wed, 8 Dec 2021 06:15:00 +0000 (UTC) Received: by unirelay06.hostedemail.com (Postfix, from userid 108) id 27ADC2106C; Wed, 8 Dec 2021 05:33:16 +0000 (UTC) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1837D20EA8 for ; Wed, 8 Dec 2021 04:29:12 +0000 (UTC) X-FDA: 78893347344.30.8609FCC Received: from relay.hostedemail.com (relay038.a.hostedemail.com [64.99.140.38]) by imf26.hostedemail.com (Postfix) with ESMTP id 9C0A520019C6 for ; Wed, 8 Dec 2021 04:29:11 +0000 (UTC) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B074B2139E for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) X-FDA: 78893332308.31.9BB6D70 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 4F4E340002 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=e0hyHWAmzlXP+aObqjdWnAJ+TFDQXw//r2+3SQKN8pQ=; b=MC0SZlqDdsPCFEh/D6J0RPxNXW zkz67m1pLbrEZR5ngrhJ94reVvmupJyALO1vhSOhi2+IhDDEn7OjzXcm5SW83shxAa7/ps5gmwmV/ UL3QZtt9/TpojRlRKpQ+4xstwEQBhzo7eDFsWPulepYtEKWpMLdt3zBhlL1uJ599OgcGbsosSOJ9A oLqFpYte+Ej/3Fu+AMn5RG4NghNAsY16LJk69ipPqN/X9OAe2e60EK6MOv3lU0NHCIqx437/qw5En +luVWqHvHUq/6oKD4LX9f5Q4CQvmWP9mNrfwaYUOvjrHPXE4lqIVKnmAxOoCYrbCXupZx/rAH7Unv RVXEGAcA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084Yp-BJ; Wed, 08 Dec 2021 04:23:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 23/48] filemap: Convert do_async_mmap_readahead to take a folio Date: Wed, 8 Dec 2021 04:22:31 +0000 Message-Id: <20211208042256.1923824-24-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 9C0A520019C6 X-Stat-Signature: q1uot73h5fkpxch8h839h8sxk3eab7tz X-HE-Tag-Orig: 1638937395-369962 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MC0SZlqD; spf=none (imf26.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.38) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638937751-780831 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call page_cache_async_ra() directly instead of indirecting through page_cache_async_readahead(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index c4f887c277d0..0838b08557f5 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3001,25 +3001,25 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) * was pinned if we have to drop the mmap_lock in order to do IO. */ static struct file *do_async_mmap_readahead(struct vm_fault *vmf, - struct page *page) + struct folio *folio) { struct file *file = vmf->vma->vm_file; struct file_ra_state *ra = &file->f_ra; - struct address_space *mapping = file->f_mapping; + DEFINE_READAHEAD(ractl, file, ra, file->f_mapping, vmf->pgoff); struct file *fpin = NULL; unsigned int mmap_miss; - pgoff_t offset = vmf->pgoff; /* If we don't want any read-ahead, don't bother */ if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages) return fpin; + mmap_miss = READ_ONCE(ra->mmap_miss); if (mmap_miss) WRITE_ONCE(ra->mmap_miss, --mmap_miss); - if (PageReadahead(page)) { + + if (folio_test_readahead(folio)) { fpin = maybe_unlock_mmap_for_io(vmf, fpin); - page_cache_async_readahead(mapping, ra, file, - page, offset, ra->ra_pages); + page_cache_async_ra(&ractl, folio, ra->ra_pages); } return fpin; } @@ -3069,12 +3069,13 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) */ page = find_get_page(mapping, offset); if (likely(page)) { + struct folio *folio = page_folio(page); /* * We found the page, so try async readahead before waiting for * the lock. */ if (!(vmf->flags & FAULT_FLAG_TRIED)) - fpin = do_async_mmap_readahead(vmf, page); + fpin = do_async_mmap_readahead(vmf, folio); if (unlikely(!PageUptodate(page))) { filemap_invalidate_lock_shared(mapping); mapping_locked = true; From patchwork Wed Dec 8 04:22:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0C79C433F5 for ; Wed, 8 Dec 2021 06:25:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5566D6B0088; Wed, 8 Dec 2021 01:19:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DED06B0089; Wed, 8 Dec 2021 01:19:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37FCB6B008A; Wed, 8 Dec 2021 01:19:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 28F446B0088 for ; Wed, 8 Dec 2021 01:19:28 -0500 (EST) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 61D3B5EE; Wed, 8 Dec 2021 06:18:01 +0000 (UTC) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id D526257B; Wed, 8 Dec 2021 05:34:08 +0000 (UTC) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 938732D6 for ; Wed, 8 Dec 2021 04:28:38 +0000 (UTC) X-FDA: 78893345916.17.893F169 Received: from relay.hostedemail.com (relay030.a.hostedemail.com [64.99.140.30]) by imf31.hostedemail.com (Postfix) with ESMTP id 545E81046300 for ; Wed, 8 Dec 2021 04:28:38 +0000 (UTC) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 540DC20A08 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.01.4D33E32 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf31.hostedemail.com (Postfix) with ESMTP id DA5701046300 for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=sGlg8/qoS2pD0+lmCeNJGiM/awa+dbIop15sAKpzu3c=; b=qKxd4LXF0iFL6SZF2ySJm+5ymB pibp64Dq52T8XBLNAqrtOYBBAw5OD4LNxlrfM6QANyz++3Qvn2qYILgFKFUCabxJ90FGj7fP7UVSW xgguWb5QR53SfITu5UBw1ZanLp3QEAZKB2tQUD923iKR0bk7aqDrO4bwZp7K9IVDgc3KvJRDjpWVo SRiUhyPhi2cj6aIid7mjvV4ycyRMvbb5K59r/VZdPbDfeKvFJthh8A38P3Z+ptNcLZWo7Rnje+c3s rV989oytNf2iiKLUL93d4nYVswhaO4h2fgbm69jNdMgI8E9W3rDBRpm/KkvHLK8wiqx0TIKnbe3ym jM2ecS+A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084Yv-Fc; Wed, 08 Dec 2021 04:23:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 24/48] filemap: Convert filemap_fault to folio Date: Wed, 8 Dec 2021 04:22:32 +0000 Message-Id: <20211208042256.1923824-25-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: m316rtstr83m7hfppyc9tkr3zkeybpth X-HE-Tag-Orig: 1638937393-936001 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qKxd4LXF; spf=none (imf31.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.30) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 545E81046300 X-HE-Tag: 1638937718-296881 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of converting back-and-forth between the actual page and the head page, just convert once at the end of the function where we set the vmf->page. Saves 241 bytes of text, or 15% of the size of filemap_fault(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 77 +++++++++++++++++++++++++--------------------------- 1 file changed, 37 insertions(+), 40 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0838b08557f5..fc0f1d9904d2 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2898,21 +2898,20 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start, #ifdef CONFIG_MMU #define MMAP_LOTSAMISS (100) /* - * lock_page_maybe_drop_mmap - lock the page, possibly dropping the mmap_lock + * lock_folio_maybe_drop_mmap - lock the page, possibly dropping the mmap_lock * @vmf - the vm_fault for this fault. - * @page - the page to lock. + * @folio - the folio to lock. * @fpin - the pointer to the file we may pin (or is already pinned). * - * This works similar to lock_page_or_retry in that it can drop the mmap_lock. - * It differs in that it actually returns the page locked if it returns 1 and 0 - * if it couldn't lock the page. If we did have to drop the mmap_lock then fpin - * will point to the pinned file and needs to be fput()'ed at a later point. + * This works similar to lock_folio_or_retry in that it can drop the + * mmap_lock. It differs in that it actually returns the folio locked + * if it returns 1 and 0 if it couldn't lock the folio. If we did have + * to drop the mmap_lock then fpin will point to the pinned file and + * needs to be fput()'ed at a later point. */ -static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, +static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, struct file **fpin) { - struct folio *folio = page_folio(page); - if (folio_trylock(folio)) return 1; @@ -3038,7 +3037,7 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf, * vma->vm_mm->mmap_lock must be held on entry. * * If our return value has VM_FAULT_RETRY set, it's because the mmap_lock - * may be dropped before doing I/O or by lock_page_maybe_drop_mmap(). + * may be dropped before doing I/O or by lock_folio_maybe_drop_mmap(). * * If our return value does not have VM_FAULT_RETRY set, the mmap_lock * has not been released. @@ -3054,29 +3053,27 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) struct file *fpin = NULL; struct address_space *mapping = file->f_mapping; struct inode *inode = mapping->host; - pgoff_t offset = vmf->pgoff; - pgoff_t max_off; - struct page *page; + pgoff_t max_idx, index = vmf->pgoff; + struct folio *folio; vm_fault_t ret = 0; bool mapping_locked = false; - max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(offset >= max_off)) + max_idx = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); + if (unlikely(index >= max_idx)) return VM_FAULT_SIGBUS; /* * Do we have something in the page cache already? */ - page = find_get_page(mapping, offset); - if (likely(page)) { - struct folio *folio = page_folio(page); + folio = filemap_get_folio(mapping, index); + if (likely(folio)) { /* * We found the page, so try async readahead before waiting for * the lock. */ if (!(vmf->flags & FAULT_FLAG_TRIED)) fpin = do_async_mmap_readahead(vmf, folio); - if (unlikely(!PageUptodate(page))) { + if (unlikely(!folio_test_uptodate(folio))) { filemap_invalidate_lock_shared(mapping); mapping_locked = true; } @@ -3088,17 +3085,17 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) fpin = do_sync_mmap_readahead(vmf); retry_find: /* - * See comment in filemap_create_page() why we need + * See comment in filemap_create_folio() why we need * invalidate_lock */ if (!mapping_locked) { filemap_invalidate_lock_shared(mapping); mapping_locked = true; } - page = pagecache_get_page(mapping, offset, + folio = __filemap_get_folio(mapping, index, FGP_CREAT|FGP_FOR_MMAP, vmf->gfp_mask); - if (!page) { + if (!folio) { if (fpin) goto out_retry; filemap_invalidate_unlock_shared(mapping); @@ -3106,22 +3103,22 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) } } - if (!lock_page_maybe_drop_mmap(vmf, page, &fpin)) + if (!lock_folio_maybe_drop_mmap(vmf, folio, &fpin)) goto out_retry; /* Did it get truncated? */ - if (unlikely(compound_head(page)->mapping != mapping)) { - unlock_page(page); - put_page(page); + if (unlikely(folio->mapping != mapping)) { + folio_unlock(folio); + folio_put(folio); goto retry_find; } - VM_BUG_ON_PAGE(page_to_pgoff(page) != offset, page); + VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio); /* * We have a locked page in the page cache, now we need to check * that it's up-to-date. If not, it is going to be due to an error. */ - if (unlikely(!PageUptodate(page))) { + if (unlikely(!folio_test_uptodate(folio))) { /* * The page was in cache and uptodate and now it is not. * Strange but possible since we didn't hold the page lock all @@ -3129,8 +3126,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * try again. */ if (!mapping_locked) { - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); goto retry_find; } goto page_not_uptodate; @@ -3142,7 +3139,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * redo the fault. */ if (fpin) { - unlock_page(page); + folio_unlock(folio); goto out_retry; } if (mapping_locked) @@ -3152,14 +3149,14 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * Found the page and have a reference on it. * We must recheck i_size under page lock. */ - max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - if (unlikely(offset >= max_off)) { - unlock_page(page); - put_page(page); + max_idx = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); + if (unlikely(index >= max_idx)) { + folio_unlock(folio); + folio_put(folio); return VM_FAULT_SIGBUS; } - vmf->page = page; + vmf->page = folio_file_page(folio, index); return ret | VM_FAULT_LOCKED; page_not_uptodate: @@ -3170,10 +3167,10 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * and we need to check for errors. */ fpin = maybe_unlock_mmap_for_io(vmf, fpin); - error = filemap_read_folio(file, mapping, page_folio(page)); + error = filemap_read_folio(file, mapping, folio); if (fpin) goto out_retry; - put_page(page); + folio_put(folio); if (!error || error == AOP_TRUNCATED_PAGE) goto retry_find; @@ -3187,8 +3184,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * re-find the vma and come back and find our hopefully still populated * page. */ - if (page) - put_page(page); + if (folio) + folio_put(folio); if (mapping_locked) filemap_invalidate_unlock_shared(mapping); if (fpin) From patchwork Wed Dec 8 04:22:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33D7FC433EF for ; Wed, 8 Dec 2021 06:29:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 138CA6B0096; Wed, 8 Dec 2021 01:20:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E7686B0098; Wed, 8 Dec 2021 01:20:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F18176B0099; Wed, 8 Dec 2021 01:20:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by kanga.kvack.org (Postfix) with ESMTP id E20576B0096 for ; Wed, 8 Dec 2021 01:20:06 -0500 (EST) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 9D4FB3CF; Wed, 8 Dec 2021 06:18:06 +0000 (UTC) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 7B7B3406; Wed, 8 Dec 2021 05:34:08 +0000 (UTC) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4A849736 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.03.B49A7B1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id F0A85900009E for ; Wed, 8 Dec 2021 04:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=qtbKUPPS3ZVhgIUCfwJ/vbY8r8bBiiA/jYBJR/fWLdI=; b=g0Wig305W2ZPr5DePosI9mOHDW lgkahCfiDnxpcl9Zfqg9YaWIt/LxXOUbIlqrhAPf2lGa6eJ7IrXlWr6V7qaYzZEM1gLLx05OZjhPG uAPIpCVdQmc46nPHoLE8yao33CwTpdRX+HahcAr1UIYnB4/glU7ue3iFNsyzh9HXgekTSZFU+/zHD JNXpVoqqRzi4HVImRAzmVbcXl3ofKNaqYhl+ij1SWhUvdn/uRYYmbQfRp7QN8K5MYxeaEABVWZICX 1SNRMRvOXdNv1ACIebBUODEhaz9Ye4BOvFuYGCP4jsLFIjuamkd+GlPWy2x6fCPX1Sdsah16pnocf q22lWlJg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084Z3-L7; Wed, 08 Dec 2021 04:23:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 25/48] filemap: Add read_cache_folio and read_mapping_folio Date: Wed, 8 Dec 2021 04:22:33 +0000 Message-Id: <20211208042256.1923824-26-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F0A85900009E X-Stat-Signature: xwredef64u5qrbfnhbnuqdhpjr6exeyh Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=g0Wig305; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937393-725206 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reimplement read_cache_page() as a wrapper around read_cache_folio(). Saves over 400 bytes of text from do_read_cache_folio() which more thn makes up for the extra 100 bytes of text added to the various wrapper functions. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 12 +++++- mm/filemap.c | 95 +++++++++++++++++++++-------------------- 2 files changed, 59 insertions(+), 48 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 30302be6977f..7bef50ea5435 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -629,8 +629,10 @@ static inline struct page *grab_cache_page(struct address_space *mapping, return find_or_create_page(mapping, index, mapping_gfp_mask(mapping)); } -extern struct page * read_cache_page(struct address_space *mapping, - pgoff_t index, filler_t *filler, void *data); +struct folio *read_cache_folio(struct address_space *, pgoff_t index, + filler_t *filler, void *data); +struct page *read_cache_page(struct address_space *, pgoff_t index, + filler_t *filler, void *data); extern struct page * read_cache_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); extern int read_cache_pages(struct address_space *mapping, @@ -642,6 +644,12 @@ static inline struct page *read_mapping_page(struct address_space *mapping, return read_cache_page(mapping, index, NULL, data); } +static inline struct folio *read_mapping_folio(struct address_space *mapping, + pgoff_t index, void *data) +{ + return read_cache_folio(mapping, index, NULL, data); +} + /* * Get index of the page within radix-tree (but not for hugetlb pages). * (TODO: remove once hugetlb pages will have ->index in PAGE_SIZE) diff --git a/mm/filemap.c b/mm/filemap.c index fc0f1d9904d2..f34dda0a7627 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3418,35 +3418,20 @@ EXPORT_SYMBOL(filemap_page_mkwrite); EXPORT_SYMBOL(generic_file_mmap); EXPORT_SYMBOL(generic_file_readonly_mmap); -static struct page *wait_on_page_read(struct page *page) +static struct folio *do_read_cache_folio(struct address_space *mapping, + pgoff_t index, filler_t filler, void *data, gfp_t gfp) { - if (!IS_ERR(page)) { - wait_on_page_locked(page); - if (!PageUptodate(page)) { - put_page(page); - page = ERR_PTR(-EIO); - } - } - return page; -} - -static struct page *do_read_cache_page(struct address_space *mapping, - pgoff_t index, - int (*filler)(void *, struct page *), - void *data, - gfp_t gfp) -{ - struct page *page; + struct folio *folio; int err; repeat: - page = find_get_page(mapping, index); - if (!page) { - page = __page_cache_alloc(gfp); - if (!page) + folio = filemap_get_folio(mapping, index); + if (!folio) { + folio = filemap_alloc_folio(gfp, 0); + if (!folio) return ERR_PTR(-ENOMEM); - err = add_to_page_cache_lru(page, mapping, index, gfp); + err = filemap_add_folio(mapping, folio, index, gfp); if (unlikely(err)) { - put_page(page); + folio_put(folio); if (err == -EEXIST) goto repeat; /* Presumably ENOMEM for xarray node */ @@ -3455,21 +3440,24 @@ static struct page *do_read_cache_page(struct address_space *mapping, filler: if (filler) - err = filler(data, page); + err = filler(data, &folio->page); else - err = mapping->a_ops->readpage(data, page); + err = mapping->a_ops->readpage(data, &folio->page); if (err < 0) { - put_page(page); + folio_put(folio); return ERR_PTR(err); } - page = wait_on_page_read(page); - if (IS_ERR(page)) - return page; + folio_wait_locked(folio); + if (!folio_test_uptodate(folio)) { + folio_put(folio); + return ERR_PTR(-EIO); + } + goto out; } - if (PageUptodate(page)) + if (folio_test_uptodate(folio)) goto out; /* @@ -3503,23 +3491,23 @@ static struct page *do_read_cache_page(struct address_space *mapping, * avoid spurious serialisations and wakeups when multiple processes * wait on the same page for IO to complete. */ - wait_on_page_locked(page); - if (PageUptodate(page)) + folio_wait_locked(folio); + if (folio_test_uptodate(folio)) goto out; /* Distinguish between all the cases under the safety of the lock */ - lock_page(page); + folio_lock(folio); /* Case c or d, restart the operation */ - if (!page->mapping) { - unlock_page(page); - put_page(page); + if (!folio->mapping) { + folio_unlock(folio); + folio_put(folio); goto repeat; } /* Someone else locked and filled the page in a very small window */ - if (PageUptodate(page)) { - unlock_page(page); + if (folio_test_uptodate(folio)) { + folio_unlock(folio); goto out; } @@ -3529,16 +3517,16 @@ static struct page *do_read_cache_page(struct address_space *mapping, * Clear page error before actual read, PG_error will be * set again if read page fails. */ - ClearPageError(page); + folio_clear_error(folio); goto filler; out: - mark_page_accessed(page); - return page; + folio_mark_accessed(folio); + return folio; } /** - * read_cache_page - read into page cache, fill it if needed + * read_cache_folio - read into page cache, fill it if needed * @mapping: the page's address_space * @index: the page index * @filler: function to perform the read @@ -3553,10 +3541,25 @@ static struct page *do_read_cache_page(struct address_space *mapping, * * Return: up to date page on success, ERR_PTR() on failure. */ +struct folio *read_cache_folio(struct address_space *mapping, pgoff_t index, + filler_t filler, void *data) +{ + return do_read_cache_folio(mapping, index, filler, data, + mapping_gfp_mask(mapping)); +} +EXPORT_SYMBOL(read_cache_folio); + +static struct page *do_read_cache_page(struct address_space *mapping, + pgoff_t index, filler_t *filler, void *data, gfp_t gfp) +{ + struct folio *folio = read_cache_folio(mapping, index, filler, data); + if (IS_ERR(folio)) + return &folio->page; + return folio_file_page(folio, index); +} + struct page *read_cache_page(struct address_space *mapping, - pgoff_t index, - int (*filler)(void *, struct page *), - void *data) + pgoff_t index, filler_t *filler, void *data) { return do_read_cache_page(mapping, index, filler, data, mapping_gfp_mask(mapping)); From patchwork Wed Dec 8 04:22:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7305C433EF for ; Wed, 8 Dec 2021 06:48:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BBF76B0071; Wed, 8 Dec 2021 01:48:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 444FE6B0073; Wed, 8 Dec 2021 01:48:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E5446B0074; Wed, 8 Dec 2021 01:48:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay026.a.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 1A8316B0071 for ; Wed, 8 Dec 2021 01:48:36 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 08943601B0 for ; Wed, 8 Dec 2021 04:28:56 +0000 (UTC) X-FDA: 78893346588.27.74151D0 Received: from relay.hostedemail.com (relay037.a.hostedemail.com [64.99.140.37]) by imf27.hostedemail.com (Postfix) with ESMTP id 8867D7000081 for ; Wed, 8 Dec 2021 04:28:55 +0000 (UTC) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 2FDCD1201C8 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332308.21.EE4E6A5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id B8C3E4002087 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=nGTKE0UdgttmA0mxrDkUyEEjWKbpIL8TqlNy2EdTN2Y=; b=hEyn3n48W+l3guJA++IEkfiI2Y iVK8k6jmg2oUNQJsrPCxZCXIFnEiRv0CvrbSKoQOTQ1Xzxl1zuuJvkBr/pFI9NBzSmImLbNBvX4jh FkLqKZCUbOFs5yXczMbrngvy58n+qVdR9QFCtsSQAlWxVEpfOkLs/mbapiEAg8eyFRVd+n+5p5thL sLDn84latXJS+Zq/jAnVfKaIM24ydC9Nh7dMCTgT9/hutecPX6bJQ9z2Z8sOVrXvinGWVq7JtsaWM pBva4x6KNMs2AyL+PabPq9CYw2sw1JnkfSJxxET6BBnwm0oJ6VorQEdxwVPjVHsLIMAbF0yqhp6YI AtjYuVbg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084ZB-P9; Wed, 08 Dec 2021 04:23:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 26/48] filemap: Convert filemap_get_pages to use folios Date: Wed, 8 Dec 2021 04:22:34 +0000 Message-Id: <20211208042256.1923824-27-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: d7osbycpinaayhgd1sxb91w3if6x7osd X-HE-Tag-Orig: 1638937395-951729 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hEyn3n48; spf=none (imf27.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.37) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8867D7000081 X-HE-Tag: 1638937735-792531 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This saves a few calls to compound_head(), including one in filemap_update_page(). Shrinks the kernel by 78 bytes. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f34dda0a7627..d191a4fd758a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2422,9 +2422,8 @@ static bool filemap_range_uptodate(struct address_space *mapping, static int filemap_update_page(struct kiocb *iocb, struct address_space *mapping, struct iov_iter *iter, - struct page *page) + struct folio *folio) { - struct folio *folio = page_folio(page); int error; if (iocb->ki_flags & IOCB_NOWAIT) { @@ -2521,13 +2520,14 @@ static int filemap_create_folio(struct file *file, } static int filemap_readahead(struct kiocb *iocb, struct file *file, - struct address_space *mapping, struct page *page, + struct address_space *mapping, struct folio *folio, pgoff_t last_index) { + DEFINE_READAHEAD(ractl, file, &file->f_ra, mapping, folio->index); + if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; - page_cache_async_readahead(mapping, &file->f_ra, file, page, - page->index, last_index - page->index); + page_cache_async_ra(&ractl, folio, last_index - folio->index); return 0; } @@ -2539,7 +2539,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, struct file_ra_state *ra = &filp->f_ra; pgoff_t index = iocb->ki_pos >> PAGE_SHIFT; pgoff_t last_index; - struct page *page; + struct folio *folio; int err = 0; last_index = DIV_ROUND_UP(iocb->ki_pos + iter->count, PAGE_SIZE); @@ -2565,16 +2565,16 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, return err; } - page = pvec->pages[pagevec_count(pvec) - 1]; - if (PageReadahead(page)) { - err = filemap_readahead(iocb, filp, mapping, page, last_index); + folio = page_folio(pvec->pages[pagevec_count(pvec) - 1]); + if (folio_test_readahead(folio)) { + err = filemap_readahead(iocb, filp, mapping, folio, last_index); if (err) goto err; } - if (!PageUptodate(page)) { + if (!folio_test_uptodate(folio)) { if ((iocb->ki_flags & IOCB_WAITQ) && pagevec_count(pvec) > 1) iocb->ki_flags |= IOCB_NOWAIT; - err = filemap_update_page(iocb, mapping, iter, page); + err = filemap_update_page(iocb, mapping, iter, folio); if (err) goto err; } @@ -2582,7 +2582,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, return 0; err: if (err < 0) - put_page(page); + folio_put(folio); if (likely(--pvec->nr)) return 0; if (err == AOP_TRUNCATED_PAGE) From patchwork Wed Dec 8 04:22:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F366C433F5 for ; Wed, 8 Dec 2021 06:05:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B4DA6B0074; Wed, 8 Dec 2021 01:05:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 763C56B0075; Wed, 8 Dec 2021 01:05:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 604C56B0078; Wed, 8 Dec 2021 01:05:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay026.a.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 4DC7C6B0074 for ; Wed, 8 Dec 2021 01:05:08 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id E3F28603FA; Wed, 8 Dec 2021 05:38:44 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 77C9A6056A; Wed, 8 Dec 2021 05:32:09 +0000 (UTC) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 90F206099B for ; Wed, 8 Dec 2021 04:48:40 +0000 (UTC) X-FDA: 78893396400.21.7EF9875 Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by imf10.hostedemail.com (Postfix) with ESMTP id 3E56B6001986 for ; Wed, 8 Dec 2021 04:48:40 +0000 (UTC) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 85439209F0 for ; Wed, 8 Dec 2021 04:39:00 +0000 (UTC) X-FDA: 78893371998.14.676F1EE Received: from relay.hostedemail.com (relay038.a.hostedemail.com [64.99.140.38]) by imf13.hostedemail.com (Postfix) with ESMTP id D5BFF1044801 for ; Wed, 8 Dec 2021 04:38:59 +0000 (UTC) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id B744F6019E for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) X-FDA: 78893332308.16.86A96C0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 27E553000100 for ; Wed, 8 Dec 2021 04:23:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=4GYC4zomnfMK1s3qCzsnkSCKwVtKcv5N2U8Vny3BnNo=; b=bzIJdgUxakOcUFNXYAWZgFkJnD Aa+e38cnGy1mKXDnYvsbjXuKGZI1iEb7oQcsyNa8axteOvrA1ar2xwnwN32TbWzb6SblLHuOMCmgv mkFDC/4kSsUlyGvTzQAj4mad9HBk66F2ldJ8uVeVwD+SonVClqfNrTvVX8pZge4F13EBy3UqGDp2w jrd4/Itk3EjZdx25fvtjfLETynP7hHhszqijFWPjzYIAT1JeK7cPeWdzviKEBQvLAazGWABSSH7Lc IVukBizVdatKzsIrZDqTT95EsW6pIWfnMFab+MU0acXYessKF7H7DaJ4hUlWpLn6JTJCRSMHSAl7C 2H9B+S6A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU4-0084ZK-Uy; Wed, 08 Dec 2021 04:23:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 27/48] filemap: Convert page_cache_delete_batch to folios Date: Wed, 8 Dec 2021 04:22:35 +0000 Message-Id: <20211208042256.1923824-28-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937394-881077 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3E56B6001986 X-Stat-Signature: xmr6cg8w76d7ios6dh516kfp8k5oj8yj X-HE-Tag-Orig: 1638938339-132204 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bzIJdgUx; dmarc=none; spf=none (imf10.hostedemail.com: domain of FILTER%FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.25) smtp.mailfrom=FILTER%FILTER%willy@infradead.org X-HE-Tag: 1638938920-813677 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves one call to compound_head() and reduces text size by 15 bytes. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/filemap.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index d191a4fd758a..3ea81adbabd8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -290,15 +290,15 @@ static void page_cache_delete_batch(struct address_space *mapping, XA_STATE(xas, &mapping->i_pages, pvec->pages[0]->index); int total_pages = 0; int i = 0; - struct page *page; + struct folio *folio; mapping_set_update(&xas, mapping); - xas_for_each(&xas, page, ULONG_MAX) { + xas_for_each(&xas, folio, ULONG_MAX) { if (i >= pagevec_count(pvec)) break; /* A swap/dax/shadow entry got inserted? Skip it. */ - if (xa_is_value(page)) + if (xa_is_value(folio)) continue; /* * A page got inserted in our range? Skip it. We have our @@ -307,16 +307,16 @@ static void page_cache_delete_batch(struct address_space *mapping, * means our page has been removed, which shouldn't be * possible because we're holding the PageLock. */ - if (page != pvec->pages[i]) { - VM_BUG_ON_PAGE(page->index > pvec->pages[i]->index, - page); + if (&folio->page != pvec->pages[i]) { + VM_BUG_ON_FOLIO(folio->index > + pvec->pages[i]->index, folio); continue; } - WARN_ON_ONCE(!PageLocked(page)); + WARN_ON_ONCE(!folio_test_locked(folio)); - if (page->index == xas.xa_index) - page->mapping = NULL; + if (folio->index == xas.xa_index) + folio->mapping = NULL; /* Leave page->index set: truncation lookup relies on it */ /* @@ -324,7 +324,7 @@ static void page_cache_delete_batch(struct address_space *mapping, * page or the index is of the last sub-page of this compound * page. */ - if (page->index + compound_nr(page) - 1 == xas.xa_index) + if (folio->index + folio_nr_pages(folio) - 1 == xas.xa_index) i++; xas_store(&xas, NULL); total_pages++; From patchwork Wed Dec 8 04:22:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02234C433EF for ; Wed, 8 Dec 2021 06:17:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 929FE6B0073; Wed, 8 Dec 2021 01:16:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B3416B0074; Wed, 8 Dec 2021 01:16:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77B066B0075; Wed, 8 Dec 2021 01:16:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay031.a.hostedemail.com [64.99.140.31]) by kanga.kvack.org (Postfix) with ESMTP id 69B986B0073 for ; Wed, 8 Dec 2021 01:16:58 -0500 (EST) Received: by unirelay06.hostedemail.com (Postfix, from userid 108) id AEB462057A; Wed, 8 Dec 2021 06:15:03 +0000 (UTC) Received: by unirelay06.hostedemail.com (Postfix, from userid 108) id 5C50A20E8A; Wed, 8 Dec 2021 05:33:12 +0000 (UTC) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E98502127F for ; Wed, 8 Dec 2021 04:28:54 +0000 (UTC) X-FDA: 78893346588.23.BF4A073 Received: from relay.hostedemail.com (relay032.a.hostedemail.com [64.99.140.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 851F39000249 for ; Wed, 8 Dec 2021 04:28:54 +0000 (UTC) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 875B6801B1 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332392.09.72DA62A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 023781044ABA for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vspXQFsQkYxygqyTSBfIIKIkyYP21RM2Z3OtZtUgtKM=; b=Mf+AtuGjXHj7NKgO8YT6qB47HB 47WlsT2Vtws44d24pLEmegarURySLywiq+9j2LDZzsZ2BRJyFL79jKRmAzs5Z02guqjY1sbFy0Yj3 YPjjLBXANKhoJ7qV/a8+lOGw5qUAWMj551aBNnXRuhiMBzSQhvz4xHPxF8ykr8jG3m11d5C4gSeoi vPdp9d9BcM9Y0WjnJFZ56dJk79xr9Tm5a4jRXxmh95Sp7fpseiH72UmvST29p6DdnuNIn4TnsuvTb whUewrpLIfqx+zKIdf3IosKmbpPpc/0hL8Ub+kXvpuq2672ykfYvUFo08ry7FjCg98gI1RORt1B2V CeLXouOg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU5-0084ZW-4F; Wed, 08 Dec 2021 04:23:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 28/48] filemap: Use folios in next_uptodate_page Date: Wed, 8 Dec 2021 04:22:36 +0000 Message-Id: <20211208042256.1923824-29-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 851F39000249 X-Stat-Signature: 5bzdadmezkdudoujbhzkuatjafwxh6nf X-HE-Tag-Orig: 1638937395-722355 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Mf+AtuGj; spf=none (imf29.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.32) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638937734-390201 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This saves 105 bytes of text. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3ea81adbabd8..47880ec789f4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3227,43 +3227,43 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct page *page) return false; } -static struct page *next_uptodate_page(struct page *page, +static struct page *next_uptodate_page(struct folio *folio, struct address_space *mapping, struct xa_state *xas, pgoff_t end_pgoff) { unsigned long max_idx; do { - if (!page) + if (!folio) return NULL; - if (xas_retry(xas, page)) + if (xas_retry(xas, folio)) continue; - if (xa_is_value(page)) + if (xa_is_value(folio)) continue; - if (PageLocked(page)) + if (folio_test_locked(folio)) continue; - if (!page_cache_get_speculative(page)) + if (!folio_try_get_rcu(folio)) continue; /* Has the page moved or been split? */ - if (unlikely(page != xas_reload(xas))) + if (unlikely(folio != xas_reload(xas))) goto skip; - if (!PageUptodate(page) || PageReadahead(page)) + if (!folio_test_uptodate(folio) || folio_test_readahead(folio)) goto skip; - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto skip; - if (page->mapping != mapping) + if (folio->mapping != mapping) goto unlock; - if (!PageUptodate(page)) + if (!folio_test_uptodate(folio)) goto unlock; max_idx = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE); if (xas->xa_index >= max_idx) goto unlock; - return page; + return &folio->page; unlock: - unlock_page(page); + folio_unlock(folio); skip: - put_page(page); - } while ((page = xas_next_entry(xas, end_pgoff)) != NULL); + folio_put(folio); + } while ((folio = xas_next_entry(xas, end_pgoff)) != NULL); return NULL; } From patchwork Wed Dec 8 04:22:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D04F9C433EF for ; Wed, 8 Dec 2021 06:19:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 901C16B0073; Wed, 8 Dec 2021 01:18:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B02A6B007D; Wed, 8 Dec 2021 01:18:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 778D76B007E; Wed, 8 Dec 2021 01:18:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay026.a.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 6809D6B0073 for ; Wed, 8 Dec 2021 01:18:38 -0500 (EST) Received: by unirelay02.hostedemail.com (Postfix, from userid 108) id EA637210A3; Wed, 8 Dec 2021 05:32:57 +0000 (UTC) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 300242107A for ; Wed, 8 Dec 2021 04:29:13 +0000 (UTC) X-FDA: 78893347386.12.56A5B00 Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by imf16.hostedemail.com (Postfix) with ESMTP id C78EBF00008F for ; Wed, 8 Dec 2021 04:29:12 +0000 (UTC) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 34831213A0 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332392.18.8264C7B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id B85D2100005 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=HmCkLmM73d68sFeDxuolEllAHZE5BSsg2WY8bvzzTFU=; b=GETSPJyRFuJqsMvXyRm+P1Lwlv AfTlnrLL1Jmnr6Zc3Mr6VQOD2hH7FZ9wvo7KmbUZhu+cuMe+j7/wZcaMcVl5Nyls6mrJ6vXocaTBi gbMEM7mfbB1nCYm1B4a3CJ4Uyc5939G77ahZL1EYl/1eKiq+Ki4eSNpGqLOk9xRO0ALsBJaA6J4JI DFvD0u2OyPG96ggBWZXRAJaSP1H/2j56vCisr2P2LcbvMiwwGAvvbVdhWYfmuvYDjAECZNL2cv7L2 FmFM/iQ0FGskhIBXu8E3WVqSNlLLgI4NKDcWr5D2Hlgvi0g0aBl8EDieDLKwaMy8/1KCvn6j0NC2U ldqa9yOA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU5-0084Zd-9k; Wed, 08 Dec 2021 04:23:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 29/48] filemap: Use a folio in filemap_map_pages Date: Wed, 8 Dec 2021 04:22:37 +0000 Message-Id: <20211208042256.1923824-30-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937395-857090 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C78EBF00008F X-Stat-Signature: 9pdfsux1xwqn3csrg6jiwhqskh8s4zp5 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GETSPJyR; spf=none (imf16.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.24) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638937752-330882 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves 61 bytes due to fewer calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 47880ec789f4..8cca04a79808 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3227,7 +3227,7 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct page *page) return false; } -static struct page *next_uptodate_page(struct folio *folio, +static struct folio *next_uptodate_page(struct folio *folio, struct address_space *mapping, struct xa_state *xas, pgoff_t end_pgoff) { @@ -3258,7 +3258,7 @@ static struct page *next_uptodate_page(struct folio *folio, max_idx = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE); if (xas->xa_index >= max_idx) goto unlock; - return &folio->page; + return folio; unlock: folio_unlock(folio); skip: @@ -3268,7 +3268,7 @@ static struct page *next_uptodate_page(struct folio *folio, return NULL; } -static inline struct page *first_map_page(struct address_space *mapping, +static inline struct folio *first_map_page(struct address_space *mapping, struct xa_state *xas, pgoff_t end_pgoff) { @@ -3276,7 +3276,7 @@ static inline struct page *first_map_page(struct address_space *mapping, mapping, xas, end_pgoff); } -static inline struct page *next_map_page(struct address_space *mapping, +static inline struct folio *next_map_page(struct address_space *mapping, struct xa_state *xas, pgoff_t end_pgoff) { @@ -3293,16 +3293,17 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, pgoff_t last_pgoff = start_pgoff; unsigned long addr; XA_STATE(xas, &mapping->i_pages, start_pgoff); - struct page *head, *page; + struct folio *folio; + struct page *page; unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); vm_fault_t ret = 0; rcu_read_lock(); - head = first_map_page(mapping, &xas, end_pgoff); - if (!head) + folio = first_map_page(mapping, &xas, end_pgoff); + if (!folio) goto out; - if (filemap_map_pmd(vmf, head)) { + if (filemap_map_pmd(vmf, &folio->page)) { ret = VM_FAULT_NOPAGE; goto out; } @@ -3310,7 +3311,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { - page = find_subpage(head, xas.xa_index); + page = folio_file_page(folio, xas.xa_index); if (PageHWPoison(page)) goto unlock; @@ -3331,12 +3332,12 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, do_set_pte(vmf, page, addr); /* no need to invalidate: a not-present page won't be cached */ update_mmu_cache(vma, addr, vmf->pte); - unlock_page(head); + folio_unlock(folio); continue; unlock: - unlock_page(head); - put_page(head); - } while ((head = next_map_page(mapping, &xas, end_pgoff)) != NULL); + folio_unlock(folio); + folio_put(folio); + } while ((folio = next_map_page(mapping, &xas, end_pgoff)) != NULL); pte_unmap_unlock(vmf->pte, vmf->ptl); out: rcu_read_unlock(); From patchwork Wed Dec 8 04:22:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F063C433EF for ; Wed, 8 Dec 2021 06:04:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 544266B0071; Wed, 8 Dec 2021 01:04:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F3176B0073; Wed, 8 Dec 2021 01:04:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BABC6B0075; Wed, 8 Dec 2021 01:04:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by kanga.kvack.org (Postfix) with ESMTP id 2AE926B0073 for ; Wed, 8 Dec 2021 01:04:43 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id D631A60430; Wed, 8 Dec 2021 05:38:45 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 8954F6058B; Wed, 8 Dec 2021 05:32:07 +0000 (UTC) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4D5F660A08 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332350.14.F23BF92 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id E602340002 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8xbmIsGq9D+rej3BrE9p2VwSQhGNmZUjepcD8zMRSIA=; b=nlhDT7vd2bgRqO+rVD4ENwDbwp ph/q9Sk2q1LlWI0cBCKBSD41oNRVG1/Rnprrwi1LVMTIrJd36uiUseWHiHV3Hgw9J3cIPFi8igzgt 9gH7bIQSWGqTRyqcS5AwiqBX0nUkhOqDOXQpQzW2F/g97ByAvePirmpoDmHqcS99yyPTmuqzezGXU GcME17NHhb1bDoiWJDD49B332EP15+1XWZpjW7iAMF7XbGMurKIBXy4+ucSaGh3083qpA9RGzBNPN YYw0IfJ8YNC1IgWWiULwXDFJO0InwXlm8uDXEPgsJ7lPK2Z+g+Zc3aH+lUYl+2Zv9vP/OHKIWXfD1 zHTJwyQg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU5-0084Zk-EL; Wed, 08 Dec 2021 04:23:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 30/48] filemap: Use a folio in filemap_page_mkwrite Date: Wed, 8 Dec 2021 04:22:38 +0000 Message-Id: <20211208042256.1923824-31-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E602340002 X-Stat-Signature: jby645q8umfhkhz3pzake83wed7bmxz4 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nlhDT7vd; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937395-949305 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This fixes a bug for tail pages. They always have a NULL mapping, so the check would fail and we would never mark the folio as dirty. Ends up growing the kernel by 19 bytes although there will be fewer calls to compound_head() dynamically. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 8cca04a79808..4ae9d5befffa 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3349,24 +3349,24 @@ EXPORT_SYMBOL(filemap_map_pages); vm_fault_t filemap_page_mkwrite(struct vm_fault *vmf) { struct address_space *mapping = vmf->vma->vm_file->f_mapping; - struct page *page = vmf->page; + struct folio *folio = page_folio(vmf->page); vm_fault_t ret = VM_FAULT_LOCKED; sb_start_pagefault(mapping->host->i_sb); file_update_time(vmf->vma->vm_file); - lock_page(page); - if (page->mapping != mapping) { - unlock_page(page); + folio_lock(folio); + if (folio->mapping != mapping) { + folio_unlock(folio); ret = VM_FAULT_NOPAGE; goto out; } /* - * We mark the page dirty already here so that when freeze is in + * We mark the folio dirty already here so that when freeze is in * progress, we are guaranteed that writeback during freezing will - * see the dirty page and writeprotect it again. + * see the dirty folio and writeprotect it again. */ - set_page_dirty(page); - wait_for_stable_page(page); + folio_mark_dirty(folio); + folio_wait_stable(folio); out: sb_end_pagefault(mapping->host->i_sb); return ret; From patchwork Wed Dec 8 04:22:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAC13C433F5 for ; Wed, 8 Dec 2021 06:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E6756B0080; Wed, 8 Dec 2021 01:43:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 696AF6B007D; Wed, 8 Dec 2021 01:43:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 515A26B0082; Wed, 8 Dec 2021 01:43:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay029.a.hostedemail.com [64.99.140.29]) by kanga.kvack.org (Postfix) with ESMTP id 3DE4F6B007E for ; Wed, 8 Dec 2021 01:43:37 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 13E22601A3 for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) X-FDA: 78893332392.14.E8A9E8E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 905C7100002 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=GCpZGNOUxHLLTl2217lTyxFtrl9wPCCH15JN3JPxAik=; b=bK2RtpqLE9Nfi/4mSmw4BFp9S5 9G0YigUiEge2HdJJqbBeu7vEJ99CtCEqyMRbScZRYO6b/bUklsKV94PL9kwg9HCTqyDfas+633Y8O vHKaJWCLvuOaxg/EvOgck8B+elIHB9xeuNR9/PW82TgtU3HgU5vF99d3qWdqh4TDxx2nk5/IZMIOJ BqAlfxI6gLLvJ99Oc4+Oo2gPvkUQaG/WfzhErUNbrvXFNxm/Cy0jsOjAzVv8DXn/LprAUQKNQZe63 Vev6m+LjrswnsVDdb1tQQbcQ86dNI4XCBzslbyS1Jwln1wewnq03lMBC5b+Dwtcl2OwM/C+N38P86 CDF4ontA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU5-0084Zt-JY; Wed, 08 Dec 2021 04:23:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 31/48] filemap: Add filemap_release_folio() Date: Wed, 8 Dec 2021 04:22:39 +0000 Message-Id: <20211208042256.1923824-32-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 905C7100002 X-Stat-Signature: hm19s8gw55tm5hgta3swfq48o4147gdp Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bK2RtpqL; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937396-354027 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reimplement try_to_release_page() as a wrapper around filemap_release_folio(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 1 - include/linux/pagemap.h | 2 ++ mm/filemap.c | 39 +++++++++++++++++++-------------------- mm/folio-compat.c | 6 ++++++ 4 files changed, 27 insertions(+), 21 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 72ca04f16711..145f045b0ddc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1970,7 +1970,6 @@ int get_kernel_pages(const struct kvec *iov, int nr_pages, int write, struct page **pages); struct page *get_dump_page(unsigned long addr); -extern int try_to_release_page(struct page * page, gfp_t gfp_mask); extern void do_invalidatepage(struct page *page, unsigned int offset, unsigned int length); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 7bef50ea5435..eb6e58e106c8 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -939,6 +939,8 @@ static inline void __delete_from_page_cache(struct page *page, void *shadow) void replace_page_cache_page(struct page *old, struct page *new); void delete_from_page_cache_batch(struct address_space *mapping, struct pagevec *pvec); +int try_to_release_page(struct page *page, gfp_t gfp); +bool filemap_release_folio(struct folio *folio, gfp_t gfp); loff_t mapping_seek_hole_data(struct address_space *, loff_t start, loff_t end, int whence); diff --git a/mm/filemap.c b/mm/filemap.c index 4ae9d5befffa..7a418f0012e5 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3920,33 +3920,32 @@ ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from) EXPORT_SYMBOL(generic_file_write_iter); /** - * try_to_release_page() - release old fs-specific metadata on a page + * filemap_release_folio() - Release fs-specific metadata on a folio. + * @folio: The folio which the kernel is trying to free. + * @gfp: Memory allocation flags (and I/O mode). * - * @page: the page which the kernel is trying to free - * @gfp_mask: memory allocation flags (and I/O mode) + * The address_space is trying to release any data attached to a folio + * (presumably at folio->private). * - * The address_space is to try to release any data against the page - * (presumably at page->private). + * This will also be called if the private_2 flag is set on a page, + * indicating that the folio has other metadata associated with it. * - * This may also be called if PG_fscache is set on a page, indicating that the - * page is known to the local caching routines. + * The @gfp argument specifies whether I/O may be performed to release + * this page (__GFP_IO), and whether the call may block + * (__GFP_RECLAIM & __GFP_FS). * - * The @gfp_mask argument specifies whether I/O may be performed to release - * this page (__GFP_IO), and whether the call may block (__GFP_RECLAIM & __GFP_FS). - * - * Return: %1 if the release was successful, otherwise return zero. + * Return: %true if the release was successful, otherwise %false. */ -int try_to_release_page(struct page *page, gfp_t gfp_mask) +bool filemap_release_folio(struct folio *folio, gfp_t gfp) { - struct address_space * const mapping = page->mapping; + struct address_space * const mapping = folio->mapping; - BUG_ON(!PageLocked(page)); - if (PageWriteback(page)) - return 0; + BUG_ON(!folio_test_locked(folio)); + if (folio_test_writeback(folio)) + return false; if (mapping && mapping->a_ops->releasepage) - return mapping->a_ops->releasepage(page, gfp_mask); - return try_to_free_buffers(page); + return mapping->a_ops->releasepage(&folio->page, gfp); + return try_to_free_buffers(&folio->page); } - -EXPORT_SYMBOL(try_to_release_page); +EXPORT_SYMBOL(filemap_release_folio); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 749a695b4217..749555a232a8 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -145,3 +145,9 @@ void delete_from_page_cache(struct page *page) { return filemap_remove_folio(page_folio(page)); } + +int try_to_release_page(struct page *page, gfp_t gfp) +{ + return filemap_release_folio(page_folio(page), gfp); +} +EXPORT_SYMBOL(try_to_release_page); From patchwork Wed Dec 8 04:22:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D106C433F5 for ; Wed, 8 Dec 2021 06:22:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99BE86B0081; Wed, 8 Dec 2021 01:19:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 94C966B0082; Wed, 8 Dec 2021 01:19:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EDCF6B0083; Wed, 8 Dec 2021 01:19:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay026.a.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 6FC1E6B0081 for ; Wed, 8 Dec 2021 01:19:08 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id E636E204B9; Wed, 8 Dec 2021 06:17:48 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 01FDE20C53; Wed, 8 Dec 2021 05:33:53 +0000 (UTC) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 016DE2139A for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) X-FDA: 78893332434.18.9C183A5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 88DB6400208A for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ywFxKMKzhbL8Q2M7esNnlvljBPXvITK4YoC9wi4Run8=; b=RI1CigpKup6foGd3wx015ayzus GnsmYsy+7LOThknpVPelSoX3PQgtjZvWY7JCdAqwrMRIPMC4knDtL7caU1nIPidrbzHa+O4ncd0xF 7ag6DrH2iwlAL0ZpyC5KPmJBW476Lq722GMDImeWhjrE7f/+/wbMAiREJtTmdMw/hzXTpcr5U4es2 7L1GtQFyBTJRDhrnIf3++LVfnKwwRuEBRqiJsMHwws59gpkiqHuM5XLktFCusOMnlL8C52z5YdqYW kK35ZBii/ZtaGYReJQqWWRDSa07diwp5nXWlAhnr+7TWdD0Ib0PKZA7ekluNp698ouBPGS11OuTHy 4DK/TvvA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU5-0084aE-PY; Wed, 08 Dec 2021 04:23:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 32/48] truncate: Add truncate_cleanup_folio() Date: Wed, 8 Dec 2021 04:22:40 +0000 Message-Id: <20211208042256.1923824-33-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RI1CigpK; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 88DB6400208A X-Stat-Signature: dgokaex9ey9oi6z1ah53z8nutbfwgcwt X-HE-Tag: 1638937396-590559 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert both callers of truncate_cleanup_page() to use truncate_cleanup_folio() instead. Signed-off-by: Matthew Wilcox (Oracle) --- mm/truncate.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index cc83a3f7c1ad..ab86b07c1e9c 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -177,21 +177,21 @@ void do_invalidatepage(struct page *page, unsigned int offset, * its lock, b) when a concurrent invalidate_mapping_pages got there first and * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space. */ -static void truncate_cleanup_page(struct page *page) +static void truncate_cleanup_folio(struct folio *folio) { - if (page_mapped(page)) - unmap_mapping_page(page); + if (folio_mapped(folio)) + unmap_mapping_page(&folio->page); - if (page_has_private(page)) - do_invalidatepage(page, 0, thp_size(page)); + if (folio_has_private(folio)) + do_invalidatepage(&folio->page, 0, folio_size(folio)); /* * Some filesystems seem to re-dirty the page even after * the VM has canceled the dirty bit (eg ext3 journaling). * Hence dirty accounting check is placed after invalidation. */ - cancel_dirty_page(page); - ClearPageMappedToDisk(page); + folio_cancel_dirty(folio); + folio_clear_mappedtodisk(folio); } /* @@ -220,13 +220,14 @@ invalidate_complete_page(struct address_space *mapping, struct page *page) int truncate_inode_page(struct address_space *mapping, struct page *page) { + struct folio *folio = page_folio(page); VM_BUG_ON_PAGE(PageTail(page), page); if (page->mapping != mapping) return -EIO; - truncate_cleanup_page(page); - delete_from_page_cache(page); + truncate_cleanup_folio(folio); + filemap_remove_folio(folio); return 0; } @@ -332,7 +333,7 @@ void truncate_inode_pages_range(struct address_space *mapping, index = indices[pagevec_count(&pvec) - 1] + 1; truncate_exceptional_pvec_entries(mapping, &pvec, indices); for (i = 0; i < pagevec_count(&pvec); i++) - truncate_cleanup_page(pvec.pages[i]); + truncate_cleanup_folio(page_folio(pvec.pages[i])); delete_from_page_cache_batch(mapping, &pvec); for (i = 0; i < pagevec_count(&pvec); i++) unlock_page(pvec.pages[i]); From patchwork Wed Dec 8 04:22:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 595A8C433F5 for ; Wed, 8 Dec 2021 06:45:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81EF56B0078; Wed, 8 Dec 2021 01:43:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CD916B007B; Wed, 8 Dec 2021 01:43:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 695306B007D; Wed, 8 Dec 2021 01:43:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay028.a.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 5A46A6B0078 for ; Wed, 8 Dec 2021 01:43:36 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 39197801B6 for ; Wed, 8 Dec 2021 04:23:18 +0000 (UTC) X-FDA: 78893332434.04.9727A03 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id AA6074002087 for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oP34gRnxYnpVikw5uBW13SizoUTUkR9UbaCi/Ytt8fE=; b=N+pqVt0IZ4UHZiUWl3CkDM93hw u5Q9Lvmk1GBLboQnh/yxZ5BtO38i98Eg11PDRhOyb+wEAVUbyB9sREQCKHFaM7tuXP/o1wR18GvBn fDr5b2V+tETvZj+ZSlHnBGS6irXnUiAhfDo9gA7K05UNKoVwG22XMn1ThcAPInCvFsnC36PIwYOLU WVZ3gd51j7lRQVENu0wEPd7QVeTuLKgIsFznCD9QWZqOouBfG4SI83dRygt9jPU1WO9Bvws5OYAqf Wl4jXTPX7PCixZbIc9zZHdFogGFECHz/m9EfdTutmKg6QfAdQjS8S+yDEeBpTTanHv9mbhVuXzh04 HO3CS6xQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU5-0084ab-U9; Wed, 08 Dec 2021 04:23:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 33/48] mm: Add unmap_mapping_folio() Date: Wed, 8 Dec 2021 04:22:41 +0000 Message-Id: <20211208042256.1923824-34-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=N+pqVt0I; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AA6074002087 X-Stat-Signature: y77wtp5huh4tdefbpkdxsah9335o31qu X-HE-Tag: 1638937397-592258 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert both callers of unmap_mapping_page() to call unmap_mapping_folio() instead. Also move zap_details from linux/mm.h to mm/internal.h Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/mm.h | 24 ------------------------ mm/internal.h | 25 ++++++++++++++++++++++++- mm/memory.c | 27 +++++++++++++-------------- mm/truncate.c | 4 ++-- 4 files changed, 39 insertions(+), 41 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 145f045b0ddc..c9cdb26802fb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1825,28 +1825,6 @@ static inline bool can_do_mlock(void) { return false; } extern int user_shm_lock(size_t, struct ucounts *); extern void user_shm_unlock(size_t, struct ucounts *); -/* - * Parameter block passed down to zap_pte_range in exceptional cases. - */ -struct zap_details { - struct address_space *zap_mapping; /* Check page->mapping if set */ - struct page *single_page; /* Locked page to be unmapped */ -}; - -/* - * We set details->zap_mappings when we want to unmap shared but keep private - * pages. Return true if skip zapping this page, false otherwise. - */ -static inline bool -zap_skip_check_mapping(struct zap_details *details, struct page *page) -{ - if (!details || !page) - return false; - - return details->zap_mapping && - (details->zap_mapping != page_rmapping(page)); -} - struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, @@ -1892,7 +1870,6 @@ extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, extern int fixup_user_fault(struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); -void unmap_mapping_page(struct page *page); void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows); void unmap_mapping_range(struct address_space *mapping, @@ -1913,7 +1890,6 @@ static inline int fixup_user_fault(struct mm_struct *mm, unsigned long address, BUG(); return -EFAULT; } -static inline void unmap_mapping_page(struct page *page) { } static inline void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { } static inline void unmap_mapping_range(struct address_space *mapping, diff --git a/mm/internal.h b/mm/internal.h index 3b79a5c9427a..3f359f4830da 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -74,6 +74,28 @@ static inline bool can_madv_lru_vma(struct vm_area_struct *vma) return !(vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP)); } +/* + * Parameter block passed down to zap_pte_range in exceptional cases. + */ +struct zap_details { + struct address_space *zap_mapping; /* Check page->mapping if set */ + struct folio *single_folio; /* Locked folio to be unmapped */ +}; + +/* + * We set details->zap_mappings when we want to unmap shared but keep private + * pages. Return true if skip zapping this page, false otherwise. + */ +static inline bool +zap_skip_check_mapping(struct zap_details *details, struct page *page) +{ + if (!details || !page) + return false; + + return details->zap_mapping && + (details->zap_mapping != page_rmapping(page)); +} + void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, @@ -388,6 +410,7 @@ void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma, void __vma_unlink_list(struct mm_struct *mm, struct vm_area_struct *vma); #ifdef CONFIG_MMU +void unmap_mapping_folio(struct folio *folio); extern long populate_vma_page_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, int *locked); extern long faultin_vma_page_range(struct vm_area_struct *vma, @@ -491,8 +514,8 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf, } return fpin; } - #else /* !CONFIG_MMU */ +static inline void unmap_mapping_folio(struct folio *folio) { } static inline void clear_page_mlock(struct page *page) { } static inline void mlock_vma_page(struct page *page) { } static inline void vunmap_range_noflush(unsigned long start, unsigned long end) diff --git a/mm/memory.c b/mm/memory.c index 8f1de811a1dc..a86027026f2a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1443,8 +1443,8 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, else if (zap_huge_pmd(tlb, vma, pmd, addr)) goto next; /* fall through */ - } else if (details && details->single_page && - PageTransCompound(details->single_page) && + } else if (details && details->single_folio && + folio_test_pmd_mappable(details->single_folio) && next - addr == HPAGE_PMD_SIZE && pmd_none(*pmd)) { spinlock_t *ptl = pmd_lock(tlb->mm, pmd); /* @@ -3332,31 +3332,30 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, } /** - * unmap_mapping_page() - Unmap single page from processes. - * @page: The locked page to be unmapped. + * unmap_mapping_folio() - Unmap single folio from processes. + * @folio: The locked folio to be unmapped. * - * Unmap this page from any userspace process which still has it mmaped. + * Unmap this folio from any userspace process which still has it mmaped. * Typically, for efficiency, the range of nearby pages has already been * unmapped by unmap_mapping_pages() or unmap_mapping_range(). But once - * truncation or invalidation holds the lock on a page, it may find that - * the page has been remapped again: and then uses unmap_mapping_page() + * truncation or invalidation holds the lock on a folio, it may find that + * the page has been remapped again: and then uses unmap_mapping_folio() * to unmap it finally. */ -void unmap_mapping_page(struct page *page) +void unmap_mapping_folio(struct folio *folio) { - struct address_space *mapping = page->mapping; + struct address_space *mapping = folio->mapping; struct zap_details details = { }; pgoff_t first_index; pgoff_t last_index; - VM_BUG_ON(!PageLocked(page)); - VM_BUG_ON(PageTail(page)); + VM_BUG_ON(!folio_test_locked(folio)); - first_index = page->index; - last_index = page->index + thp_nr_pages(page) - 1; + first_index = folio->index; + last_index = folio->index + folio_nr_pages(folio) - 1; details.zap_mapping = mapping; - details.single_page = page; + details.single_folio = folio; i_mmap_lock_write(mapping); if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) diff --git a/mm/truncate.c b/mm/truncate.c index ab86b07c1e9c..c98feea75a10 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -180,7 +180,7 @@ void do_invalidatepage(struct page *page, unsigned int offset, static void truncate_cleanup_folio(struct folio *folio) { if (folio_mapped(folio)) - unmap_mapping_page(&folio->page); + unmap_mapping_folio(folio); if (folio_has_private(folio)) do_invalidatepage(&folio->page, 0, folio_size(folio)); @@ -670,7 +670,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, wait_on_page_writeback(page); if (page_mapped(page)) - unmap_mapping_page(page); + unmap_mapping_folio(page_folio(page)); BUG_ON(page_mapped(page)); ret2 = do_launder_page(mapping, page); From patchwork Wed Dec 8 04:22:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACF3CC433EF for ; Wed, 8 Dec 2021 06:30:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 466D66B0071; Wed, 8 Dec 2021 01:23:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EF6C6B0073; Wed, 8 Dec 2021 01:23:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28F506B0075; Wed, 8 Dec 2021 01:23:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 141116B0071 for ; Wed, 8 Dec 2021 01:23:40 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 46EFF6059B; Wed, 8 Dec 2021 05:38:47 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 32BF3604C3; Wed, 8 Dec 2021 05:32:07 +0000 (UTC) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7E74560A10 for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) X-FDA: 78893332434.25.7CA8A67 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 26482400208A for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rsEJxdywVEoM0ervtUyhDSP7KMwK7lVPrAeU8/731OE=; b=qz7qSp9TRh7ufJAZHOEWtvrC1s +otedc/SjuvSkhoNGBamGLmEk6/Nqh1nBjaa3f4ZIfkFVTGWbrZA4uz+VmGndZ/S1WDXB+Qs7pYqy YrOx1isibEPxZqb+X8j1xW74iFmIGSY95jRod+875T3k8ZuiZpsa0H7Us1B7vmtyeSsH/NKl0MOer Dv0rPgXENpRCozTrexfXAn+QHK6vE1pZrQ1keQNYqLTv1roMHTVFson/JAASxTQLnkenlldv6ePVh MXsyLRBp19oDd7+4MaJdGCl3lBIvmH/5qXM2pPbUs/RAyz8g66JG5gS4qCLXIRfxcosuHedgsB8Xo O+p6ERYg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU6-0084ax-5i; Wed, 08 Dec 2021 04:23:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 34/48] shmem: Convert part of shmem_undo_range() to use a folio Date: Wed, 8 Dec 2021 04:22:42 +0000 Message-Id: <20211208042256.1923824-35-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qz7qSp9T; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 26482400208A X-Stat-Signature: hsowd6y43dr443wbfzirtzasuu84cx6z X-HE-Tag: 1638937397-72670 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: find_lock_entries() never returns tail pages. We cannot use page_folio() here as the pagevec may also contain swap entries, so simply cast. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- mm/shmem.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 18f93c2d68f1..40da9075374b 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -936,22 +936,22 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, while (index < end && find_lock_entries(mapping, index, end - 1, &pvec, indices)) { for (i = 0; i < pagevec_count(&pvec); i++) { - struct page *page = pvec.pages[i]; + struct folio *folio = (struct folio *)pvec.pages[i]; index = indices[i]; - if (xa_is_value(page)) { + if (xa_is_value(folio)) { if (unfalloc) continue; nr_swaps_freed += !shmem_free_swap(mapping, - index, page); + index, folio); continue; } - index += thp_nr_pages(page) - 1; + index += folio_nr_pages(folio) - 1; - if (!unfalloc || !PageUptodate(page)) - truncate_inode_page(mapping, page); - unlock_page(page); + if (!unfalloc || !folio_test_uptodate(folio)) + truncate_inode_page(mapping, &folio->page); + folio_unlock(folio); } pagevec_remove_exceptionals(&pvec); pagevec_release(&pvec); From patchwork Wed Dec 8 04:22:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F25D7C433F5 for ; Wed, 8 Dec 2021 06:45:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3622D6B0075; Wed, 8 Dec 2021 01:43:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3128D6B0078; Wed, 8 Dec 2021 01:43:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2016E6B007B; Wed, 8 Dec 2021 01:43:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id 059286B0075 for ; Wed, 8 Dec 2021 01:43:36 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 54507601AB for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332350.15.F4F9D96 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id DB46B900009E for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=lNZrSgSL6qgITtDseTbflhdNUzinpmzsRmApffTrpVs=; b=jZSVSzz01fdqJru3kVCNcfaEDw f5oPA2b926j3xg7y6R58nMzkXIR+QZXoSQiWqXTFP9MhsQ6W4sLJkGuZE+JScvPoIE/l6bXttrl44 yZ9Ovuy5fpn3Nldm+8dLQPfuxi3MDyhTnMdyP+Jv2EXUgAmUAqRYeLqFmDSecvgnNgW/S6jsGH2ge h9ueqx68nFVzmH+bHiYVn3C1E9pc6xK+0XrXh6jbdabyq4xO4Vi8vv/conyrLzoAIxrREMzaiyWSE +8xLK7e3ZdUlogQG9yEdY4QYSI5OMb7M0YNmYqsxfYP2o7OYbuJHB9g30zqvHgxfeAyUMaZ1IQQg0 0USuxYZg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU6-0084b9-BQ; Wed, 08 Dec 2021 04:23:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 35/48] truncate,shmem: Add truncate_inode_folio() Date: Wed, 8 Dec 2021 04:22:43 +0000 Message-Id: <20211208042256.1923824-36-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: DB46B900009E X-Stat-Signature: khemgzguxdrkucg89sg8po1w1fu3pkha Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jZSVSzz0; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937395-134023 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert all callers of truncate_inode_page() to call truncate_inode_folio() instead, and move the declaration to mm/internal.h. Move the assertion that the caller is not passing in a tail page to generic_error_remove_page(). We can't entirely remove the struct page from the callers yet because the page pointer in the pvec might be a shadow/dax/swap entry instead of actually a page. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 1 - mm/internal.h | 1 + mm/shmem.c | 5 +++-- mm/truncate.c | 23 ++++++++++++----------- 4 files changed, 16 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c9cdb26802fb..d8b7d7ed14dd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1859,7 +1859,6 @@ extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); void truncate_pagecache_range(struct inode *inode, loff_t offset, loff_t end); -int truncate_inode_page(struct address_space *mapping, struct page *page); int generic_error_remove_page(struct address_space *mapping, struct page *page); int invalidate_inode_page(struct page *page); diff --git a/mm/internal.h b/mm/internal.h index 3f359f4830da..b05515d3b07b 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -113,6 +113,7 @@ static inline void force_page_cache_readahead(struct address_space *mapping, unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct pagevec *pvec, pgoff_t *indices); +int truncate_inode_folio(struct address_space *mapping, struct folio *folio); /** * folio_evictable - Test whether a folio is evictable. diff --git a/mm/shmem.c b/mm/shmem.c index 40da9075374b..dbef008fb6e5 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -950,7 +950,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, index += folio_nr_pages(folio) - 1; if (!unfalloc || !folio_test_uptodate(folio)) - truncate_inode_page(mapping, &folio->page); + truncate_inode_folio(mapping, folio); folio_unlock(folio); } pagevec_remove_exceptionals(&pvec); @@ -1027,7 +1027,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, } VM_BUG_ON_PAGE(PageWriteback(page), page); if (shmem_punch_compound(page, start, end)) - truncate_inode_page(mapping, page); + truncate_inode_folio(mapping, + page_folio(page)); else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { /* Wipe the page and don't get stuck */ clear_highpage(page); diff --git a/mm/truncate.c b/mm/truncate.c index c98feea75a10..0000424fc56b 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -218,12 +218,9 @@ invalidate_complete_page(struct address_space *mapping, struct page *page) return ret; } -int truncate_inode_page(struct address_space *mapping, struct page *page) +int truncate_inode_folio(struct address_space *mapping, struct folio *folio) { - struct folio *folio = page_folio(page); - VM_BUG_ON_PAGE(PageTail(page), page); - - if (page->mapping != mapping) + if (folio->mapping != mapping) return -EIO; truncate_cleanup_folio(folio); @@ -236,6 +233,8 @@ int truncate_inode_page(struct address_space *mapping, struct page *page) */ int generic_error_remove_page(struct address_space *mapping, struct page *page) { + VM_BUG_ON_PAGE(PageTail(page), page); + if (!mapping) return -EINVAL; /* @@ -244,7 +243,7 @@ int generic_error_remove_page(struct address_space *mapping, struct page *page) */ if (!S_ISREG(mapping->host->i_mode)) return -EIO; - return truncate_inode_page(mapping, page); + return truncate_inode_folio(mapping, page_folio(page)); } EXPORT_SYMBOL(generic_error_remove_page); @@ -395,18 +394,20 @@ void truncate_inode_pages_range(struct address_space *mapping, for (i = 0; i < pagevec_count(&pvec); i++) { struct page *page = pvec.pages[i]; + struct folio *folio; /* We rely upon deletion not changing page->index */ index = indices[i]; if (xa_is_value(page)) continue; + folio = page_folio(page); - lock_page(page); - WARN_ON(page_to_index(page) != index); - wait_on_page_writeback(page); - truncate_inode_page(mapping, page); - unlock_page(page); + folio_lock(folio); + VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio); + folio_wait_writeback(folio); + truncate_inode_folio(mapping, folio); + folio_unlock(folio); } truncate_exceptional_pvec_entries(mapping, &pvec, indices); pagevec_release(&pvec); From patchwork Wed Dec 8 04:22:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E9CBC433EF for ; Wed, 8 Dec 2021 06:24:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B977F6B0087; Wed, 8 Dec 2021 01:19:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B483E6B0088; Wed, 8 Dec 2021 01:19:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0F896B0089; Wed, 8 Dec 2021 01:19:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay028.a.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 8F6D86B0087 for ; Wed, 8 Dec 2021 01:19:26 -0500 (EST) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id 9B1423C9; Wed, 8 Dec 2021 06:18:01 +0000 (UTC) Received: by unirelay10.hostedemail.com (Postfix, from userid 108) id D30A43EC; Wed, 8 Dec 2021 05:34:11 +0000 (UTC) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B9B61542 for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) X-FDA: 78893332392.13.665CF56 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 4C3C9100002 for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=R0LtaHX+Nty2s7UBxTeFk5sP72cAhoO4NqqGhAST8A0=; b=bf4bCUp3zS1a+WG54pBkLY+ZTG yjbb50yIBv+0/xSJNVl/oepDXVAF0DloEfg4QJnMc2jArGc1q47spasIf0KIrKG1tDLRG69IwCtNP QlrLbFVPbV0HhHqASRhPcZRVHfhhAKShi4XVMlqHCADqG8DlRma+bAnk0c2reZLnJk0LuEMQoF4Ix Ra3l6/Xwu/kwqCp2eLAbO3W97w8A9DKHl827qZIqFKVrjn7viaXgtwarqGsRsmZPJ7Ca3CxZX68Mx 4PBnnswWljYP1wMpUAbGzqqV1vug9aF4CiHxqprASXEtRoZXJR+cyfrzKY9IIoPiEn4zpYSbJDUHm +37gaQcg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU6-0084bf-HN; Wed, 08 Dec 2021 04:23:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 36/48] truncate: Skip known-truncated indices Date: Wed, 8 Dec 2021 04:22:44 +0000 Message-Id: <20211208042256.1923824-37-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4C3C9100002 X-Stat-Signature: 8imxryqabxwch9n4f5749s7wsyfjyw5w Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bf4bCUp3; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937397-410217 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we've truncated an entire folio, we can skip over all the indices covered by this folio. Signed-off-by: Matthew Wilcox (Oracle) --- mm/truncate.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/truncate.c b/mm/truncate.c index 0000424fc56b..0df420c1cf5b 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -408,6 +408,7 @@ void truncate_inode_pages_range(struct address_space *mapping, folio_wait_writeback(folio); truncate_inode_folio(mapping, folio); folio_unlock(folio); + index = folio_index(folio) + folio_nr_pages(folio) - 1; } truncate_exceptional_pvec_entries(mapping, &pvec, indices); pagevec_release(&pvec); From patchwork Wed Dec 8 04:22:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B2E2C433EF for ; Wed, 8 Dec 2021 06:46:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A20D6B007B; Wed, 8 Dec 2021 01:43:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 250356B007D; Wed, 8 Dec 2021 01:43:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13E716B007E; Wed, 8 Dec 2021 01:43:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id 04E0C6B007B for ; Wed, 8 Dec 2021 01:43:37 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 7AA4512019C for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332392.10.962853C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 2492030000A8 for ; Wed, 8 Dec 2021 04:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AMdXWqM9y2naxpi8sOt9C2UydZsRiUJuoqUPrvDFvVk=; b=ByebKbQ1Gn0JnlxAc57DWSS9Ff Cc3iOKVahJoBfHov4cJb+ytLbC3XERdLjG1wLH5fkj/CO986sltyZgmUNUYBtC6b2n+Wa+xhZ+wbX PzYHzbmc/ZjhQtIwCZdjEqRSGyadsj0qjbQ6ca4Qd5O4CeeRyl5C2RoxCoybIwrP8YndDmRb9aYYo rGLA4ubMV4bwH7Q4byEX0oNFPIFEhVKi3AwADwxbozACEx3Jt/+dK4U6EfXkBeKDtVsBx3AYlwXXy 1MOzHPQMQkBAzNgQgcFn4AczCnKYCMItIIIR6b7n1iB074x0TB2xxanZIZb68/hufQF+xP7WtQhQr GhyvAcjA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU6-0084br-Oc; Wed, 08 Dec 2021 04:23:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 37/48] truncate: Convert invalidate_inode_pages2_range() to use a folio Date: Wed, 8 Dec 2021 04:22:45 +0000 Message-Id: <20211208042256.1923824-38-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2492030000A8 X-Stat-Signature: kthkdkontswghf4h96dnzpoupbujkduf Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ByebKbQ1; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937395-557160 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we're going to unmap a folio, we have to be sure to unmap the entire folio, not just the part of it which lies after the search index. We cannot yet remove the struct page from invalidate_inode_pages2_range() because the page pointer in the pvec might be a shadow/dax/swap entry instead of actually a page. Signed-off-by: Matthew Wilcox (Oracle) --- mm/truncate.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 0df420c1cf5b..ef6980b240e2 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -642,8 +642,9 @@ int invalidate_inode_pages2_range(struct address_space *mapping, while (find_get_entries(mapping, index, end, &pvec, indices)) { for (i = 0; i < pagevec_count(&pvec); i++) { struct page *page = pvec.pages[i]; + struct folio *folio; - /* We rely upon deletion not changing page->index */ + /* We rely upon deletion not changing folio->index */ index = indices[i]; if (xa_is_value(page)) { @@ -652,10 +653,11 @@ int invalidate_inode_pages2_range(struct address_space *mapping, ret = -EBUSY; continue; } + folio = page_folio(page); - if (!did_range_unmap && page_mapped(page)) { + if (!did_range_unmap && folio_mapped(folio)) { /* - * If page is mapped, before taking its lock, + * If folio is mapped, before taking its lock, * zap the rest of the file in one hit. */ unmap_mapping_pages(mapping, index, @@ -663,26 +665,27 @@ int invalidate_inode_pages2_range(struct address_space *mapping, did_range_unmap = 1; } - lock_page(page); - WARN_ON(page_to_index(page) != index); - if (page->mapping != mapping) { - unlock_page(page); + folio_lock(folio); + VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio); + if (folio->mapping != mapping) { + folio_unlock(folio); continue; } - wait_on_page_writeback(page); + folio_wait_writeback(folio); - if (page_mapped(page)) - unmap_mapping_folio(page_folio(page)); - BUG_ON(page_mapped(page)); + if (folio_mapped(folio)) + unmap_mapping_folio(folio); + BUG_ON(folio_mapped(folio)); - ret2 = do_launder_page(mapping, page); + ret2 = do_launder_page(mapping, &folio->page); if (ret2 == 0) { - if (!invalidate_complete_page2(mapping, page)) + if (!invalidate_complete_page2(mapping, + &folio->page)) ret2 = -EBUSY; } if (ret2 < 0) ret = ret2; - unlock_page(page); + folio_unlock(folio); } pagevec_remove_exceptionals(&pvec); pagevec_release(&pvec); From patchwork Wed Dec 8 04:22:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B304DC433EF for ; Wed, 8 Dec 2021 06:21:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E307D6B0080; Wed, 8 Dec 2021 01:19:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DDFEC6B0081; Wed, 8 Dec 2021 01:19:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C80C46B0082; Wed, 8 Dec 2021 01:19:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay027.a.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id B8E7F6B0080 for ; Wed, 8 Dec 2021 01:19:00 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id B1AA121003; Wed, 8 Dec 2021 06:17:47 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 681EC2100A; Wed, 8 Dec 2021 05:33:54 +0000 (UTC) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9A9F3213A1 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) X-FDA: 78893332392.03.F62F16C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 5BBCF40004 for ; Wed, 8 Dec 2021 04:23:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=O2rXYGwuMrrnf3QKuW5hfKeCWLF/1U/uGtPuofV+Hos=; b=mnArESWZgEj9j0aKIwRhyceMXW CoTGKDMFwU+07rPdBMg2zmf8ogUqG8Igo1xjVIMo3XeuSdMs+Qyz/Wgq/8CEZymkWRcenJ7dBXTVQ 6TeDnVoedFJPwE7j4yxDzU0ozth4XPq2L5CV3ZaEl3Q5X1uEP/MOEGLI9QGd3o3/qkkyl/eF6lNNg ckx2pFteVauMPiNWuFBbo9HnvhIRfNauxGeB/SOGwgsDjgJcPNccDbwTVFcJAkREaxZyap44L6dVZ lfDNly5QXTssa5y/LWowHDU8pDXZj1Cm+s6UU6b5Y5CyiUycgX+lGjFRQu4W6AO8ARhi+0o/YSEJn 4P4nGkug==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU6-0084cD-St; Wed, 08 Dec 2021 04:23:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 38/48] truncate: Add invalidate_complete_folio2() Date: Wed, 8 Dec 2021 04:22:46 +0000 Message-Id: <20211208042256.1923824-39-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5BBCF40004 X-Stat-Signature: mcas4fgsbyb8d4yexkw8r9q6b8bkszps Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mnArESWZ; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937396-88328 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert invalidate_complete_page2() to invalidate_complete_folio2(). Use filemap_free_folio() to free the page instead of calling ->freepage manually. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 3 +-- mm/internal.h | 1 + mm/truncate.c | 23 ++++++++++------------- 3 files changed, 12 insertions(+), 15 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7a418f0012e5..fb3cdb7aeffc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -229,8 +229,7 @@ void __filemap_remove_folio(struct folio *folio, void *shadow) page_cache_delete(mapping, folio, shadow); } -static void filemap_free_folio(struct address_space *mapping, - struct folio *folio) +void filemap_free_folio(struct address_space *mapping, struct folio *folio) { void (*freepage)(struct page *); diff --git a/mm/internal.h b/mm/internal.h index b05515d3b07b..d3c7b35934ed 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -113,6 +113,7 @@ static inline void force_page_cache_readahead(struct address_space *mapping, unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct pagevec *pvec, pgoff_t *indices); +void filemap_free_folio(struct address_space *mapping, struct folio *folio); int truncate_inode_folio(struct address_space *mapping, struct folio *folio); /** diff --git a/mm/truncate.c b/mm/truncate.c index ef6980b240e2..5370094641d6 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -571,31 +571,29 @@ void invalidate_mapping_pagevec(struct address_space *mapping, * shrink_page_list() has a temp ref on them, or because they're transiently * sitting in the lru_cache_add() pagevecs. */ -static int -invalidate_complete_page2(struct address_space *mapping, struct page *page) +static int invalidate_complete_folio2(struct address_space *mapping, + struct folio *folio) { - if (page->mapping != mapping) + if (folio->mapping != mapping) return 0; - if (page_has_private(page) && !try_to_release_page(page, GFP_KERNEL)) + if (folio_has_private(folio) && + !filemap_release_folio(folio, GFP_KERNEL)) return 0; spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); - if (PageDirty(page)) + if (folio_test_dirty(folio)) goto failed; - BUG_ON(page_has_private(page)); - __delete_from_page_cache(page, NULL); + BUG_ON(folio_has_private(folio)); + __filemap_remove_folio(folio, NULL); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock); - if (mapping->a_ops->freepage) - mapping->a_ops->freepage(page); - - put_page(page); /* pagecache ref */ + filemap_free_folio(mapping, folio); return 1; failed: xa_unlock_irq(&mapping->i_pages); @@ -679,8 +677,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, ret2 = do_launder_page(mapping, &folio->page); if (ret2 == 0) { - if (!invalidate_complete_page2(mapping, - &folio->page)) + if (!invalidate_complete_folio2(mapping, folio)) ret2 = -EBUSY; } if (ret2 < 0) From patchwork Wed Dec 8 04:22:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B377C433F5 for ; Wed, 8 Dec 2021 06:05:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84C586B0073; Wed, 8 Dec 2021 01:04:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67BB66B0078; Wed, 8 Dec 2021 01:04:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 456EE6B0074; Wed, 8 Dec 2021 01:04:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by kanga.kvack.org (Postfix) with ESMTP id 29F036B0071 for ; Wed, 8 Dec 2021 01:04:43 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id D3B73605AD; Wed, 8 Dec 2021 05:38:47 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 35C39604D5; Wed, 8 Dec 2021 05:32:08 +0000 (UTC) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5C0D960A0F for ; Wed, 8 Dec 2021 04:23:18 +0000 (UTC) X-FDA: 78893332476.30.C37890A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id DCC5B100002 for ; Wed, 8 Dec 2021 04:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PRh0VjIPJ/xdMLO8RLWdCCSHILXWRHuUOZvS3c2PDV4=; b=s97Po3qmIMWLOvQDRaqmMdsAs0 XR/0Vf2i8cLRnwljdB4wuLGg+s3w2/e1Xjnr6RC7VJmHL9whKiJcaKGFXMD90JPld+IAhuJH8wzJO NP8UchXQyG78QbxcL8wy/Yg4wOlB8A0Jg0ztwZYK0tK3UMr3XvB3DoWypdkU/FTKcyFlWgRNLtnck i/5O008ECD51WMWj3fz7oyn36xy36mMdfqC6Fe+LDzDP8JebbHStijG9Giu2jIhasyBnzn9nJMRnf vNJnMQ+K5gqYb9BxguUDhBlvIOogRkyV0zA1dcZX/ottrv6pWiinRtyDYwVGDn9+q6pJVCgA8i69O 8clIHFQQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU7-0084cK-0S; Wed, 08 Dec 2021 04:23:15 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 39/48] filemap: Convert filemap_read() to use a folio Date: Wed, 8 Dec 2021 04:22:47 +0000 Message-Id: <20211208042256.1923824-40-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DCC5B100002 X-Stat-Signature: oi5rtcb1kbop9u81zmdjxtfkrdu63dk6 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=s97Po3qm; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937397-817479 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We know the pagevec always contains folios, but use page_folio() anyway instead of casting. Removes a few calls to legacy functions. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index fb3cdb7aeffc..91399027b349 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2668,30 +2668,26 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, mark_page_accessed(pvec.pages[0]); for (i = 0; i < pagevec_count(&pvec); i++) { - struct page *page = pvec.pages[i]; - size_t page_size = thp_size(page); - size_t offset = iocb->ki_pos & (page_size - 1); + struct folio *folio = page_folio(pvec.pages[i]); + size_t fsize = folio_size(folio); + size_t offset = iocb->ki_pos & (fsize - 1); size_t bytes = min_t(loff_t, end_offset - iocb->ki_pos, - page_size - offset); + fsize - offset); size_t copied; - if (end_offset < page_offset(page)) + if (end_offset < folio_pos(folio)) break; if (i > 0) - mark_page_accessed(page); + folio_mark_accessed(folio); /* - * If users can be writing to this page using arbitrary - * virtual addresses, take care about potential aliasing - * before reading the page on the kernel side. + * If users can be writing to this folio using arbitrary + * virtual addresses, take care of potential aliasing + * before reading the folio on the kernel side. */ - if (writably_mapped) { - int j; - - for (j = 0; j < thp_nr_pages(page); j++) - flush_dcache_page(page + j); - } + if (writably_mapped) + flush_dcache_folio(folio); - copied = copy_page_to_iter(page, offset, bytes, iter); + copied = copy_folio_to_iter(folio, offset, bytes, iter); already_read += copied; iocb->ki_pos += copied; From patchwork Wed Dec 8 04:22:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F11BAC433EF for ; Wed, 8 Dec 2021 06:17:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 867466B0074; Wed, 8 Dec 2021 01:17:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 816A56B0075; Wed, 8 Dec 2021 01:17:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B6B66B0078; Wed, 8 Dec 2021 01:17:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay027.a.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 5DB856B0074 for ; Wed, 8 Dec 2021 01:17:39 -0500 (EST) Received: by unirelay06.hostedemail.com (Postfix, from userid 108) id 5E9B020E42; Wed, 8 Dec 2021 06:14:59 +0000 (UTC) Received: by unirelay06.hostedemail.com (Postfix, from userid 108) id CA0E820585; Wed, 8 Dec 2021 05:33:16 +0000 (UTC) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8B1D221021 for ; Wed, 8 Dec 2021 04:59:13 +0000 (UTC) X-FDA: 78893422944.15.B134990 Received: from relay.hostedemail.com (relay029.a.hostedemail.com [64.99.140.29]) by imf01.hostedemail.com (Postfix) with ESMTP id 1A4F240002 for ; Wed, 8 Dec 2021 04:59:12 +0000 (UTC) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E998021265 for ; Wed, 8 Dec 2021 04:49:50 +0000 (UTC) X-FDA: 78893399340.16.8AE7C05 Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by imf19.hostedemail.com (Postfix) with ESMTP id 7DDACB000093 for ; Wed, 8 Dec 2021 04:49:50 +0000 (UTC) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1BCE460594 for ; Wed, 8 Dec 2021 04:39:51 +0000 (UTC) X-FDA: 78893374182.03.8519ED0 Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by imf31.hostedemail.com (Postfix) with ESMTP id B5A421046300 for ; Wed, 8 Dec 2021 04:39:50 +0000 (UTC) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1C84960A0C for ; Wed, 8 Dec 2021 04:23:21 +0000 (UTC) X-FDA: 78893332602.08.899015A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 9C883100005 for ; Wed, 8 Dec 2021 04:23:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gaCPSqljRYIcG8B5kqR4NUVjXXOP3yiRv/GkZidMzj4=; b=S5ZLi4rSMuQa9yPXFQS4hNHrCv 4ViEp2NFC2FO7kcsVp72QAHCUhSUFasNqc+cD4nw6Xu22PvDhg9oIQ/f4tY5HKwzVqj4u3sGmirEn mx3R/o12jzsmFAORORDzUQ8w1PhePn72F2VMVXXotSN68WWLb6cQIWfPPi96mAZSgnqduPpLMbc6+ ksnM1SelAri6bEB9c3YBESfyp8C8Lv+8WGOW0ZcGngZFBZ4Pl8E4JiC4Q4eAQh4WWmy5uDEt9MRSS /i2RMwSdzo/Vo6l8YMMpUjBEoC8Pwe16IY8dLFR5WHCCJfDpYbgT/fqSnh2KbV3iTXgFU/5gpCVQq CzmgjyOw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU7-0084cX-6x; Wed, 08 Dec 2021 04:23:15 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 40/48] filemap: Convert filemap_get_read_batch() to use a folio_batch Date: Wed, 8 Dec 2021 04:22:48 +0000 Message-Id: <20211208042256.1923824-41-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937400-792298 X-HE-Tag-Orig: 1638938390-415105 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1A4F240002 X-Stat-Signature: yikwkuqcd6yqimhuwtywuhbf5qk53338 X-HE-Tag-Orig: 1638938990-206585 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=S5ZLi4rS; spf=none (imf01.hostedemail.com: domain of FILTER%FILTER%FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.29) smtp.mailfrom=FILTER%FILTER%FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638939552-316962 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This change ripples all the way through the filemap_read() call chain and removes a lot of messing about converting folios to pages and back again. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 65 ++++++++++++++++++++++++++-------------------------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 91399027b349..38726ca96f0e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2325,16 +2325,16 @@ static void shrink_readahead_size_eio(struct file_ra_state *ra) } /* - * filemap_get_read_batch - Get a batch of pages for read + * filemap_get_read_batch - Get a batch of folios for read * - * Get a batch of pages which represent a contiguous range of bytes - * in the file. No tail pages will be returned. If @index is in the - * middle of a THP, the entire THP will be returned. The last page in - * the batch may have Readahead set or be not Uptodate so that the - * caller can take the appropriate action. + * Get a batch of folios which represent a contiguous range of bytes in + * the file. No exceptional entries will be returned. If @index is in + * the middle of a folio, the entire folio will be returned. The last + * folio in the batch may have the readahead flag set or the uptodate flag + * clear so that the caller can take the appropriate action. */ static void filemap_get_read_batch(struct address_space *mapping, - pgoff_t index, pgoff_t max, struct pagevec *pvec) + pgoff_t index, pgoff_t max, struct folio_batch *fbatch) { XA_STATE(xas, &mapping->i_pages, index); struct folio *folio; @@ -2349,9 +2349,9 @@ static void filemap_get_read_batch(struct address_space *mapping, goto retry; if (unlikely(folio != xas_reload(&xas))) - goto put_page; + goto put_folio; - if (!pagevec_add(pvec, &folio->page)) + if (!folio_batch_add(fbatch, folio)) break; if (!folio_test_uptodate(folio)) break; @@ -2360,7 +2360,7 @@ static void filemap_get_read_batch(struct address_space *mapping, xas.xa_index = folio->index + folio_nr_pages(folio) - 1; xas.xa_offset = (xas.xa_index >> xas.xa_shift) & XA_CHUNK_MASK; continue; -put_page: +put_folio: folio_put(folio); retry: xas_reset(&xas); @@ -2475,7 +2475,7 @@ static int filemap_update_page(struct kiocb *iocb, static int filemap_create_folio(struct file *file, struct address_space *mapping, pgoff_t index, - struct pagevec *pvec) + struct folio_batch *fbatch) { struct folio *folio; int error; @@ -2510,7 +2510,7 @@ static int filemap_create_folio(struct file *file, goto error; filemap_invalidate_unlock_shared(mapping); - pagevec_add(pvec, &folio->page); + folio_batch_add(fbatch, folio); return 0; error: filemap_invalidate_unlock_shared(mapping); @@ -2531,7 +2531,7 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, } static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, - struct pagevec *pvec) + struct folio_batch *fbatch) { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; @@ -2546,32 +2546,33 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, if (fatal_signal_pending(current)) return -EINTR; - filemap_get_read_batch(mapping, index, last_index, pvec); - if (!pagevec_count(pvec)) { + filemap_get_read_batch(mapping, index, last_index, fbatch); + if (!folio_batch_count(fbatch)) { if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; page_cache_sync_readahead(mapping, ra, filp, index, last_index - index); - filemap_get_read_batch(mapping, index, last_index, pvec); + filemap_get_read_batch(mapping, index, last_index, fbatch); } - if (!pagevec_count(pvec)) { + if (!folio_batch_count(fbatch)) { if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) return -EAGAIN; err = filemap_create_folio(filp, mapping, - iocb->ki_pos >> PAGE_SHIFT, pvec); + iocb->ki_pos >> PAGE_SHIFT, fbatch); if (err == AOP_TRUNCATED_PAGE) goto retry; return err; } - folio = page_folio(pvec->pages[pagevec_count(pvec) - 1]); + folio = fbatch->folios[folio_batch_count(fbatch) - 1]; if (folio_test_readahead(folio)) { err = filemap_readahead(iocb, filp, mapping, folio, last_index); if (err) goto err; } if (!folio_test_uptodate(folio)) { - if ((iocb->ki_flags & IOCB_WAITQ) && pagevec_count(pvec) > 1) + if ((iocb->ki_flags & IOCB_WAITQ) && + folio_batch_count(fbatch) > 1) iocb->ki_flags |= IOCB_NOWAIT; err = filemap_update_page(iocb, mapping, iter, folio); if (err) @@ -2582,7 +2583,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, err: if (err < 0) folio_put(folio); - if (likely(--pvec->nr)) + if (likely(--fbatch->nr)) return 0; if (err == AOP_TRUNCATED_PAGE) goto retry; @@ -2609,7 +2610,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, struct file_ra_state *ra = &filp->f_ra; struct address_space *mapping = filp->f_mapping; struct inode *inode = mapping->host; - struct pagevec pvec; + struct folio_batch fbatch; int i, error = 0; bool writably_mapped; loff_t isize, end_offset; @@ -2620,7 +2621,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, return 0; iov_iter_truncate(iter, inode->i_sb->s_maxbytes); - pagevec_init(&pvec); + folio_batch_init(&fbatch); do { cond_resched(); @@ -2636,7 +2637,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, if (unlikely(iocb->ki_pos >= i_size_read(inode))) break; - error = filemap_get_pages(iocb, iter, &pvec); + error = filemap_get_pages(iocb, iter, &fbatch); if (error < 0) break; @@ -2650,7 +2651,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, */ isize = i_size_read(inode); if (unlikely(iocb->ki_pos >= isize)) - goto put_pages; + goto put_folios; end_offset = min_t(loff_t, isize, iocb->ki_pos + iter->count); /* @@ -2665,10 +2666,10 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, */ if (iocb->ki_pos >> PAGE_SHIFT != ra->prev_pos >> PAGE_SHIFT) - mark_page_accessed(pvec.pages[0]); + folio_mark_accessed(fbatch.folios[0]); - for (i = 0; i < pagevec_count(&pvec); i++) { - struct folio *folio = page_folio(pvec.pages[i]); + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; size_t fsize = folio_size(folio); size_t offset = iocb->ki_pos & (fsize - 1); size_t bytes = min_t(loff_t, end_offset - iocb->ki_pos, @@ -2698,10 +2699,10 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, break; } } -put_pages: - for (i = 0; i < pagevec_count(&pvec); i++) - put_page(pvec.pages[i]); - pagevec_reinit(&pvec); +put_folios: + for (i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); file_accessed(filp); From patchwork Wed Dec 8 04:22:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8638C433F5 for ; Wed, 8 Dec 2021 06:59:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C00BB6B0073; Wed, 8 Dec 2021 01:58:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B6A556B0078; Wed, 8 Dec 2021 01:58:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B8BE6B0073; Wed, 8 Dec 2021 01:58:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay039.a.hostedemail.com [64.99.140.39]) by kanga.kvack.org (Postfix) with ESMTP id 89B626B0071 for ; Wed, 8 Dec 2021 01:58:35 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id CA1C7801BA for ; Wed, 8 Dec 2021 04:29:10 +0000 (UTC) X-FDA: 78893347260.23.4593C54 Received: from relay.hostedemail.com (relay035.a.hostedemail.com [64.99.140.35]) by imf28.hostedemail.com (Postfix) with ESMTP id 6A7FE900009E for ; Wed, 8 Dec 2021 04:29:10 +0000 (UTC) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4C51221398 for ; Wed, 8 Dec 2021 04:23:23 +0000 (UTC) X-FDA: 78893332686.26.9A8867F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id ABAD63000100 for ; Wed, 8 Dec 2021 04:23:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=2pAlN3Jv5xfx1nt9+sFDKAqPwC3DtkGU2AVdY8nIpv4=; b=YOZ7bCS4ftD9n4wBqnZ0CUI/Ml wuna8g/SEXz34ZZdfvhlcMcDQSQ6HrVvx0kIaofyHkpDzG00tci4avK0Ilx5VLBugixNEZEgb2+NH 53iWfN4t1Nr71HwNF7OIktUSz2zLt+ksuLPb3G8H2PVFvCmYxfdp4Js/6ncEt2mX2nLAScjQvahs4 +SseUp+3O3oogd0JglCNbO3EjZMSBT5OYKrTa9qq9oaIPsz3CZlXDQ4SNUAFBiLqDMFxoJUJdsHcd et0sfg1AQVW0bRVucRoMsH6W14Zxp0LK/7ba74CBTS/x7oTKBFg2C/0+0WhwWUTc/Bw5F4z8mqN4c ZOOEBbOQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU7-0084cj-E6; Wed, 08 Dec 2021 04:23:15 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Jan Kara , William Kucharski Subject: [PATCH 41/48] filemap: Return only folios from find_get_entries() Date: Wed, 8 Dec 2021 04:22:49 +0000 Message-Id: <20211208042256.1923824-42-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: qj8d1zqbnybiioo9max11qro5595nb3o X-HE-Tag-Orig: 1638937402-540324 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YOZ7bCS4; spf=none (imf28.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.35) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 6A7FE900009E X-HE-Tag: 1638937750-198047 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The callers have all been converted to work on folios, so convert find_get_entries() to return a batch of folios instead of pages. We also now return multiple large folios in a single call. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jan Kara Reviewed-by: William Kucharski --- include/linux/pagemap.h | 2 -- mm/filemap.c | 43 +++++++++++------------------------------ mm/internal.h | 4 ++++ mm/shmem.c | 36 +++++++++++++++++++--------------- mm/truncate.c | 43 +++++++++++++++++++++++------------------ 5 files changed, 59 insertions(+), 69 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index eb6e58e106c8..d2259a1da51c 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -592,8 +592,6 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index) return head + (index & (thp_nr_pages(head) - 1)); } -unsigned find_get_entries(struct address_space *mapping, pgoff_t start, - pgoff_t end, struct pagevec *pvec, pgoff_t *indices); unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start, pgoff_t end, unsigned int nr_pages, struct page **pages); diff --git a/mm/filemap.c b/mm/filemap.c index 38726ca96f0e..4f00412d72d3 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2015,57 +2015,36 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max, * @mapping: The address_space to search * @start: The starting page cache index * @end: The final page index (inclusive). - * @pvec: Where the resulting entries are placed. + * @fbatch: Where the resulting entries are placed. * @indices: The cache indices corresponding to the entries in @entries * * find_get_entries() will search for and return a batch of entries in - * the mapping. The entries are placed in @pvec. find_get_entries() - * takes a reference on any actual pages it returns. + * the mapping. The entries are placed in @fbatch. find_get_entries() + * takes a reference on any actual folios it returns. * - * The search returns a group of mapping-contiguous page cache entries - * with ascending indexes. There may be holes in the indices due to - * not-present pages. + * The entries have ascending indexes. The indices may not be consecutive + * due to not-present entries or large folios. * - * Any shadow entries of evicted pages, or swap entries from + * Any shadow entries of evicted folios, or swap entries from * shmem/tmpfs, are included in the returned array. * - * If it finds a Transparent Huge Page, head or tail, find_get_entries() - * stops at that page: the caller is likely to have a better way to handle - * the compound page as a whole, and then skip its extent, than repeatedly - * calling find_get_entries() to return all its tails. - * - * Return: the number of pages and shadow entries which were found. + * Return: The number of entries which were found. */ unsigned find_get_entries(struct address_space *mapping, pgoff_t start, - pgoff_t end, struct pagevec *pvec, pgoff_t *indices) + pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices) { XA_STATE(xas, &mapping->i_pages, start); struct folio *folio; - unsigned int ret = 0; - unsigned nr_entries = PAGEVEC_SIZE; rcu_read_lock(); while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) { - struct page *page = &folio->page; - /* - * Terminate early on finding a THP, to allow the caller to - * handle it all at once; but continue if this is hugetlbfs. - */ - if (!xa_is_value(folio) && folio_test_large(folio) && - !folio_test_hugetlb(folio)) { - page = folio_file_page(folio, xas.xa_index); - nr_entries = ret + 1; - } - - indices[ret] = xas.xa_index; - pvec->pages[ret] = page; - if (++ret == nr_entries) + indices[fbatch->nr] = xas.xa_index; + if (!folio_batch_add(fbatch, folio)) break; } rcu_read_unlock(); - pvec->nr = ret; - return ret; + return folio_batch_count(fbatch); } /** diff --git a/mm/internal.h b/mm/internal.h index d3c7b35934ed..36ad6ffe53bf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -12,6 +12,8 @@ #include #include +struct folio_batch; + /* * The set of flags that only affect watermark checking and reclaim * behaviour. This is used by the MM to obey the caller constraints @@ -113,6 +115,8 @@ static inline void force_page_cache_readahead(struct address_space *mapping, unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct pagevec *pvec, pgoff_t *indices); +unsigned find_get_entries(struct address_space *mapping, pgoff_t start, + pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); void filemap_free_folio(struct address_space *mapping, struct folio *folio); int truncate_inode_folio(struct address_space *mapping, struct folio *folio); diff --git a/mm/shmem.c b/mm/shmem.c index dbef008fb6e5..e909c163fb38 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -920,6 +920,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, unsigned int partial_start = lstart & (PAGE_SIZE - 1); unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1); struct pagevec pvec; + struct folio_batch fbatch; pgoff_t indices[PAGEVEC_SIZE]; long nr_swaps_freed = 0; pgoff_t index; @@ -987,11 +988,12 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, if (start >= end) return; + folio_batch_init(&fbatch); index = start; while (index < end) { cond_resched(); - if (!find_get_entries(mapping, index, end - 1, &pvec, + if (!find_get_entries(mapping, index, end - 1, &fbatch, indices)) { /* If all gone or hole-punch or unfalloc, we're done */ if (index == start || end != -1) @@ -1000,14 +1002,14 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, index = start; continue; } - for (i = 0; i < pagevec_count(&pvec); i++) { - struct page *page = pvec.pages[i]; + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; index = indices[i]; - if (xa_is_value(page)) { + if (xa_is_value(folio)) { if (unfalloc) continue; - if (shmem_free_swap(mapping, index, page)) { + if (shmem_free_swap(mapping, index, folio)) { /* Swap was replaced by page: retry */ index--; break; @@ -1016,33 +1018,35 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, continue; } - lock_page(page); + folio_lock(folio); - if (!unfalloc || !PageUptodate(page)) { - if (page_mapping(page) != mapping) { + if (!unfalloc || !folio_test_uptodate(folio)) { + struct page *page = folio_file_page(folio, + index); + if (folio_mapping(folio) != mapping) { /* Page was replaced by swap: retry */ - unlock_page(page); + folio_unlock(folio); index--; break; } - VM_BUG_ON_PAGE(PageWriteback(page), page); + VM_BUG_ON_FOLIO(folio_test_writeback(folio), + folio); if (shmem_punch_compound(page, start, end)) - truncate_inode_folio(mapping, - page_folio(page)); + truncate_inode_folio(mapping, folio); else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { /* Wipe the page and don't get stuck */ clear_highpage(page); flush_dcache_page(page); - set_page_dirty(page); + folio_mark_dirty(folio); if (index < round_up(start, HPAGE_PMD_NR)) start = index + 1; } } - unlock_page(page); + folio_unlock(folio); } - pagevec_remove_exceptionals(&pvec); - pagevec_release(&pvec); + folio_batch_remove_exceptionals(&fbatch); + folio_batch_release(&fbatch); index++; } diff --git a/mm/truncate.c b/mm/truncate.c index 5370094641d6..357af144df63 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -108,6 +108,13 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping, pvec->nr = j; } +static void truncate_folio_batch_exceptionals(struct address_space *mapping, + struct folio_batch *fbatch, pgoff_t *indices) +{ + truncate_exceptional_pvec_entries(mapping, (struct pagevec *)fbatch, + indices); +} + /* * Invalidate exceptional entry if easily possible. This handles exceptional * entries for invalidate_inode_pages(). @@ -297,6 +304,7 @@ void truncate_inode_pages_range(struct address_space *mapping, unsigned int partial_start; /* inclusive */ unsigned int partial_end; /* exclusive */ struct pagevec pvec; + struct folio_batch fbatch; pgoff_t indices[PAGEVEC_SIZE]; pgoff_t index; int i; @@ -379,10 +387,11 @@ void truncate_inode_pages_range(struct address_space *mapping, if (start >= end) goto out; + folio_batch_init(&fbatch); index = start; for ( ; ; ) { cond_resched(); - if (!find_get_entries(mapping, index, end - 1, &pvec, + if (!find_get_entries(mapping, index, end - 1, &fbatch, indices)) { /* If all gone from start onwards, we're done */ if (index == start) @@ -392,16 +401,14 @@ void truncate_inode_pages_range(struct address_space *mapping, continue; } - for (i = 0; i < pagevec_count(&pvec); i++) { - struct page *page = pvec.pages[i]; - struct folio *folio; + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; /* We rely upon deletion not changing page->index */ index = indices[i]; - if (xa_is_value(page)) + if (xa_is_value(folio)) continue; - folio = page_folio(page); folio_lock(folio); VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio); @@ -410,8 +417,8 @@ void truncate_inode_pages_range(struct address_space *mapping, folio_unlock(folio); index = folio_index(folio) + folio_nr_pages(folio) - 1; } - truncate_exceptional_pvec_entries(mapping, &pvec, indices); - pagevec_release(&pvec); + truncate_folio_batch_exceptionals(mapping, &fbatch, indices); + folio_batch_release(&fbatch); index++; } @@ -625,7 +632,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, pgoff_t start, pgoff_t end) { pgoff_t indices[PAGEVEC_SIZE]; - struct pagevec pvec; + struct folio_batch fbatch; pgoff_t index; int i; int ret = 0; @@ -635,23 +642,21 @@ int invalidate_inode_pages2_range(struct address_space *mapping, if (mapping_empty(mapping)) goto out; - pagevec_init(&pvec); + folio_batch_init(&fbatch); index = start; - while (find_get_entries(mapping, index, end, &pvec, indices)) { - for (i = 0; i < pagevec_count(&pvec); i++) { - struct page *page = pvec.pages[i]; - struct folio *folio; + while (find_get_entries(mapping, index, end, &fbatch, indices)) { + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; /* We rely upon deletion not changing folio->index */ index = indices[i]; - if (xa_is_value(page)) { + if (xa_is_value(folio)) { if (!invalidate_exceptional_entry2(mapping, - index, page)) + index, folio)) ret = -EBUSY; continue; } - folio = page_folio(page); if (!did_range_unmap && folio_mapped(folio)) { /* @@ -684,8 +689,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, ret = ret2; folio_unlock(folio); } - pagevec_remove_exceptionals(&pvec); - pagevec_release(&pvec); + folio_batch_remove_exceptionals(&fbatch); + folio_batch_release(&fbatch); cond_resched(); index++; } From patchwork Wed Dec 8 04:22:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68FDDC433F5 for ; Wed, 8 Dec 2021 06:27:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A5776B0078; Wed, 8 Dec 2021 01:19:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 553A26B0092; Wed, 8 Dec 2021 01:19:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F4866B0093; Wed, 8 Dec 2021 01:19:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay029.a.hostedemail.com [64.99.140.29]) by kanga.kvack.org (Postfix) with ESMTP id 2F2246B0078 for ; Wed, 8 Dec 2021 01:19:52 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 429F720E94; Wed, 8 Dec 2021 06:17:49 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id C3F8220FE6; Wed, 8 Dec 2021 05:33:56 +0000 (UTC) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B952F213D7 for ; Wed, 8 Dec 2021 04:48:17 +0000 (UTC) X-FDA: 78893395434.15.3E536FE Received: from relay.hostedemail.com (relay038.a.hostedemail.com [64.99.140.38]) by imf30.hostedemail.com (Postfix) with ESMTP id 64A19E0016A1 for ; Wed, 8 Dec 2021 04:48:17 +0000 (UTC) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3EEEE20834 for ; Wed, 8 Dec 2021 04:23:21 +0000 (UTC) X-FDA: 78893332602.28.A4BEA57 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 66EE24002087 for ; Wed, 8 Dec 2021 04:23:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=biuqP+A2N13cdbfNyPGAJFwZ1O/isFh5dNQAV3yeGDo=; b=qKFhICvbMl1ap6IAkDrGUz9TD4 N1ddfkarBoa3ssq03SzygQ4fhC9U42i9NYefVgSU04L8BIZw/Sg6jLVKMo3dgS75Vu4V5YwiSjt/5 FibbRS9EJ1gFtkTt1VhIoaOHAZzUYG0Lv3wVUDnq0TFdv0x+uwjpqtUjCn2dXzAuqJB8BycGX4W0t CDYO7vp8F8oZaMnX9+ljjimLdB15N2YzyKP8ntH3ByrkBMfJvIlkLh2q4RaXIniuiQO0Y4CTh7VC/ 1gzlK8QhPGTMRj18aZSh1ve86H+P0z3i46muwZD9kEcxoegvadKvxODhBZaHt2cQw64w8yhMCdIjn QxEQRouQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU7-0084d9-Jx; Wed, 08 Dec 2021 04:23:15 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 42/48] mm: Convert find_lock_entries() to use a folio_batch Date: Wed, 8 Dec 2021 04:22:50 +0000 Message-Id: <20211208042256.1923824-43-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 64A19E0016A1 X-Stat-Signature: 8j1b5nbxtjiwx7rjeajm9wdrs7z33ocx X-HE-Tag-Orig: 1638937400-329472 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qKFhICvb; spf=none (imf30.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.38) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638938897-794004 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: find_lock_entries() already only returned the head page of folios, so convert it to return a folio_batch instead of a pagevec. That cascades through converting truncate_inode_pages_range() to delete_from_page_cache_batch() and page_cache_delete_batch(). Signed-off-by: Matthew Wilcox (Oracle) Reported-by: kernel test robot --- include/linux/pagemap.h | 4 +-- mm/filemap.c | 60 ++++++++++++++++++------------------ mm/internal.h | 2 +- mm/shmem.c | 14 ++++----- mm/truncate.c | 67 ++++++++++++++++++----------------------- 5 files changed, 67 insertions(+), 80 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d2259a1da51c..6e038811f4c8 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -16,7 +16,7 @@ #include /* for in_interrupt() */ #include -struct pagevec; +struct folio_batch; static inline bool mapping_empty(struct address_space *mapping) { @@ -936,7 +936,7 @@ static inline void __delete_from_page_cache(struct page *page, void *shadow) } void replace_page_cache_page(struct page *old, struct page *new); void delete_from_page_cache_batch(struct address_space *mapping, - struct pagevec *pvec); + struct folio_batch *fbatch); int try_to_release_page(struct page *page, gfp_t gfp); bool filemap_release_folio(struct folio *folio, gfp_t gfp); loff_t mapping_seek_hole_data(struct address_space *, loff_t start, loff_t end, diff --git a/mm/filemap.c b/mm/filemap.c index 4f00412d72d3..89a10624e361 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -270,30 +270,29 @@ void filemap_remove_folio(struct folio *folio) } /* - * page_cache_delete_batch - delete several pages from page cache - * @mapping: the mapping to which pages belong - * @pvec: pagevec with pages to delete + * page_cache_delete_batch - delete several folios from page cache + * @mapping: the mapping to which folios belong + * @fbatch: batch of folios to delete * - * The function walks over mapping->i_pages and removes pages passed in @pvec - * from the mapping. The function expects @pvec to be sorted by page index - * and is optimised for it to be dense. - * It tolerates holes in @pvec (mapping entries at those indices are not - * modified). The function expects only THP head pages to be present in the - * @pvec. + * The function walks over mapping->i_pages and removes folios passed in + * @fbatch from the mapping. The function expects @fbatch to be sorted + * by page index and is optimised for it to be dense. + * It tolerates holes in @fbatch (mapping entries at those indices are not + * modified). * * The function expects the i_pages lock to be held. */ static void page_cache_delete_batch(struct address_space *mapping, - struct pagevec *pvec) + struct folio_batch *fbatch) { - XA_STATE(xas, &mapping->i_pages, pvec->pages[0]->index); + XA_STATE(xas, &mapping->i_pages, fbatch->folios[0]->index); int total_pages = 0; int i = 0; struct folio *folio; mapping_set_update(&xas, mapping); xas_for_each(&xas, folio, ULONG_MAX) { - if (i >= pagevec_count(pvec)) + if (i >= folio_batch_count(fbatch)) break; /* A swap/dax/shadow entry got inserted? Skip it. */ @@ -306,9 +305,9 @@ static void page_cache_delete_batch(struct address_space *mapping, * means our page has been removed, which shouldn't be * possible because we're holding the PageLock. */ - if (&folio->page != pvec->pages[i]) { + if (folio != fbatch->folios[i]) { VM_BUG_ON_FOLIO(folio->index > - pvec->pages[i]->index, folio); + fbatch->folios[i]->index, folio); continue; } @@ -316,12 +315,11 @@ static void page_cache_delete_batch(struct address_space *mapping, if (folio->index == xas.xa_index) folio->mapping = NULL; - /* Leave page->index set: truncation lookup relies on it */ + /* Leave folio->index set: truncation lookup relies on it */ /* - * Move to the next page in the vector if this is a regular - * page or the index is of the last sub-page of this compound - * page. + * Move to the next folio in the batch if this is a regular + * folio or the index is of the last sub-page of this folio. */ if (folio->index + folio_nr_pages(folio) - 1 == xas.xa_index) i++; @@ -332,29 +330,29 @@ static void page_cache_delete_batch(struct address_space *mapping, } void delete_from_page_cache_batch(struct address_space *mapping, - struct pagevec *pvec) + struct folio_batch *fbatch) { int i; - if (!pagevec_count(pvec)) + if (!folio_batch_count(fbatch)) return; spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); - for (i = 0; i < pagevec_count(pvec); i++) { - struct folio *folio = page_folio(pvec->pages[i]); + for (i = 0; i < folio_batch_count(fbatch); i++) { + struct folio *folio = fbatch->folios[i]; trace_mm_filemap_delete_from_page_cache(folio); filemap_unaccount_folio(mapping, folio); } - page_cache_delete_batch(mapping, pvec); + page_cache_delete_batch(mapping, fbatch); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock); - for (i = 0; i < pagevec_count(pvec); i++) - filemap_free_folio(mapping, page_folio(pvec->pages[i])); + for (i = 0; i < folio_batch_count(fbatch); i++) + filemap_free_folio(mapping, fbatch->folios[i]); } int filemap_check_errors(struct address_space *mapping) @@ -2052,8 +2050,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start, * @mapping: The address_space to search. * @start: The starting page cache index. * @end: The final page index (inclusive). - * @pvec: Where the resulting entries are placed. - * @indices: The cache indices of the entries in @pvec. + * @fbatch: Where the resulting entries are placed. + * @indices: The cache indices of the entries in @fbatch. * * find_lock_entries() will return a batch of entries from @mapping. * Swap, shadow and DAX entries are included. Folios are returned @@ -2068,7 +2066,7 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start, * Return: The number of entries which were found. */ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, - pgoff_t end, struct pagevec *pvec, pgoff_t *indices) + pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices) { XA_STATE(xas, &mapping->i_pages, start); struct folio *folio; @@ -2088,8 +2086,8 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, VM_BUG_ON_FOLIO(!folio_contains(folio, xas.xa_index), folio); } - indices[pvec->nr] = xas.xa_index; - if (!pagevec_add(pvec, &folio->page)) + indices[fbatch->nr] = xas.xa_index; + if (!folio_batch_add(fbatch, folio)) break; goto next; unlock: @@ -2106,7 +2104,7 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, } rcu_read_unlock(); - return pagevec_count(pvec); + return folio_batch_count(fbatch); } /** diff --git a/mm/internal.h b/mm/internal.h index 36ad6ffe53bf..7759d4ff3323 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -114,7 +114,7 @@ static inline void force_page_cache_readahead(struct address_space *mapping, } unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, - pgoff_t end, struct pagevec *pvec, pgoff_t *indices); + pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); unsigned find_get_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); void filemap_free_folio(struct address_space *mapping, struct folio *folio); diff --git a/mm/shmem.c b/mm/shmem.c index e909c163fb38..bbfa2d05e787 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -919,7 +919,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, pgoff_t end = (lend + 1) >> PAGE_SHIFT; unsigned int partial_start = lstart & (PAGE_SIZE - 1); unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1); - struct pagevec pvec; struct folio_batch fbatch; pgoff_t indices[PAGEVEC_SIZE]; long nr_swaps_freed = 0; @@ -932,12 +931,12 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, if (info->fallocend > start && info->fallocend <= end && !unfalloc) info->fallocend = start; - pagevec_init(&pvec); + folio_batch_init(&fbatch); index = start; while (index < end && find_lock_entries(mapping, index, end - 1, - &pvec, indices)) { - for (i = 0; i < pagevec_count(&pvec); i++) { - struct folio *folio = (struct folio *)pvec.pages[i]; + &fbatch, indices)) { + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; index = indices[i]; @@ -954,8 +953,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, truncate_inode_folio(mapping, folio); folio_unlock(folio); } - pagevec_remove_exceptionals(&pvec); - pagevec_release(&pvec); + folio_batch_remove_exceptionals(&fbatch); + folio_batch_release(&fbatch); cond_resched(); index++; } @@ -988,7 +987,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, if (start >= end) return; - folio_batch_init(&fbatch); index = start; while (index < end) { cond_resched(); diff --git a/mm/truncate.c b/mm/truncate.c index 357af144df63..e7f5762c43d3 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -56,11 +56,11 @@ static void clear_shadow_entry(struct address_space *mapping, pgoff_t index, /* * Unconditionally remove exceptional entries. Usually called from truncate - * path. Note that the pagevec may be altered by this function by removing + * path. Note that the folio_batch may be altered by this function by removing * exceptional entries similar to what pagevec_remove_exceptionals does. */ -static void truncate_exceptional_pvec_entries(struct address_space *mapping, - struct pagevec *pvec, pgoff_t *indices) +static void truncate_folio_batch_exceptionals(struct address_space *mapping, + struct folio_batch *fbatch, pgoff_t *indices) { int i, j; bool dax; @@ -69,11 +69,11 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping, if (shmem_mapping(mapping)) return; - for (j = 0; j < pagevec_count(pvec); j++) - if (xa_is_value(pvec->pages[j])) + for (j = 0; j < folio_batch_count(fbatch); j++) + if (xa_is_value(fbatch->folios[j])) break; - if (j == pagevec_count(pvec)) + if (j == folio_batch_count(fbatch)) return; dax = dax_mapping(mapping); @@ -82,12 +82,12 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping, xa_lock_irq(&mapping->i_pages); } - for (i = j; i < pagevec_count(pvec); i++) { - struct page *page = pvec->pages[i]; + for (i = j; i < folio_batch_count(fbatch); i++) { + struct folio *folio = fbatch->folios[i]; pgoff_t index = indices[i]; - if (!xa_is_value(page)) { - pvec->pages[j++] = page; + if (!xa_is_value(folio)) { + fbatch->folios[j++] = folio; continue; } @@ -96,7 +96,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping, continue; } - __clear_shadow_entry(mapping, index, page); + __clear_shadow_entry(mapping, index, folio); } if (!dax) { @@ -105,14 +105,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping, inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock); } - pvec->nr = j; -} - -static void truncate_folio_batch_exceptionals(struct address_space *mapping, - struct folio_batch *fbatch, pgoff_t *indices) -{ - truncate_exceptional_pvec_entries(mapping, (struct pagevec *)fbatch, - indices); + fbatch->nr = j; } /* @@ -303,7 +296,6 @@ void truncate_inode_pages_range(struct address_space *mapping, pgoff_t end; /* exclusive */ unsigned int partial_start; /* inclusive */ unsigned int partial_end; /* exclusive */ - struct pagevec pvec; struct folio_batch fbatch; pgoff_t indices[PAGEVEC_SIZE]; pgoff_t index; @@ -333,18 +325,18 @@ void truncate_inode_pages_range(struct address_space *mapping, else end = (lend + 1) >> PAGE_SHIFT; - pagevec_init(&pvec); + folio_batch_init(&fbatch); index = start; while (index < end && find_lock_entries(mapping, index, end - 1, - &pvec, indices)) { - index = indices[pagevec_count(&pvec) - 1] + 1; - truncate_exceptional_pvec_entries(mapping, &pvec, indices); - for (i = 0; i < pagevec_count(&pvec); i++) - truncate_cleanup_folio(page_folio(pvec.pages[i])); - delete_from_page_cache_batch(mapping, &pvec); - for (i = 0; i < pagevec_count(&pvec); i++) - unlock_page(pvec.pages[i]); - pagevec_release(&pvec); + &fbatch, indices)) { + index = indices[folio_batch_count(&fbatch) - 1] + 1; + truncate_folio_batch_exceptionals(mapping, &fbatch, indices); + for (i = 0; i < folio_batch_count(&fbatch); i++) + truncate_cleanup_folio(fbatch.folios[i]); + delete_from_page_cache_batch(mapping, &fbatch); + for (i = 0; i < folio_batch_count(&fbatch); i++) + folio_unlock(fbatch.folios[i]); + folio_batch_release(&fbatch); cond_resched(); } @@ -387,7 +379,6 @@ void truncate_inode_pages_range(struct address_space *mapping, if (start >= end) goto out; - folio_batch_init(&fbatch); index = start; for ( ; ; ) { cond_resched(); @@ -489,16 +480,16 @@ static unsigned long __invalidate_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t end, unsigned long *nr_pagevec) { pgoff_t indices[PAGEVEC_SIZE]; - struct pagevec pvec; + struct folio_batch fbatch; pgoff_t index = start; unsigned long ret; unsigned long count = 0; int i; - pagevec_init(&pvec); - while (find_lock_entries(mapping, index, end, &pvec, indices)) { - for (i = 0; i < pagevec_count(&pvec); i++) { - struct page *page = pvec.pages[i]; + folio_batch_init(&fbatch); + while (find_lock_entries(mapping, index, end, &fbatch, indices)) { + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct page *page = &fbatch.folios[i]->page; /* We rely upon deletion not changing page->index */ index = indices[i]; @@ -525,8 +516,8 @@ static unsigned long __invalidate_mapping_pages(struct address_space *mapping, } count += ret; } - pagevec_remove_exceptionals(&pvec); - pagevec_release(&pvec); + folio_batch_remove_exceptionals(&fbatch); + folio_batch_release(&fbatch); cond_resched(); index++; } From patchwork Wed Dec 8 04:22:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 310B4C433F5 for ; Wed, 8 Dec 2021 06:21:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E7C26B007E; Wed, 8 Dec 2021 01:18:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 295B76B0080; Wed, 8 Dec 2021 01:18:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 110316B0081; Wed, 8 Dec 2021 01:18:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id 01D4A6B007E for ; Wed, 8 Dec 2021 01:18:56 -0500 (EST) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 4B0BD20BB0; Wed, 8 Dec 2021 06:17:48 +0000 (UTC) Received: by unirelay09.hostedemail.com (Postfix, from userid 108) id 08C4220DBB; Wed, 8 Dec 2021 05:33:52 +0000 (UTC) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 621E1213A6 for ; Wed, 8 Dec 2021 04:23:38 +0000 (UTC) X-FDA: 78893333316.09.33F8F9B Received: from relay.hostedemail.com (relay026.a.hostedemail.com [64.99.140.26]) by imf31.hostedemail.com (Postfix) with ESMTP id 0C5571046302 for ; Wed, 8 Dec 2021 04:23:37 +0000 (UTC) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0002273B for ; Wed, 8 Dec 2021 04:23:18 +0000 (UTC) X-FDA: 78893332476.10.4B269C3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 7FFB84002087 for ; Wed, 8 Dec 2021 04:23:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=LzRPcOvyDDpS1jsrCMbiNg3YzKKOXZo0QJAK9qeJoI8=; b=BdwPdLA/HMldvSRT7Ya8AkcgLW aG/1slEVWDDVmq+ciFHkcc+WZ0E1qyTn5S43Lkd7avC/VcRd/nEEaN447f96TlJyZZltXmrFC4q9T G1cumgdT4paszQFwEtrriIxqOs2JyITXvPAQ36/fILd4lV4ue2rLGiHexGqYS33xio9wbH4Ids2QF c2bJrOrg9pETbshjNAzd946pHG7PADgnrR31SwP31/8dmMMIB3uYP11sEsgre0zTFTgqsQsAJ+ekm gJgFUPuFL8gV2wOi4bUvZ5blEx1bTfAMW98xtHN2kpCaSy/9/3anEfmc5darWzJ8qloKW36ziopRm tSlrpJqQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU7-0084dN-PX; Wed, 08 Dec 2021 04:23:15 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 43/48] mm: Remove pagevec_remove_exceptionals() Date: Wed, 8 Dec 2021 04:22:51 +0000 Message-Id: <20211208042256.1923824-44-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0C5571046302 X-Stat-Signature: u7fgqptakfp1s9wm7rrw74unny8z6s9r X-HE-Tag-Orig: 1638937398-577967 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="BdwPdLA/"; spf=none (imf31.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.26) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638937417-271446 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All of its callers now call folio_batch_remove_exceptionals(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagevec.h | 6 +----- mm/swap.c | 26 +++++++++++++------------- mm/truncate.c | 2 +- 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h index 4483e6ad7607..7d3494f7fb70 100644 --- a/include/linux/pagevec.h +++ b/include/linux/pagevec.h @@ -26,7 +26,6 @@ struct pagevec { void __pagevec_release(struct pagevec *pvec); void __pagevec_lru_add(struct pagevec *pvec); -void pagevec_remove_exceptionals(struct pagevec *pvec); unsigned pagevec_lookup_range(struct pagevec *pvec, struct address_space *mapping, pgoff_t *start, pgoff_t end); @@ -140,8 +139,5 @@ static inline void folio_batch_release(struct folio_batch *fbatch) pagevec_release((struct pagevec *)fbatch); } -static inline void folio_batch_remove_exceptionals(struct folio_batch *fbatch) -{ - pagevec_remove_exceptionals((struct pagevec *)fbatch); -} +void folio_batch_remove_exceptionals(struct folio_batch *fbatch); #endif /* _LINUX_PAGEVEC_H */ diff --git a/mm/swap.c b/mm/swap.c index e8c9dc6d0377..74f6b311d7ee 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1077,24 +1077,24 @@ void __pagevec_lru_add(struct pagevec *pvec) } /** - * pagevec_remove_exceptionals - pagevec exceptionals pruning - * @pvec: The pagevec to prune + * folio_batch_remove_exceptionals() - Prune non-folios from a batch. + * @fbatch: The batch to prune * - * find_get_entries() fills both pages and XArray value entries (aka - * exceptional entries) into the pagevec. This function prunes all - * exceptionals from @pvec without leaving holes, so that it can be - * passed on to page-only pagevec operations. + * find_get_entries() fills a batch with both folios and shadow/swap/DAX + * entries. This function prunes all the non-folio entries from @fbatch + * without leaving holes, so that it can be passed on to folio-only batch + * operations. */ -void pagevec_remove_exceptionals(struct pagevec *pvec) +void folio_batch_remove_exceptionals(struct folio_batch *fbatch) { - int i, j; + unsigned int i, j; - for (i = 0, j = 0; i < pagevec_count(pvec); i++) { - struct page *page = pvec->pages[i]; - if (!xa_is_value(page)) - pvec->pages[j++] = page; + for (i = 0, j = 0; i < folio_batch_count(fbatch); i++) { + struct folio *folio = fbatch->folios[i]; + if (!xa_is_value(folio)) + fbatch->folios[j++] = folio; } - pvec->nr = j; + fbatch->nr = j; } /** diff --git a/mm/truncate.c b/mm/truncate.c index e7f5762c43d3..a1113b0abb30 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -57,7 +57,7 @@ static void clear_shadow_entry(struct address_space *mapping, pgoff_t index, /* * Unconditionally remove exceptional entries. Usually called from truncate * path. Note that the folio_batch may be altered by this function by removing - * exceptional entries similar to what pagevec_remove_exceptionals does. + * exceptional entries similar to what folio_batch_remove_exceptionals() does. */ static void truncate_folio_batch_exceptionals(struct address_space *mapping, struct folio_batch *fbatch, pgoff_t *indices) From patchwork Wed Dec 8 04:22:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26745C433F5 for ; Wed, 8 Dec 2021 06:58:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C6326B0071; Wed, 8 Dec 2021 01:58:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 968FB6B0075; Wed, 8 Dec 2021 01:58:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E2AE6B0074; Wed, 8 Dec 2021 01:58:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay034.a.hostedemail.com [64.99.140.34]) by kanga.kvack.org (Postfix) with ESMTP id 708AA6B0071 for ; Wed, 8 Dec 2021 01:58:35 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 308C71201C0 for ; Wed, 8 Dec 2021 04:29:10 +0000 (UTC) X-FDA: 78893347260.12.D481061 Received: from forward.hostedemail.com (ofarelay.outbound [64.99.140.10]) by imf06.hostedemail.com (Postfix) with ESMTP id C105B801A88D for ; Wed, 8 Dec 2021 04:29:09 +0000 (UTC) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 76FB7212B2 for ; Wed, 8 Dec 2021 04:29:09 +0000 (UTC) X-FDA: 78893347218.18.7288B79 Received: from relay.hostedemail.com (relay039.a.hostedemail.com [64.99.140.39]) by imf29.hostedemail.com (Postfix) with ESMTP id 1A3C7900013E for ; Wed, 8 Dec 2021 04:29:09 +0000 (UTC) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0EBA5213A4 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) X-FDA: 78893332518.30.A98FB9D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 96127100002 for ; Wed, 8 Dec 2021 04:23:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=p1J1LC0MyxviplPLiO8TdL6iVzAbSDLqoF+pJtmuEdk=; b=g4TCCDla8G3lteFiStk9tiJUl6 NoQj0W/akKWQelVt0BfzSUNK+Gaie60xndgWgnBoWkYX68cn+Sd+XNgk/1Dfk1v4UQk5+smSL4sPX 4DhKnGAbpNO41CKBPDlPls/FwBbP1rzdp/CKDljd/CStm0UKdrjj5VVeMgpm168JlQXhUFVwGTeb5 2sSJPPJDOlaiOqxfatKWeLvi1XMBqM4QBfsE0CH5l92qOeQUKPMj66lt0EBxZqNpSUw4J35gGtkD2 YaSdvZTisO8anlOoghlWKeUd0+N+V+4fDuBOTPllTi1yoD9tBWPa+synZc0g9Wc9zxtcKWDQ0jyAg 4nnMBNTA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU7-0084dd-Vu; Wed, 08 Dec 2021 04:23:16 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 44/48] fs: Convert vfs_dedupe_file_range_compare to folios Date: Wed, 8 Dec 2021 04:22:52 +0000 Message-Id: <20211208042256.1923824-45-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-HE-Tag-Orig: 1638937398-380551 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C105B801A88D X-Stat-Signature: q3ehtikiwn9y3em9sg1tfb8mhmnwpyeu X-HE-Tag-Orig: 1638937749-172074 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=g4TCCDla; spf=none (imf06.hostedemail.com: domain of FILTER%FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.10) smtp.mailfrom=FILTER%FILTER%willy@infradead.org; dmarc=none X-HE-Tag: 1638937749-719784 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We still only operate on a single page of data at a time due to using kmap(). A more complex implementation would work on each page in a folio, but it's not clear that such a complex implementation would be worthwhile. Signed-off-by: Matthew Wilcox (Oracle) --- fs/remap_range.c | 116 ++++++++++++++++++++++------------------------- 1 file changed, 55 insertions(+), 61 deletions(-) diff --git a/fs/remap_range.c b/fs/remap_range.c index 6d4a9beaa097..231159682907 100644 --- a/fs/remap_range.c +++ b/fs/remap_range.c @@ -146,41 +146,41 @@ static int generic_remap_check_len(struct inode *inode_in, } /* Read a page's worth of file data into the page cache. */ -static struct page *vfs_dedupe_get_page(struct inode *inode, loff_t offset) +static struct folio *vfs_dedupe_get_folio(struct inode *inode, loff_t pos) { - struct page *page; + struct folio *folio; - page = read_mapping_page(inode->i_mapping, offset >> PAGE_SHIFT, NULL); - if (IS_ERR(page)) - return page; - if (!PageUptodate(page)) { - put_page(page); + folio = read_mapping_folio(inode->i_mapping, pos >> PAGE_SHIFT, NULL); + if (IS_ERR(folio)) + return folio; + if (!folio_test_uptodate(folio)) { + folio_put(folio); return ERR_PTR(-EIO); } - return page; + return folio; } /* - * Lock two pages, ensuring that we lock in offset order if the pages are from - * the same file. + * Lock two folios, ensuring that we lock in offset order if the folios + * are from the same file. */ -static void vfs_lock_two_pages(struct page *page1, struct page *page2) +static void vfs_lock_two_folios(struct folio *folio1, struct folio *folio2) { /* Always lock in order of increasing index. */ - if (page1->index > page2->index) - swap(page1, page2); + if (folio1->index > folio2->index) + swap(folio1, folio2); - lock_page(page1); - if (page1 != page2) - lock_page(page2); + folio_lock(folio1); + if (folio1 != folio2) + folio_lock(folio2); } -/* Unlock two pages, being careful not to unlock the same page twice. */ -static void vfs_unlock_two_pages(struct page *page1, struct page *page2) +/* Unlock two folios, being careful not to unlock the same folio twice. */ +static void vfs_unlock_two_folios(struct folio *folio1, struct folio *folio2) { - unlock_page(page1); - if (page1 != page2) - unlock_page(page2); + folio_unlock(folio1); + if (folio1 != folio2) + folio_unlock(folio2); } /* @@ -188,77 +188,71 @@ static void vfs_unlock_two_pages(struct page *page1, struct page *page2) * Caller must have locked both inodes to prevent write races. */ static int vfs_dedupe_file_range_compare(struct inode *src, loff_t srcoff, - struct inode *dest, loff_t destoff, + struct inode *dest, loff_t dstoff, loff_t len, bool *is_same) { - loff_t src_poff; - loff_t dest_poff; - void *src_addr; - void *dest_addr; - struct page *src_page; - struct page *dest_page; - loff_t cmp_len; - bool same; - int error; - - error = -EINVAL; - same = true; + bool same = true; + int error = -EINVAL; + while (len) { - src_poff = srcoff & (PAGE_SIZE - 1); - dest_poff = destoff & (PAGE_SIZE - 1); - cmp_len = min(PAGE_SIZE - src_poff, - PAGE_SIZE - dest_poff); + struct folio *src_folio, *dst_folio; + void *src_addr, *dst_addr; + loff_t cmp_len = min(PAGE_SIZE - offset_in_page(srcoff), + PAGE_SIZE - offset_in_page(dstoff)); + cmp_len = min(cmp_len, len); if (cmp_len <= 0) goto out_error; - src_page = vfs_dedupe_get_page(src, srcoff); - if (IS_ERR(src_page)) { - error = PTR_ERR(src_page); + src_folio = vfs_dedupe_get_folio(src, srcoff); + if (IS_ERR(src_folio)) { + error = PTR_ERR(src_folio); goto out_error; } - dest_page = vfs_dedupe_get_page(dest, destoff); - if (IS_ERR(dest_page)) { - error = PTR_ERR(dest_page); - put_page(src_page); + dst_folio = vfs_dedupe_get_folio(dest, dstoff); + if (IS_ERR(dst_folio)) { + error = PTR_ERR(dst_folio); + folio_put(src_folio); goto out_error; } - vfs_lock_two_pages(src_page, dest_page); + vfs_lock_two_folios(src_folio, dst_folio); /* - * Now that we've locked both pages, make sure they're still + * Now that we've locked both folios, make sure they're still * mapped to the file data we're interested in. If not, * someone is invalidating pages on us and we lose. */ - if (!PageUptodate(src_page) || !PageUptodate(dest_page) || - src_page->mapping != src->i_mapping || - dest_page->mapping != dest->i_mapping) { + if (!folio_test_uptodate(src_folio) || !folio_test_uptodate(dst_folio) || + src_folio->mapping != src->i_mapping || + dst_folio->mapping != dest->i_mapping) { same = false; goto unlock; } - src_addr = kmap_atomic(src_page); - dest_addr = kmap_atomic(dest_page); + src_addr = kmap_local_folio(src_folio, + offset_in_folio(src_folio, srcoff)); + dst_addr = kmap_local_folio(dst_folio, + offset_in_folio(dst_folio, dstoff)); - flush_dcache_page(src_page); - flush_dcache_page(dest_page); + flush_dcache_folio(src_folio); + flush_dcache_folio(dst_folio); - if (memcmp(src_addr + src_poff, dest_addr + dest_poff, cmp_len)) + if (memcmp(src_addr, dst_addr, cmp_len)) same = false; - kunmap_atomic(dest_addr); - kunmap_atomic(src_addr); + kunmap_local(dst_addr); + kunmap_local(src_addr); unlock: - vfs_unlock_two_pages(src_page, dest_page); - put_page(dest_page); - put_page(src_page); + vfs_unlock_two_folios(src_folio, dst_folio); + folio_put(dst_folio); + folio_put(src_folio); if (!same) break; srcoff += cmp_len; - destoff += cmp_len; + dstoff += cmp_len; len -= cmp_len; } From patchwork Wed Dec 8 04:22:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15B7DC433EF for ; Wed, 8 Dec 2021 06:18:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C6566B0078; Wed, 8 Dec 2021 01:17:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 974C56B007B; Wed, 8 Dec 2021 01:17:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83D036B007D; Wed, 8 Dec 2021 01:17:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay040.a.hostedemail.com [64.99.140.40]) by kanga.kvack.org (Postfix) with ESMTP id 7490E6B0078 for ; Wed, 8 Dec 2021 01:17:42 -0500 (EST) Received: by unirelay02.hostedemail.com (Postfix, from userid 108) id 52AAD20E56; Wed, 8 Dec 2021 06:16:37 +0000 (UTC) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2BFB820E7A for ; Wed, 8 Dec 2021 04:39:00 +0000 (UTC) X-FDA: 78893372040.07.349138D Received: from relay.hostedemail.com (relay026.a.hostedemail.com [64.99.140.26]) by imf03.hostedemail.com (Postfix) with ESMTP id D3F0D20004 for ; Wed, 8 Dec 2021 04:38:59 +0000 (UTC) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 4951360191 for ; Wed, 8 Dec 2021 04:23:20 +0000 (UTC) X-FDA: 78893332518.15.ABCF3BD Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id D2FD2400208A for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9FuAZvEtMvxjRhXYotUloBtGh5RYj0EuwlnJq5DbYZk=; b=j2TCVe3IdQvXco4UDlUZ2iSgHT 4O297skJBjiAOe+4WDlIKE0KCtnBUq6n+lTXiBEFHMefV0+VNh+v1N5m++HP73uVEaoXQV9+y7r/A Lsoa6HtmItRUBT2DKF5G+0HFXPyiQEGG4mSD1/bfh251eIqsFnR8DoYdzw9ia0BydIEHVIMiqFU+C g5NE88smItA4IfPeVRpJay9O136IWdnTaeTB1oWbKC1h/TmvjwkRA+7aXAPaQfaAlZ5r4NXeNx3Fw RDFemEoyv5yBiC23mORAbKAo+Ch1wccxs3U9ee7TbKo7NZ6b67as5jQdBa2PRZjeTkFgqqlmb4p6a CtlQI7iw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU8-0084dz-3h; Wed, 08 Dec 2021 04:23:16 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 45/48] truncate: Convert invalidate_inode_pages2_range to folios Date: Wed, 8 Dec 2021 04:22:53 +0000 Message-Id: <20211208042256.1923824-46-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D3F0D20004 X-HE-Tag-Orig: 1638937399-487121 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=j2TCVe3I; spf=none (imf03.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.26) smtp.mailfrom=FILTER%willy@infradead.org; dmarc=none X-Rspamd-Server: rspam04 X-Stat-Signature: uodx6t93frzswyupd6bghqe7wg3wugdb X-HE-Tag: 1638938339-561365 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we're going to unmap a folio, we have to be sure to unmap the entire folio, not just the part of it which lies after the search index. Signed-off-by: Matthew Wilcox (Oracle) --- mm/truncate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index a1113b0abb30..2d1dae085acb 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -599,13 +599,13 @@ static int invalidate_complete_folio2(struct address_space *mapping, return 0; } -static int do_launder_page(struct address_space *mapping, struct page *page) +static int do_launder_folio(struct address_space *mapping, struct folio *folio) { - if (!PageDirty(page)) + if (!folio_test_dirty(folio)) return 0; - if (page->mapping != mapping || mapping->a_ops->launder_page == NULL) + if (folio->mapping != mapping || mapping->a_ops->launder_page == NULL) return 0; - return mapping->a_ops->launder_page(page); + return mapping->a_ops->launder_page(&folio->page); } /** @@ -671,7 +671,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, unmap_mapping_folio(folio); BUG_ON(folio_mapped(folio)); - ret2 = do_launder_page(mapping, &folio->page); + ret2 = do_launder_folio(mapping, folio); if (ret2 == 0) { if (!invalidate_complete_folio2(mapping, folio)) ret2 = -EBUSY; From patchwork Wed Dec 8 04:22:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E532DC433EF for ; Wed, 8 Dec 2021 06:33:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43FD16B0080; Wed, 8 Dec 2021 01:23:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C8156B0085; Wed, 8 Dec 2021 01:23:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21B656B0098; Wed, 8 Dec 2021 01:23:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay025.a.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 101046B0080 for ; Wed, 8 Dec 2021 01:23:57 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 51ED760617; Wed, 8 Dec 2021 05:38:44 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id 7A5306077B; Wed, 8 Dec 2021 05:32:07 +0000 (UTC) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7D5F560A1B for ; Wed, 8 Dec 2021 04:23:38 +0000 (UTC) X-FDA: 78893333316.26.732AEA3 Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by imf18.hostedemail.com (Postfix) with ESMTP id 10FDF4002087 for ; Wed, 8 Dec 2021 04:23:37 +0000 (UTC) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8A06F740 for ; Wed, 8 Dec 2021 04:23:20 +0000 (UTC) X-FDA: 78893332560.07.311EE44 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id D128CD0000A6 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Dq/cUR5SbTMPbPSqbVO2t1AYxOZw6BdCsm3TCwKwRHU=; b=ArN31cJb6BZgTNuvg/cFwnm+AO Ii58uFTd5p4H36ILo5WZR/HecCIZWdQ4zlIPuGCQg70/xzeQdtY1V63YSYox0RRz3o7Vp1GombxeY VyOiC9gB9HmL9+b4XqMy1hNnw7w6O8cKk/GKGMoSH8xVKwZmgCHjQ83w0xjBfA4Qq5E+cSMz9NHVy K4vwiJTFKSccjdDuHh/eW0cq/5Y+vR6We756Y4F7pjszb19Go83LwmeiLnOtI28kbPa5xhSxj+RBF SbtetWxJzX3mBpNIhWrd1vGfEYe07poUtljYsNYXcOHys7+bfRve7bZI+g3nLSc+1rzsNrj+9SbvD trzrjToQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU8-0084eL-GD; Wed, 08 Dec 2021 04:23:16 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Jan Kara , William Kucharski Subject: [PATCH 46/48] truncate,shmem: Handle truncates that split large folios Date: Wed, 8 Dec 2021 04:22:54 +0000 Message-Id: <20211208042256.1923824-47-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ArN31cJb; dmarc=none; spf=none (imf18.hostedemail.com: domain of FILTER%willy@infradead.org has no SPF policy when checking 64.99.140.24) smtp.mailfrom=FILTER%willy@infradead.org X-HE-Tag-Orig: 1638937399-247441 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 10FDF4002087 X-Stat-Signature: 3oksaac9iaqg6rxeew1nh966xkzjhgjd X-HE-Tag: 1638937417-234624 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle folio splitting in the parts of the truncation functions which already handle partial pages. Factor all that code out into a new function called truncate_inode_partial_folio(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jan Kara Reviewed-by: William Kucharski --- mm/internal.h | 2 + mm/shmem.c | 107 ++++++++++++++++++--------------------------- mm/truncate.c | 118 ++++++++++++++++++++++++++++++++------------------ 3 files changed, 120 insertions(+), 107 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 7759d4ff3323..e989d8ceec91 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -119,6 +119,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); void filemap_free_folio(struct address_space *mapping, struct folio *folio); int truncate_inode_folio(struct address_space *mapping, struct folio *folio); +bool truncate_inode_partial_folio(struct folio *folio, loff_t start, + loff_t end); /** * folio_evictable - Test whether a folio is evictable. diff --git a/mm/shmem.c b/mm/shmem.c index bbfa2d05e787..7f0b07845c1f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -151,6 +151,19 @@ int shmem_getpage(struct inode *inode, pgoff_t index, mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); } +int shmem_get_folio(struct inode *inode, pgoff_t index, + struct folio **foliop, enum sgp_type sgp) +{ + struct page *page = NULL; + int ret = shmem_getpage(inode, index, &page, sgp); + + if (page) + *foliop = page_folio(page); + else + *foliop = NULL; + return ret; +} + static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb) { return sb->s_fs_info; @@ -880,32 +893,6 @@ void shmem_unlock_mapping(struct address_space *mapping) } } -/* - * Check whether a hole-punch or truncation needs to split a huge page, - * returning true if no split was required, or the split has been successful. - * - * Eviction (or truncation to 0 size) should never need to split a huge page; - * but in rare cases might do so, if shmem_undo_range() failed to trylock on - * head, and then succeeded to trylock on tail. - * - * A split can only succeed when there are no additional references on the - * huge page: so the split below relies upon find_get_entries() having stopped - * when it found a subpage of the huge page, without getting further references. - */ -static bool shmem_punch_compound(struct page *page, pgoff_t start, pgoff_t end) -{ - if (!PageTransCompound(page)) - return true; - - /* Just proceed to delete a huge page wholly within the range punched */ - if (PageHead(page) && - page->index >= start && page->index + HPAGE_PMD_NR <= end) - return true; - - /* Try to split huge page, so we can truly punch the hole or truncate */ - return split_huge_page(page) >= 0; -} - /* * Remove range of pages and swap entries from page cache, and free them. * If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate. @@ -917,13 +904,13 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, struct shmem_inode_info *info = SHMEM_I(inode); pgoff_t start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT; pgoff_t end = (lend + 1) >> PAGE_SHIFT; - unsigned int partial_start = lstart & (PAGE_SIZE - 1); - unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1); struct folio_batch fbatch; pgoff_t indices[PAGEVEC_SIZE]; + struct folio *folio; long nr_swaps_freed = 0; pgoff_t index; int i; + bool partial_end; if (lend == -1) end = -1; /* unsigned, so actually very big */ @@ -959,33 +946,34 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, index++; } - if (partial_start) { - struct page *page = NULL; - shmem_getpage(inode, start - 1, &page, SGP_READ); - if (page) { - unsigned int top = PAGE_SIZE; - if (start > end) { - top = partial_end; - partial_end = 0; - } - zero_user_segment(page, partial_start, top); - set_page_dirty(page); - unlock_page(page); - put_page(page); + partial_end = ((lend + 1) % PAGE_SIZE) > 0; + shmem_get_folio(inode, lstart >> PAGE_SHIFT, &folio, SGP_READ); + if (folio) { + bool same_page; + + same_page = lend < folio_pos(folio) + folio_size(folio); + if (same_page) + partial_end = false; + folio_mark_dirty(folio); + if (!truncate_inode_partial_folio(folio, lstart, lend)) { + start = folio->index + folio_nr_pages(folio); + if (same_page) + end = folio->index; } + folio_unlock(folio); + folio_put(folio); + folio = NULL; } - if (partial_end) { - struct page *page = NULL; - shmem_getpage(inode, end, &page, SGP_READ); - if (page) { - zero_user_segment(page, 0, partial_end); - set_page_dirty(page); - unlock_page(page); - put_page(page); - } + + if (partial_end) + shmem_get_folio(inode, end, &folio, SGP_READ); + if (folio) { + folio_mark_dirty(folio); + if (!truncate_inode_partial_folio(folio, lstart, lend)) + end = folio->index; + folio_unlock(folio); + folio_put(folio); } - if (start >= end) - return; index = start; while (index < end) { @@ -1019,8 +1007,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, folio_lock(folio); if (!unfalloc || !folio_test_uptodate(folio)) { - struct page *page = folio_file_page(folio, - index); if (folio_mapping(folio) != mapping) { /* Page was replaced by swap: retry */ folio_unlock(folio); @@ -1029,18 +1015,9 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, } VM_BUG_ON_FOLIO(folio_test_writeback(folio), folio); - if (shmem_punch_compound(page, start, end)) - truncate_inode_folio(mapping, folio); - else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - /* Wipe the page and don't get stuck */ - clear_highpage(page); - flush_dcache_page(page); - folio_mark_dirty(folio); - if (index < - round_up(start, HPAGE_PMD_NR)) - start = index + 1; - } + truncate_inode_folio(mapping, folio); } + index = folio->index + folio_nr_pages(folio) - 1; folio_unlock(folio); } folio_batch_remove_exceptionals(&fbatch); diff --git a/mm/truncate.c b/mm/truncate.c index 2d1dae085acb..336c8d099efa 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -228,6 +228,58 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio) return 0; } +/* + * Handle partial folios. The folio may be entirely within the + * range if a split has raced with us. If not, we zero the part of the + * folio that's within the [start, end] range, and then split the folio if + * it's large. split_page_range() will discard pages which now lie beyond + * i_size, and we rely on the caller to discard pages which lie within a + * newly created hole. + * + * Returns false if splitting failed so the caller can avoid + * discarding the entire folio which is stubbornly unsplit. + */ +bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) +{ + loff_t pos = folio_pos(folio); + unsigned int offset, length; + + if (pos < start) + offset = start - pos; + else + offset = 0; + length = folio_size(folio); + if (pos + length <= (u64)end) + length = length - offset; + else + length = end + 1 - pos - offset; + + folio_wait_writeback(folio); + if (length == folio_size(folio)) { + truncate_inode_folio(folio->mapping, folio); + return true; + } + + /* + * We may be zeroing pages we're about to discard, but it avoids + * doing a complex calculation here, and then doing the zeroing + * anyway if the page split fails. + */ + folio_zero_range(folio, offset, length); + + cleancache_invalidate_page(folio->mapping, &folio->page); + if (folio_has_private(folio)) + do_invalidatepage(&folio->page, offset, length); + if (!folio_test_large(folio)) + return true; + if (split_huge_page(&folio->page) == 0) + return true; + if (folio_test_dirty(folio)) + return false; + truncate_inode_folio(folio->mapping, folio); + return true; +} + /* * Used to get rid of pages on hardware memory corruption. */ @@ -294,20 +346,16 @@ void truncate_inode_pages_range(struct address_space *mapping, { pgoff_t start; /* inclusive */ pgoff_t end; /* exclusive */ - unsigned int partial_start; /* inclusive */ - unsigned int partial_end; /* exclusive */ struct folio_batch fbatch; pgoff_t indices[PAGEVEC_SIZE]; pgoff_t index; int i; + struct folio * folio; + bool partial_end; if (mapping_empty(mapping)) goto out; - /* Offsets within partial pages */ - partial_start = lstart & (PAGE_SIZE - 1); - partial_end = (lend + 1) & (PAGE_SIZE - 1); - /* * 'start' and 'end' always covers the range of pages to be fully * truncated. Partial pages are covered with 'partial_start' at the @@ -340,47 +388,33 @@ void truncate_inode_pages_range(struct address_space *mapping, cond_resched(); } - if (partial_start) { - struct page *page = find_lock_page(mapping, start - 1); - if (page) { - unsigned int top = PAGE_SIZE; - if (start > end) { - /* Truncation within a single page */ - top = partial_end; - partial_end = 0; - } - wait_on_page_writeback(page); - zero_user_segment(page, partial_start, top); - cleancache_invalidate_page(mapping, page); - if (page_has_private(page)) - do_invalidatepage(page, partial_start, - top - partial_start); - unlock_page(page); - put_page(page); + partial_end = ((lend + 1) % PAGE_SIZE) > 0; + folio = __filemap_get_folio(mapping, lstart >> PAGE_SHIFT, FGP_LOCK, 0); + if (folio) { + bool same_folio = lend < folio_pos(folio) + folio_size(folio); + if (same_folio) + partial_end = false; + if (!truncate_inode_partial_folio(folio, lstart, lend)) { + start = folio->index + folio_nr_pages(folio); + if (same_folio) + end = folio->index; } + folio_unlock(folio); + folio_put(folio); + folio = NULL; } - if (partial_end) { - struct page *page = find_lock_page(mapping, end); - if (page) { - wait_on_page_writeback(page); - zero_user_segment(page, 0, partial_end); - cleancache_invalidate_page(mapping, page); - if (page_has_private(page)) - do_invalidatepage(page, 0, - partial_end); - unlock_page(page); - put_page(page); - } + + if (partial_end) + folio = __filemap_get_folio(mapping, end, FGP_LOCK, 0); + if (folio) { + if (!truncate_inode_partial_folio(folio, lstart, lend)) + end = folio->index; + folio_unlock(folio); + folio_put(folio); } - /* - * If the truncation happened within a single page no pages - * will be released, just zeroed, so we can bail out now. - */ - if (start >= end) - goto out; index = start; - for ( ; ; ) { + while (index < end) { cond_resched(); if (!find_get_entries(mapping, index, end - 1, &fbatch, indices)) { From patchwork Wed Dec 8 04:22:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BD51C433F5 for ; Wed, 8 Dec 2021 06:30:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B5616B0073; Wed, 8 Dec 2021 01:23:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 416FC6B0075; Wed, 8 Dec 2021 01:23:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B84F6B007D; Wed, 8 Dec 2021 01:23:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay032.a.hostedemail.com [64.99.140.32]) by kanga.kvack.org (Postfix) with ESMTP id 1C4E76B0073 for ; Wed, 8 Dec 2021 01:23:45 -0500 (EST) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id A6DA960415; Wed, 8 Dec 2021 05:38:45 +0000 (UTC) Received: by unirelay01.hostedemail.com (Postfix, from userid 108) id C2A0F605C7; Wed, 8 Dec 2021 05:32:09 +0000 (UTC) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 949A460A13 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) X-FDA: 78893332518.03.6D48692 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 282654002087 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=p/v1MPFlGwVbfYsmas4PVpJ8hDUuzrJOYjz6XpsPNfA=; b=j7XgVYa7JGB4uUrwY82lww50++ TWoEOzfjKih0daSh1t7pMCCAUT/KcnbdGe+cnlRMSYn2STHxsGZp8bDjnIjcEZccXHJ33ScGqHw3u MdZaS9mX5iOTWIqBuniEOCq03InAsBNLXHwNBmfuHj/mJyg7sad76LC821iIBepHApiXRW1r/ABxa Q3Xv5A25X4Yw9idApTa2hQ3JFqEhD2p5g3RTV/9nRLXJW9KSVJ1sdiRfkH2f94ceM5NVBYmvlD4Ky j48uBakB1VsJvk6TNSxk1BDBSgIZTDPiSBDm1R9zD/OKnouredPIKGwfHhEQ3+3AV7YtHDYclSeKl lgqgVJLQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU8-0084eW-MM; Wed, 08 Dec 2021 04:23:16 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 47/48] XArray: Add xas_advance() Date: Wed, 8 Dec 2021 04:22:55 +0000 Message-Id: <20211208042256.1923824-48-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=j7XgVYa7; dmarc=none; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 282654002087 X-Stat-Signature: 19sz5sgu64r9wfksgnjmh5yqh451h6db X-HE-Tag: 1638937399-834833 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new helper function to help iterate over multi-index entries. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/xarray.h | 18 ++++++++++++++++++ lib/xarray.c | 6 +++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index a91e3d90df8a..d6d5da6ed735 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1580,6 +1580,24 @@ static inline void xas_set(struct xa_state *xas, unsigned long index) xas->xa_node = XAS_RESTART; } +/** + * xas_advance() - Skip over sibling entries. + * @xas: XArray operation state. + * @index: Index of last sibling entry. + * + * Move the operation state to refer to the last sibling entry. + * This is useful for loops that normally want to see sibling + * entries but sometimes want to skip them. Use xas_set() if you + * want to move to an index which is not part of this entry. + */ +static inline void xas_advance(struct xa_state *xas, unsigned long index) +{ + unsigned char shift = xas_is_node(xas) ? xas->xa_node->shift : 0; + + xas->xa_index = index; + xas->xa_offset = (index >> shift) & XA_CHUNK_MASK; +} + /** * xas_set_order() - Set up XArray operation state for a multislot entry. * @xas: XArray operation state. diff --git a/lib/xarray.c b/lib/xarray.c index f5d8f54907b4..6f47f6375808 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -157,7 +157,7 @@ static void xas_move_index(struct xa_state *xas, unsigned long offset) xas->xa_index += offset << shift; } -static void xas_advance(struct xa_state *xas) +static void xas_next_offset(struct xa_state *xas) { xas->xa_offset++; xas_move_index(xas, xas->xa_offset); @@ -1250,7 +1250,7 @@ void *xas_find(struct xa_state *xas, unsigned long max) xas->xa_offset = ((xas->xa_index - 1) & XA_CHUNK_MASK) + 1; } - xas_advance(xas); + xas_next_offset(xas); while (xas->xa_node && (xas->xa_index <= max)) { if (unlikely(xas->xa_offset == XA_CHUNK_SIZE)) { @@ -1268,7 +1268,7 @@ void *xas_find(struct xa_state *xas, unsigned long max) if (entry && !xa_is_sibling(entry)) return entry; - xas_advance(xas); + xas_next_offset(xas); } if (!xas->xa_node) From patchwork Wed Dec 8 04:22:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 12663747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B525C433F5 for ; Wed, 8 Dec 2021 06:43:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1BC56B0074; Wed, 8 Dec 2021 01:43:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C91C6B007B; Wed, 8 Dec 2021 01:43:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81BA36B0078; Wed, 8 Dec 2021 01:43:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay033.a.hostedemail.com [64.99.140.33]) by kanga.kvack.org (Postfix) with ESMTP id 6E69B6B0073 for ; Wed, 8 Dec 2021 01:43:35 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id D6682120162 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) X-FDA: 78893332518.28.64D52F7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 4583C100004 for ; Wed, 8 Dec 2021 04:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=G/mukvSZncXf2yTMkAGDJr68zKEeRGuzQ/EY/A/EV+Y=; b=Y2KXMUrgOkaFp/ihSr5h+aRaUT GfDxaR4ZCzRqdzeDB8rbpS1wVBUB58pbE3Gh8JKSk0BlG/IsvLcCBXXXJjndG2Gn+ktTsuL0cKqDO sZqG6nLS/sOftfm/iXdpBI669LoOfHs1gbI3+Qhptb2I15fp8hF6Y/4svUttikeDSK4yuH9S7J+mb f8L43FFuDZPAHcNXZANy6+0CTAPDi/bENyHgR8R+7ucl2IPUS3qtPyOkBqy1nu4XjsFL60Vszs1/D rlhqbJ+lk2DrzK64EeT19+RzN+6tshgnPB/a76mQPYiav8Im4uSW5ENW1isDtdTgySs9VNLR2HhcU eLZmv3pg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1muoU8-0084ec-R1; Wed, 08 Dec 2021 04:23:16 +0000 From: "Matthew Wilcox (Oracle)" To: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 48/48] mm: Use multi-index entries in the page cache Date: Wed, 8 Dec 2021 04:22:56 +0000 Message-Id: <20211208042256.1923824-49-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211208042256.1923824-1-willy@infradead.org> References: <20211208042256.1923824-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4583C100004 X-Stat-Signature: 3fqohqiqaeh658hk4o6ucigmsn77mpej Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Y2KXMUrg; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1638937399-977794 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We currently store large folios as 2^N consecutive entries. While this consumes rather more memory than necessary, it also turns out to be buggy. A writeback operation which starts within a tail page of a dirty folio will not write back the folio as the xarray's dirty bit is only set on the head index. With multi-index entries, the dirty bit will be found no matter where in the folio the operation starts. This does end up simplifying the page cache slightly, although not as much as I had hoped. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 10 ------- mm/filemap.c | 61 ++++++++++++++++++++++++++--------------- mm/huge_memory.c | 20 +++++++++++--- mm/khugepaged.c | 12 +++++++- mm/migrate.c | 8 ------ mm/shmem.c | 16 ++++------- 6 files changed, 72 insertions(+), 55 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 6e038811f4c8..704cb1b4b15d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1125,16 +1125,6 @@ static inline unsigned int __readahead_batch(struct readahead_control *rac, VM_BUG_ON_PAGE(PageTail(page), page); array[i++] = page; rac->_batch_count += thp_nr_pages(page); - - /* - * The page cache isn't using multi-index entries yet, - * so the xas cursor needs to be manually moved to the - * next index. This can be removed once the page cache - * is converted. - */ - if (PageHead(page)) - xas_set(&xas, rac->_index + rac->_batch_count); - if (i == array_sz) break; } diff --git a/mm/filemap.c b/mm/filemap.c index 89a10624e361..9b5b2d962c37 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -135,7 +135,6 @@ static void page_cache_delete(struct address_space *mapping, } VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(nr != 1 && shadow, folio); xas_store(&xas, shadow); xas_init_marks(&xas); @@ -286,7 +285,7 @@ static void page_cache_delete_batch(struct address_space *mapping, struct folio_batch *fbatch) { XA_STATE(xas, &mapping->i_pages, fbatch->folios[0]->index); - int total_pages = 0; + long total_pages = 0; int i = 0; struct folio *folio; @@ -313,18 +312,12 @@ static void page_cache_delete_batch(struct address_space *mapping, WARN_ON_ONCE(!folio_test_locked(folio)); - if (folio->index == xas.xa_index) - folio->mapping = NULL; + folio->mapping = NULL; /* Leave folio->index set: truncation lookup relies on it */ - /* - * Move to the next folio in the batch if this is a regular - * folio or the index is of the last sub-page of this folio. - */ - if (folio->index + folio_nr_pages(folio) - 1 == xas.xa_index) - i++; + i++; xas_store(&xas, NULL); - total_pages++; + total_pages += folio_nr_pages(folio); } mapping->nrpages -= total_pages; } @@ -2089,24 +2082,27 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t start, indices[fbatch->nr] = xas.xa_index; if (!folio_batch_add(fbatch, folio)) break; - goto next; + continue; unlock: folio_unlock(folio); put: folio_put(folio); -next: - if (!xa_is_value(folio) && folio_test_large(folio)) { - xas_set(&xas, folio->index + folio_nr_pages(folio)); - /* Did we wrap on 32-bit? */ - if (!xas.xa_index) - break; - } } rcu_read_unlock(); return folio_batch_count(fbatch); } +static inline +bool folio_more_pages(struct folio *folio, pgoff_t index, pgoff_t max) +{ + if (!folio_test_large(folio) || folio_test_hugetlb(folio)) + return false; + if (index >= max) + return false; + return index < folio->index + folio_nr_pages(folio) - 1; +} + /** * find_get_pages_range - gang pagecache lookup * @mapping: The address_space to search @@ -2145,11 +2141,17 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start, if (xa_is_value(folio)) continue; +again: pages[ret] = folio_file_page(folio, xas.xa_index); if (++ret == nr_pages) { *start = xas.xa_index + 1; goto out; } + if (folio_more_pages(folio, xas.xa_index, end)) { + xas.xa_index++; + folio_ref_inc(folio); + goto again; + } } /* @@ -2207,9 +2209,15 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index, if (unlikely(folio != xas_reload(&xas))) goto put_page; - pages[ret] = &folio->page; +again: + pages[ret] = folio_file_page(folio, xas.xa_index); if (++ret == nr_pages) break; + if (folio_more_pages(folio, xas.xa_index, ULONG_MAX)) { + xas.xa_index++; + folio_ref_inc(folio); + goto again; + } continue; put_page: folio_put(folio); @@ -2334,8 +2342,7 @@ static void filemap_get_read_batch(struct address_space *mapping, break; if (folio_test_readahead(folio)) break; - xas.xa_index = folio->index + folio_nr_pages(folio) - 1; - xas.xa_offset = (xas.xa_index >> xas.xa_shift) & XA_CHUNK_MASK; + xas_advance(&xas, folio->index + folio_nr_pages(folio) - 1); continue; put_folio: folio_put(folio); @@ -3284,6 +3291,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { +again: page = folio_file_page(folio, xas.xa_index); if (PageHWPoison(page)) goto unlock; @@ -3305,9 +3313,18 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, do_set_pte(vmf, page, addr); /* no need to invalidate: a not-present page won't be cached */ update_mmu_cache(vma, addr, vmf->pte); + if (folio_more_pages(folio, xas.xa_index, end_pgoff)) { + xas.xa_index++; + folio_ref_inc(folio); + goto again; + } folio_unlock(folio); continue; unlock: + if (folio_more_pages(folio, xas.xa_index, end_pgoff)) { + xas.xa_index++; + goto again; + } folio_unlock(folio); folio_put(folio); } while ((folio = next_map_page(mapping, &xas, end_pgoff)) != NULL); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e5483347291c..e57af3a0af43 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2614,6 +2614,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) { struct page *head = compound_head(page); struct deferred_split *ds_queue = get_deferred_split_queue(head); + XA_STATE(xas, &head->mapping->i_pages, head->index); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int extra_pins, ret; @@ -2678,16 +2679,24 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) unmap_page(head); + if (mapping) { + xas_split_alloc(&xas, head, compound_order(head), + mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out_unlock; + } + } + /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); if (mapping) { - XA_STATE(xas, &mapping->i_pages, page_index(head)); - /* * Check if the head page is present in page cache. * We assume all tail are present too, if head is there. */ - xa_lock(&mapping->i_pages); + xas_lock(&xas); + xas_reset(&xas); if (xas_load(&xas) != head) goto fail; } @@ -2703,6 +2712,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (mapping) { int nr = thp_nr_pages(head); + xas_split(&xas, head, thp_order(head)); if (PageSwapBacked(head)) { __mod_lruvec_page_state(head, NR_SHMEM_THPS, -nr); @@ -2719,7 +2729,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) spin_unlock(&ds_queue->split_queue_lock); fail: if (mapping) - xa_unlock(&mapping->i_pages); + xas_unlock(&xas); local_irq_enable(); remap_page(head, thp_nr_pages(head)); ret = -EBUSY; @@ -2733,6 +2743,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (mapping) i_mmap_unlock_read(mapping); out: + /* Free any memory we didn't use */ + xas_nomem(&xas, 0); count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED); return ret; } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index e99101162f1a..2e1911cc3466 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1667,7 +1667,10 @@ static void collapse_file(struct mm_struct *mm, } count_memcg_page_event(new_page, THP_COLLAPSE_ALLOC); - /* This will be less messy when we use multi-index entries */ + /* + * Ensure we have slots for all the pages in the range. This is + * almost certainly a no-op because most of the pages must be present + */ do { xas_lock_irq(&xas); xas_create_range(&xas); @@ -1892,6 +1895,9 @@ static void collapse_file(struct mm_struct *mm, __mod_lruvec_page_state(new_page, NR_SHMEM, nr_none); } + /* Join all the small entries into a single multi-index entry */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, new_page); xa_locked: xas_unlock_irq(&xas); xa_unlocked: @@ -2013,6 +2019,10 @@ static void khugepaged_scan_file(struct mm_struct *mm, continue; } + /* + * XXX: khugepaged should compact smaller compound pages + * into a PMD sized page + */ if (PageTransCompound(page)) { result = SCAN_PAGE_COMPOUND; break; diff --git a/mm/migrate.c b/mm/migrate.c index 311638177536..7079e6b7dbe7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -433,14 +433,6 @@ int folio_migrate_mapping(struct address_space *mapping, } xas_store(&xas, newfolio); - if (nr > 1) { - int i; - - for (i = 1; i < nr; i++) { - xas_next(&xas); - xas_store(&xas, newfolio); - } - } /* * Drop cache reference from old page by unfreezing diff --git a/mm/shmem.c b/mm/shmem.c index 7f0b07845c1f..4f80cf4c74d3 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -707,7 +707,6 @@ static int shmem_add_to_page_cache(struct page *page, struct mm_struct *charge_mm) { XA_STATE_ORDER(xas, &mapping->i_pages, index, compound_order(page)); - unsigned long i = 0; unsigned long nr = compound_nr(page); int error; @@ -736,18 +735,15 @@ static int shmem_add_to_page_cache(struct page *page, do { void *entry; xas_lock_irq(&xas); - entry = xas_find_conflict(&xas); - if (entry != expected) + while ((entry = xas_find_conflict(&xas)) != NULL) { + if (entry == expected) + continue; xas_set_err(&xas, -EEXIST); - xas_create_range(&xas); - if (xas_error(&xas)) goto unlock; -next: - xas_store(&xas, page); - if (++i < nr) { - xas_next(&xas); - goto next; } + xas_store(&xas, page); + if (xas_error(&xas)) + goto unlock; if (PageTransHuge(page)) { count_vm_event(THP_FILE_ALLOC); __mod_lruvec_page_state(page, NR_SHMEM_THPS, nr);