From patchwork Mon Aug 14 14:41:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13352995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79E15C001DB for ; Mon, 14 Aug 2023 14:41:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A52C46B0075; Mon, 14 Aug 2023 10:41:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A01E46B0078; Mon, 14 Aug 2023 10:41:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F0B28E0001; Mon, 14 Aug 2023 10:41:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7F0A96B0075 for ; Mon, 14 Aug 2023 10:41:12 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1DDEDB21B8 for ; Mon, 14 Aug 2023 14:41:12 +0000 (UTC) X-FDA: 81122972784.07.5931F83 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 74680A000E for ; Mon, 14 Aug 2023 14:41:09 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aooSz6Do; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692024069; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=ZEcKxV+OG7yAAAoqK3p3o4fR1eb6giyew8o7UtGpAeg=; b=iCEcTBbImxHqmu3uA4C1X3U1dbTLbFYm1+s5pAgWrWrZLpLzgJ5pPPid6ZioF/op6Agsdj olOnDJkQZgdeKsPKuTvdg/V9gpJuq7DLcwoNCZkvykIVcog7bn4qNuk+y6buuIplc21eo3 +dNur8ZaQ1iEaVQPDDnx+DrpECzMGoI= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aooSz6Do; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692024069; a=rsa-sha256; cv=none; b=DyDaILodhODBBe4pLS/eG2ThjHPcUi7eRbdiZGzsufP+7oq3RGY9XuAdZwMmilL+0xOHfA gpPrq52PnHuDCjLQ99bxvEQiV3isEQH9LqOLVC2z/VN4loH6k/Pdr6S/BJeBMNDFnF602m QDzkh4JeF7VLmUPRbMvnvCRK9riLcTY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:In-Reply-To:References; bh=ZEcKxV+OG7yAAAoqK3p3o4fR1eb6giyew8o7UtGpAeg=; b=aooSz6Do+KBjGJGkpgWrnn0vIx xDDRlBc70ov4lG2yocyAedKGB+FCy8SYtPtxhCawV/2FUPJZgsps7qzIGZWJQ/fSHubSDMkhg7yvs UYRsc2bhxBWUwlk4x2wh0vb45dfivQYCHCbDFXS3NWfZZVBw9TgB7ZggT0dhYc3nYwvD8toKPe4n0 ZCYWtYl1+MYLOAs/GJBtJxMJ8yqHmwgM7NBxAKiLvQcfMCwNhAF0T2g66PRwTtnAG25WqqExf/a0H P9Ko14yTCJLTeTpnjNWj+QiaiKpTnIR6Uom/K+tDu+JZw8SF6z22VB903u3mewbr8oZnty2q6SHC0 uGNniv7A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVYkg-002VFG-TN; Mon, 14 Aug 2023 14:41:03 +0000 From: "Matthew Wilcox (Oracle)" To: Jens Axboe Cc: "Matthew Wilcox (Oracle)" , Andrew Morton , "Kirill A . Shutemov" , Hugh Dickins , linux-mm@kvack.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Subject: [PATCH] block: Remove special-casing of compound pages Date: Mon, 14 Aug 2023 15:41:00 +0100 Message-Id: <20230814144100.596749-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 74680A000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4x84gddxirn6tmnz51ju5c4gtjq966hp X-HE-Tag: 1692024069-703095 X-HE-Meta: U2FsdGVkX18N4o5zrv9MZaZRT1FU+0DhFltBLneKpGB9v92T5crMzFi9Ie2aJL1hIfYmEVkBvei0shqD0m6OnksCVX6YgRIWkMKPp6y9prmBQMrb3iRSWaSLLhQUXI1+9YrFj7FIAP3JXzIzjIb0Y6MgS774XFIjS3kt436LDu5Bd75KaPtdxW27aoxvJQSpGkxkJg4WNOeDHY2aLnnZ8vAYdmhZ3+kfGOaabaSKR8z+NlAeg+3mrrKkNbHd6KkWoNoy9yV3IT3Pd4FEzc55c/K8hTPtIcL0mHdiGsyTdl2Q7KCq43fzVrR2VUdA5pVW3WIIOkpwZmKjsXxxVahBc+COm6XorSoKlEwGeo7/TWLQcBJW4VbNoPIEDvNlTmMo4X/Mr/XunZmExsF/RFUPTxt5lhNL96Zzsl40irLQv1reNRtV/wIATypO2iH6OP1GTOJWyGv18fsKoA0ebGzSZXuirlCWpa+BGExBEpgnHTOsP3EA1iOi/81IQ4HY/WysEXbbpJCduO/io0sAEErg4GL7kuzQq4G04a+mM2rPysMIqlPCAZ4WEWuiW01hH83WPPROSCDdsVXdY7gLgRqKvXyMk42bFSc3XbuUgqWBPWsZiP/6MXBRFHOcEPc8GwLb617FnGre7iYEVjpwvsmNPwzhpIEcz2Avd7i6brQceBJ8CyR4upVGBsfnRgFMPOG8qwLTELG983GUXvT5QS2LwNX4+l5FNoQFanzxdA29wOYT+mKe8yNQE99koKILiIl25ZyBmEDQ8fbbBxz+HPbmS1KcT3PRhsz/v0R1qAL/KKIqpC58iZDKoycQWnk8H2mdXdwJV+uuQA5P6l97Adba111/0HF/5xGgstFEuny/e3ECs/nud728zs/KNnLMjI3RRODUNHd9BCvJOH7F0uh7W/uHHHP0G0PeOoP36JG1bZT9+W57uBa0MCmjEwNdjGnILEifDyGSibw8Hicu46H 5TyQigxz JDIAJ9AFjk0mCK8d95ZraPp4tRznwMVUVlLsyI+yVjZlXFkB39qiVgecdhGaYjKfbn60BWtyuI0uAPVnAwcPNC7i/hW4r+ebHY2sgO4U+nhnZH7sVbSj5/Px77EWdUBpBQh/KX+w5VEszJEun2djcJOal42+TExRITotFBLA7XMskuhmcDWitTm1O6uQ992o+h6uY3Mdf0W1AJ1rag0LDA81I8N4jlwYOXtonTSKGC2P1c5RsPysaH84jVwl6yi865ZO/AlWOmyX44mT20z4Vajp9j2WYNzdR39V9oxXo1mvMyqY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The special casing was originally added in pre-git history; reproducing the commit log here: > commit a318a92567d77 > Author: Andrew Morton > Date: Sun Sep 21 01:42:22 2003 -0700 > > [PATCH] Speed up direct-io hugetlbpage handling > > This patch short-circuits all the direct-io page dirtying logic for > higher-order pages. Without this, we pointlessly bounce BIOs up to > keventd all the time. In the last twenty years, compound pages have become used for more than just hugetlb. Rewrite these functions to operate on folios instead of pages and remove the special case for hugetlbfs; I don't think it's needed any more (and if it is, we can put it back in as a call to folio_test_hugetlb()). This was found by inspection; as far as I can tell, this bug can lead to pages used as the destination of a direct I/O read not being marked as dirty. If those pages are then reclaimed by the MM without being dirtied for some other reason, they won't be written out. Then when they're faulted back in, they will not contain the data they should. It'll take a pretty unusual setup to produce this problem with several races all going the wrong way. This problem predates the folio work; it could for example have been triggered by mmaping a THP in tmpfs and using that as the target of an O_DIRECT read. Fixes: 800d8c63b2e98 ("shmem: add huge pages support") Cc: stable@vger.kernel.org Signed-off-by: Matthew Wilcox (Oracle) --- block/bio.c | 46 ++++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/block/bio.c b/block/bio.c index 8672179213b9..f46d8ec71fbd 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1171,13 +1171,22 @@ EXPORT_SYMBOL(bio_add_folio); void __bio_release_pages(struct bio *bio, bool mark_dirty) { - struct bvec_iter_all iter_all; - struct bio_vec *bvec; + struct folio_iter fi; + + bio_for_each_folio_all(fi, bio) { + struct page *page; + size_t done = 0; - bio_for_each_segment_all(bvec, bio, iter_all) { - if (mark_dirty && !PageCompound(bvec->bv_page)) - set_page_dirty_lock(bvec->bv_page); - bio_release_page(bio, bvec->bv_page); + if (mark_dirty) { + folio_lock(fi.folio); + folio_mark_dirty(fi.folio); + folio_unlock(fi.folio); + } + page = folio_page(fi.folio, fi.offset / PAGE_SIZE); + do { + bio_release_page(bio, page++); + done += PAGE_SIZE; + } while (done < fi.length); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1455,18 +1464,12 @@ EXPORT_SYMBOL(bio_free_pages); * bio_set_pages_dirty() and bio_check_pages_dirty() are support functions * for performing direct-IO in BIOs. * - * The problem is that we cannot run set_page_dirty() from interrupt context + * The problem is that we cannot run folio_mark_dirty() from interrupt context * because the required locks are not interrupt-safe. So what we can do is to * mark the pages dirty _before_ performing IO. And in interrupt context, * check that the pages are still dirty. If so, fine. If not, redirty them * in process context. * - * We special-case compound pages here: normally this means reads into hugetlb - * pages. The logic in here doesn't really work right for compound pages - * because the VM does not uniformly chase down the head page in all cases. - * But dirtiness of compound pages is pretty meaningless anyway: the VM doesn't - * handle them at all. So we skip compound pages here at an early stage. - * * Note that this code is very hard to test under normal circumstances because * direct-io pins the pages with get_user_pages(). This makes * is_page_cache_freeable return false, and the VM will not clean the pages. @@ -1482,12 +1485,12 @@ EXPORT_SYMBOL(bio_free_pages); */ void bio_set_pages_dirty(struct bio *bio) { - struct bio_vec *bvec; - struct bvec_iter_all iter_all; + struct folio_iter fi; - bio_for_each_segment_all(bvec, bio, iter_all) { - if (!PageCompound(bvec->bv_page)) - set_page_dirty_lock(bvec->bv_page); + bio_for_each_folio_all(fi, bio) { + folio_lock(fi.folio); + folio_mark_dirty(fi.folio); + folio_unlock(fi.folio); } } @@ -1530,12 +1533,11 @@ static void bio_dirty_fn(struct work_struct *work) void bio_check_pages_dirty(struct bio *bio) { - struct bio_vec *bvec; + struct folio_iter fi; unsigned long flags; - struct bvec_iter_all iter_all; - bio_for_each_segment_all(bvec, bio, iter_all) { - if (!PageDirty(bvec->bv_page) && !PageCompound(bvec->bv_page)) + bio_for_each_folio_all(fi, bio) { + if (!folio_test_dirty(fi.folio)) goto defer; }