From patchwork Thu Feb 15 06:36:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13557447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30AE3C48BEF for ; Thu, 15 Feb 2024 06:37:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD02C6B00B9; Thu, 15 Feb 2024 01:37:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D2F546B00BA; Thu, 15 Feb 2024 01:37:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B81216B00BB; Thu, 15 Feb 2024 01:37:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9EA986B00B9 for ; Thu, 15 Feb 2024 01:37:32 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7D0EEC017B for ; Thu, 15 Feb 2024 06:37:32 +0000 (UTC) X-FDA: 81793081944.13.716AF57 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf28.hostedemail.com (Postfix) with ESMTP id D8E37C000C for ; Thu, 15 Feb 2024 06:37:30 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=QjdPwyrd; dmarc=none; spf=none (imf28.hostedemail.com: domain of BATV+a252b21bd4a653ddb6e6+7480+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+a252b21bd4a653ddb6e6+7480+infradead.org+hch@bombadil.srs.infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707979051; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OECBbDLIiwPC3FYe8tkqoQZRVMW91GpPrLleij5hqbk=; b=4dt1DK2xFBgKzjQ2SOoqtasuSOhp9esA2YrpMTO2JzOO/e1bTNLG+kO06buRZKFBrO0DFy BtkXWybV4h3X80W5Jd+Y5sZDZhmSAPLYumjeMZz8ZMACaJ7mt/pVxaNHXMMMkv6yiY6l1f SkdZwdxzYN5kRREc13YOFRQTNR+JNWA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=QjdPwyrd; dmarc=none; spf=none (imf28.hostedemail.com: domain of BATV+a252b21bd4a653ddb6e6+7480+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+a252b21bd4a653ddb6e6+7480+infradead.org+hch@bombadil.srs.infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707979051; a=rsa-sha256; cv=none; b=aXksf47QlKP/H7uC6rylbXJZ4EGv2jeMsPhLBHy/6+QEtWBSxAuAtj4Q3FY/NuRwj/p5AZ Bh55KGh1nLa8gwZYs3ETk2w7Ld9UKie82xPp8W3TKZEB6b/9vDktRsnLCU/31P98HBcvZi hOiTxlUj7r+Qn8HNNviZNgiF2eE1h9g= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=OECBbDLIiwPC3FYe8tkqoQZRVMW91GpPrLleij5hqbk=; b=QjdPwyrdAWQtgMvuz6HvA9OaPL +rJ+zC0xNw3KqgNiPdHKWpsrHoOXbjtlmCwBr7OU/IJ+LBKj7D8ZbteSOhfRdHIZyYpNZ5gZ6POcu 2gsFFj8UgqwQHL6YprBdHZYIvZKHpnFb0SA/DISEiWAY7XU/3S/k+DBBLCBM5OaV8EDFTY8qgZcFi 1ci70GwBs3EQTYtReC0VoeEPJXxMJksS0IR23qonnVZvPnPOydv/NWt/cN0V2T4GNZsUqQ1Y/ei3t H+8lJCxYF4Q6V78mLVu52f3+FjyjDVtOG5//o7xN5BMjEk56XY/2OrNgp/bDZeuoxnAkCQj/BqJfY W9MdGamA==; Received: from 2a02-8389-2341-5b80-39d3-4735-9a3c-88d8.cable.dynamic.v6.surfer.at ([2a02:8389:2341:5b80:39d3:4735:9a3c:88d8] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.97.1 #2 (Red Hat Linux)) id 1raVNB-0000000F72u-2aI7; Thu, 15 Feb 2024 06:37:29 +0000 From: Christoph Hellwig To: linux-mm@kvack.org Cc: Matthew Wilcox , Jan Kara , David Howells , Brian Foster , Christian Brauner , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Jan Kara Subject: [PATCH 13/14] writeback: add a writeback iterator Date: Thu, 15 Feb 2024 07:36:48 +0100 Message-Id: <20240215063649.2164017-14-hch@lst.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240215063649.2164017-1-hch@lst.de> References: <20240215063649.2164017-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspam-User: X-Stat-Signature: rtk4ntenutogitt6signts41if111gz1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D8E37C000C X-HE-Tag: 1707979050-77841 X-HE-Meta: U2FsdGVkX1/GYS09VSkuVKO+1ZDic6++SltO9htX7ahaJFxw0uvSaXC7WDti2kx2WyVufRLsyVhi3PwDOWELF3jEWtPRAF7ZRDeO2F5XTeagBUq3kdz2HQ/G3n6iMNddG9DQANWckSiSjfWuhwJalWRn5TizgkHcLpqnjS3qdi0tbooStj4GNdE3CB0qrDUuihGZAyp6IBDdHH2J7vbgO61rvn8E28Js0gPgTuDmlqwU5nTKfi4D+hJhxU63Cx1N7yzI59PAn4kIG24+lF6KYJSIO3hLgYPztYF6Nz/E1BGtmxeaZTaw2NlwHoalVsC3aaDuTSBynS7CYZTC1jy2Az94rm+ZEHkrT9AUhjmkXcmUP2fe3A5oDUV7JD82AkXABVVL2SjZ/EQ+AxGJl4n5n6hHPuv0yk1Wd3mqIRnAYEs1DHwN/H0fBXc3XlKOmrXeqJOuDaOZs+W38+a/PAPAvBO+zP41uJXuma7T2HHvofx+lHrh+cVqnuWgj5prUoaEnwmOP8d65cbHIVp8XzdpMNF1GhSCd18qOezZlix7CUyHIukoEiZUr9EDY5HFpOf8Ghy6xD8uyKjKzs7mMlKWx/j3ISo8HZtyB8AXtlSsMUCtwyre3SpCrMFRRwbfqzOguhkeLcWkj4lMMJ+c0NZyURACElKQD71zMsa/1e9Hd0Ppm9Q5CrQujMseTzE4xth8dfduQLf67VrifOdbyQZirfcyRcNmxhWY/1N/RgfrbUMT7xGbeh1j+lBpjN6xfE6w3jPm2EcOjXU+lGO5U/L/XHH5P6jm8dOhvV2ABkkW8MV9w9AJZwcZhWs/FlPJ1bNCoQ3sINmIu3kmdgf+MSrsZkig9vFTBQf474f4OeyXaE6yC5XOyGZtt+9+lKsEcrZhMMvdmMRiT22KIPQbnuITmWRKBSpjbTDCVPpleA0yzrqQJuPAkkmQMdigSonGhlpuMzKQ/sKfhcmeyIZBMSs bSlg34PR sKFF4zLw5g83xxJn0d8xVDLhfcpSXE5yPtzPVR7Jx223D64ztH+TOMoKNy48N/2oJ9DJ0qob4EE0rMqsUcQTI5sCrbA4zhe7NQw391iKMdiOecFyWnSzC7zOrDKAlvzZTfoxw1AGL7ym2vXD51JTXD48XyhKPSIOuDC/1Ps5kvYyOphT3uWSDaY15gSNOC8WUZc7ESJmFlKWCbBFaLVJrrRaPb4gI74TUUET2nFUvXjZch4/o2Sa8zsRC6zmTJNGyM6iaIfbUlpjsFQv2CPV4Dg24G586qDrRAa0hHqBKsjxTJufuSKb3wJIvRrChB0OV4L0/GZZN6Hd3Bs3ORXty9Hy7AsKyBRuyEj1wzS9j++WArEk1N7F69Wq4OU7sCoSXShqhRe8iVpKgLhd474a4mWqeIHpzB/xoQob/aBhybxZ+hFhXWZHmdzmt+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor the code left in write_cache_pages into an iterator that the file system can call to get the next folio for a writeback operation: struct folio *folio = NULL; while ((folio = writeback_iter(mapping, wbc, folio, &error))) { error = ; } The twist here is that the error value is passed by reference, so that the iterator can restore it when breaking out of the loop. Handling of the magic AOP_WRITEPAGE_ACTIVATE value stays outside the iterator and needs is just kept in the write_cache_pages legacy wrapper. in preparation for eventually killing it off. Heavily based on a for_each* based iterator from Matthew Wilcox. Signed-off-by: Christoph Hellwig Reviewed-by: Brian Foster Reviewed-by: Jan Kara --- include/linux/writeback.h | 4 + mm/page-writeback.c | 192 ++++++++++++++++++++++---------------- 2 files changed, 118 insertions(+), 78 deletions(-) diff --git a/include/linux/writeback.h b/include/linux/writeback.h index f67b3ea866a0fb..9845cb62e40b2d 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -82,6 +82,7 @@ struct writeback_control { /* internal fields used by the ->writepages implementation: */ struct folio_batch fbatch; pgoff_t index; + int saved_err; #ifdef CONFIG_CGROUP_WRITEBACK struct bdi_writeback *wb; /* wb this writeback is issued under */ @@ -366,6 +367,9 @@ int balance_dirty_pages_ratelimited_flags(struct address_space *mapping, bool wb_over_bg_thresh(struct bdi_writeback *wb); +struct folio *writeback_iter(struct address_space *mapping, + struct writeback_control *wbc, struct folio *folio, int *error); + typedef int (*writepage_t)(struct folio *folio, struct writeback_control *wbc, void *data); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index fc421402f81881..3da4345f08a335 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2325,18 +2325,18 @@ void __init page_writeback_init(void) } /** - * tag_pages_for_writeback - tag pages to be written by write_cache_pages + * tag_pages_for_writeback - tag pages to be written by writeback * @mapping: address space structure to write * @start: starting page index * @end: ending page index (inclusive) * * This function scans the page range from @start to @end (inclusive) and tags - * all pages that have DIRTY tag set with a special TOWRITE tag. The idea is - * that write_cache_pages (or whoever calls this function) will then use - * TOWRITE tag to identify pages eligible for writeback. This mechanism is - * used to avoid livelocking of writeback by a process steadily creating new - * dirty pages in the file (thus it is important for this function to be quick - * so that it can tag pages faster than a dirtying process can create them). + * all pages that have DIRTY tag set with a special TOWRITE tag. The caller + * can then use the TOWRITE tag to identify pages eligible for writeback. + * This mechanism is used to avoid livelocking of writeback by a process + * steadily creating new dirty pages in the file (thus it is important for this + * function to be quick so that it can tag pages faster than a dirtying process + * can create them). */ void tag_pages_for_writeback(struct address_space *mapping, pgoff_t start, pgoff_t end) @@ -2434,69 +2434,68 @@ static struct folio *writeback_get_folio(struct address_space *mapping, } /** - * write_cache_pages - walk the list of dirty pages of the given address space and write all of them. + * writeback_iter - iterate folio of a mapping for writeback * @mapping: address space structure to write - * @wbc: subtract the number of written pages from *@wbc->nr_to_write - * @writepage: function called for each page - * @data: data passed to writepage function + * @wbc: writeback context + * @folio: previously iterated folio (%NULL to start) + * @error: in-out pointer for writeback errors (see below) * - * If a page is already under I/O, write_cache_pages() skips it, even - * if it's dirty. This is desirable behaviour for memory-cleaning writeback, - * but it is INCORRECT for data-integrity system calls such as fsync(). fsync() - * and msync() need to guarantee that all the data which was dirty at the time - * the call was made get new I/O started against them. If wbc->sync_mode is - * WB_SYNC_ALL then we were called for data integrity and we must wait for - * existing IO to complete. - * - * To avoid livelocks (when other process dirties new pages), we first tag - * pages which should be written back with TOWRITE tag and only then start - * writing them. For data-integrity sync we have to be careful so that we do - * not miss some pages (e.g., because some other process has cleared TOWRITE - * tag we set). The rule we follow is that TOWRITE tag can be cleared only - * by the process clearing the DIRTY tag (and submitting the page for IO). - * - * To avoid deadlocks between range_cyclic writeback and callers that hold - * pages in PageWriteback to aggregate IO until write_cache_pages() returns, - * we do not loop back to the start of the file. Doing so causes a page - * lock/page writeback access order inversion - we should only ever lock - * multiple pages in ascending page->index order, and looping back to the start - * of the file violates that rule and causes deadlocks. + * This function returns the next folio for the writeback operation described by + * @wbc on @mapping and should be called in a while loop in the ->writepages + * implementation. * - * Return: %0 on success, negative error code otherwise + * To start the writeback operation, %NULL is passed in the @folio argument, and + * for every subsequent iteration the folio returned previously should be passed + * back in. + * + * If there was an error in the per-folio writeback inside the writeback_iter() + * loop, @error should be set to the error value. + * + * Once the writeback described in @wbc has finished, this function will return + * %NULL and if there was an error in any iteration restore it to @error. + * + * Note: callers should not manually break out of the loop using break or goto + * but must keep calling writeback_iter() until it returns %NULL. + * + * Return: the folio to write or %NULL if the loop is done. */ -int write_cache_pages(struct address_space *mapping, - struct writeback_control *wbc, writepage_t writepage, - void *data) +struct folio *writeback_iter(struct address_space *mapping, + struct writeback_control *wbc, struct folio *folio, int *error) { - int ret = 0; - int error; - struct folio *folio; - pgoff_t end; /* Inclusive */ - - if (wbc->range_cyclic) { - wbc->index = mapping->writeback_index; /* prev offset */ - end = -1; - } else { - wbc->index = wbc->range_start >> PAGE_SHIFT; - end = wbc->range_end >> PAGE_SHIFT; - } - if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages) - tag_pages_for_writeback(mapping, wbc->index, end); - - folio_batch_init(&wbc->fbatch); + if (!folio) { + folio_batch_init(&wbc->fbatch); + wbc->saved_err = *error = 0; - for (;;) { - folio = writeback_get_folio(mapping, wbc); - if (!folio) - break; + /* + * For range cyclic writeback we remember where we stopped so + * that we can continue where we stopped. + * + * For non-cyclic writeback we always start at the beginning of + * the passed in range. + */ + if (wbc->range_cyclic) + wbc->index = mapping->writeback_index; + else + wbc->index = wbc->range_start >> PAGE_SHIFT; - error = writepage(folio, wbc, data); + /* + * To avoid livelocks when other processes dirty new pages, we + * first tag pages which should be written back and only then + * start writing them. + * + * For data-integrity writeback we have to be careful so that we + * do not miss some pages (e.g., because some other process has + * cleared the TOWRITE tag we set). The rule we follow is that + * TOWRITE tag can be cleared only by the process clearing the + * DIRTY tag (and submitting the page for I/O). + */ + if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages) + tag_pages_for_writeback(mapping, wbc->index, + wbc_end(wbc)); + } else { wbc->nr_to_write -= folio_nr_pages(folio); - if (error == AOP_WRITEPAGE_ACTIVATE) { - folio_unlock(folio); - error = 0; - } + WARN_ON_ONCE(*error > 0); /* * For integrity writeback we have to keep going until we have @@ -2510,33 +2509,70 @@ int write_cache_pages(struct address_space *mapping, * wbc->nr_to_write or encounter the first error. */ if (wbc->sync_mode == WB_SYNC_ALL) { - if (error && !ret) - ret = error; + if (*error && !wbc->saved_err) + wbc->saved_err = *error; } else { - if (error || wbc->nr_to_write <= 0) + if (*error || wbc->nr_to_write <= 0) goto done; } } - /* - * For range cyclic writeback we need to remember where we stopped so - * that we can continue there next time we are called. If we hit the - * last page and there is more work to be done, wrap back to the start - * of the file. - * - * For non-cyclic writeback we always start looking up at the beginning - * of the file if we are called again, which can only happen due to - * -ENOMEM from the file system. - */ - folio_batch_release(&wbc->fbatch); - if (wbc->range_cyclic) - mapping->writeback_index = 0; - return ret; + folio = writeback_get_folio(mapping, wbc); + if (!folio) { + /* + * To avoid deadlocks between range_cyclic writeback and callers + * that hold pages in PageWriteback to aggregate I/O until + * the writeback iteration finishes, we do not loop back to the + * start of the file. Doing so causes a page lock/page + * writeback access order inversion - we should only ever lock + * multiple pages in ascending page->index order, and looping + * back to the start of the file violates that rule and causes + * deadlocks. + */ + if (wbc->range_cyclic) + mapping->writeback_index = 0; + + /* + * Return the first error we encountered (if there was any) to + * the caller. + */ + *error = wbc->saved_err; + } + return folio; done: if (wbc->range_cyclic) mapping->writeback_index = folio->index + folio_nr_pages(folio); folio_batch_release(&wbc->fbatch); + return NULL; +} + +/** + * write_cache_pages - walk the list of dirty pages of the given address space and write all of them. + * @mapping: address space structure to write + * @wbc: subtract the number of written pages from *@wbc->nr_to_write + * @writepage: function called for each page + * @data: data passed to writepage function + * + * Return: %0 on success, negative error code otherwise + * + * Note: please use writeback_iter() instead. + */ +int write_cache_pages(struct address_space *mapping, + struct writeback_control *wbc, writepage_t writepage, + void *data) +{ + struct folio *folio = NULL; + int error; + + while ((folio = writeback_iter(mapping, wbc, folio, &error))) { + error = writepage(folio, wbc, data); + if (error == AOP_WRITEPAGE_ACTIVATE) { + folio_unlock(folio); + error = 0; + } + } + return error; } EXPORT_SYMBOL(write_cache_pages);