From patchwork Mon Nov 11 23:37:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13871496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FDB8D3ABF4 for ; Mon, 11 Nov 2024 23:49:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5881B6B0092; Mon, 11 Nov 2024 18:49:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 537C16B00E3; Mon, 11 Nov 2024 18:49:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B15E6B00E4; Mon, 11 Nov 2024 18:49:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1772B6B0092 for ; Mon, 11 Nov 2024 18:49:10 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CCBF612131C for ; Mon, 11 Nov 2024 23:49:09 +0000 (UTC) X-FDA: 82775455896.27.87586AC Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 89B9C120021 for ; Mon, 11 Nov 2024 23:48:12 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=YLdE+fI9; spf=pass (imf29.hostedemail.com: domain of axboe@kernel.dk designates 209.85.210.172 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731368894; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TkS4Adcluut9E9TBSf0Q5Dr6BdVnPYdslAPkTGEhwiY=; b=Y/q1sfEXzw7nPX1Cd7J2d64FN8cN00KBZAKab8n4goatVfownnO41qudc9ymS30hsaiNue 5a2pMMCd8zxiKPSfh1fG3gmeTRAHZAX6FBGzkeum7QgjoiSr2zUN9/eMfaZPqSzSPSSea8 aFiuGGhQhK7fefhfCSVOb0C5xt4dS60= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731368894; a=rsa-sha256; cv=none; b=6lnKtkYtZWTpflDzZsdSeh7AVAUB9jHQXOQLb3FVoV9X4uTbcdL1aup6YpVwMjUusPNbTI RgsKfS2HHwe+mQ3yGpHROZ4ptTxBGU2ZDSXtVvEUr1Qy2cvMAQDKv4e9OE3jLURXyDPf6+ /2AbPjzD/aAfFaoHSNqxgOtTHuDvFDc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=YLdE+fI9; spf=pass (imf29.hostedemail.com: domain of axboe@kernel.dk designates 209.85.210.172 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-71e681bc315so3584369b3a.0 for ; Mon, 11 Nov 2024 15:49:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731368946; x=1731973746; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TkS4Adcluut9E9TBSf0Q5Dr6BdVnPYdslAPkTGEhwiY=; b=YLdE+fI9R9wQWL+kmGTjJV3CN1+eVnF+gOugIn/dpPsA6jJdHdfa6J8lCDEjIcI4lS v8UuYZwuT8cN2bvYqDwk1dZ8lY2WBwu7XSYzUfwq4YGtNffhPkGSv6BuobfMbDaiBmSN Pok/XXBKz0+1t/2ZPK2sjlaf/cEEtTgwRwt8wZA6XsErf0F/Oc5c2D1iUrL9WaW9pRYD dUdUU+GJDSGFFkF2NQHYMWFUoaxgCqNnN4bwF2aIQphyBmtE35b/wsDrNGIfBXACxlZG ULbKhFeaEa0OchVNPXqGx62KjLgu4F6qVaILaWJHaXMgSAvLto6ET7CvdkH0jcLEcqmV vcjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731368946; x=1731973746; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TkS4Adcluut9E9TBSf0Q5Dr6BdVnPYdslAPkTGEhwiY=; b=UXNSNJgPy383sOupaYjz0XfB4YM6LsCyCbKK3YuIAo5IUzMd6cDquYwLt9E+6kuHLV vH8H/2/4Y5pZ860Q+Iyw4dQ4tUN4eo9GxYjcG9hN/pBbANgZvgElA78gGk8QvcPjaU1e XcdFkNeITgaMVgIs0Q17llJBjcrfs6UmrEMLcykYBcc9Fhbyr7j0+ud1yFI4m1qUMYz1 uefOV5ASrInzt3di7TTGtKG5PlwjO8Q6d7Io9vLGYxPcic93IVxTDiAP4+/4QsZFYWn8 A91PTeCpduyKloKOhhmecJ1zLYBxpTulL/BF6oZ6TRc/U1qKQHwqtAiAjeTTBqa3Tvrw YlfQ== X-Gm-Message-State: AOJu0YyNQYeXyghoJ8D5WehEFlVQKXQmHaKjwhJs7TR70DrPhdZhkyxd jr1HRGsEPsRJ2d/0wOa1xioTtky98HjCQJROYWTWDh3cXJIwxSW2uRurNzUUk8SlBXOiGdjVJk3 Lbtk= X-Google-Smtp-Source: AGHT+IH28qnmC+d/sJZ3EdxK+NK8za6V+wXT31dNckiLvxFVwcmETmxIGu/BBIMvr9bfEpwgK4E2eA== X-Received: by 2002:a05:6a00:a1a:b0:71e:71ba:9056 with SMTP id d2e1a72fcca58-7241407b632mr20781928b3a.10.1731368946274; Mon, 11 Nov 2024 15:49:06 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-724078a7ee9sm10046057b3a.64.2024.11.11.15.49.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2024 15:49:05 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, Jens Axboe Subject: [PATCH 12/16] ext4: add RWF_UNCACHED write support Date: Mon, 11 Nov 2024 16:37:39 -0700 Message-ID: <20241111234842.2024180-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241111234842.2024180-1-axboe@kernel.dk> References: <20241111234842.2024180-1-axboe@kernel.dk> MIME-Version: 1.0 X-Stat-Signature: weqb3a7hr91n8jzg51mbq13b47uf33hm X-Rspam-User: X-Rspamd-Queue-Id: 89B9C120021 X-Rspamd-Server: rspam02 X-HE-Tag: 1731368892-996565 X-HE-Meta: U2FsdGVkX18BneiZenvwuiOGUkMzEJ8IrXqbYZKEDe+xLmArUmY3pkFtuDWt9crF60Sv9uOYXSIYheKOtD9iHZIDKPWWpFeM+XUuDCmWblmlzCxiSzBiWraZQwWAW5dvBXK9/pxxjeoRujEm/TIeb0to3H3HuR8CvJvu3Gv7SkjHYKTXnNstcPu75LQaVPYZLdbfRgq9SLwB/oTduKa0WyDZuA25GkbSmgtB6CEgq0zVZPuCEgsu49spekrVhoOcnhifBB2xbySFJOe5rH6AoAegRQ2sK4cRUSumDYn1tJTo/6CtsCyTeyleo1qk02ZAXS70gyhRTUIE2v62T4Zxr5DsTMxJhQekCH1tb1eSiwas1sI0yPwllP30EJYFm1Mk0iZaFI/E64OPzYtExsrNdAdTzP8b2EQkM58eu8cwqslsTeYeTi+vq3srOp4qDbVIcXMuZnJJ2unt9cOz+QVf0HMU6pdSRjhgLkG4TJhEmIAS74l6cG0cqqa3QL5kJ3+lGZ12wUV7cSc8+78ntVvo7OGank0tFUOt8whimyTO3+AeBr9ztICyFoVVxFLIv20WO9IgdTRQUMqg/rGvNT7/FlZg6RGp/sLlQgl2vpBtd8CtaJ84YoZ5BYAXjcaLLfMr4gJMvlmN6jgqelnG4iQwUj00ZTeeITVs8K0JVbFR6sQDKKw7hsTOOnAZuSPqZIrdr8KOkqJuV6Fq3iTUrPi7DbRO8VCxtHopbnSAiPjrWSpRvrDMvU0LofyDyVArd/W3thaXUpEVX9tqZtwrMtEUH0KaoUVilOJ1odVNuYkBLkv9PvbQ5X9qoF5wfvbUAgUc1WS6r8c0IVhnib4pWSiWEZM5Xd+spedAuCPTpP1DD9hm6iYYR4kavabqycWN7OmNyFP3v4AYZlMf4Q9MR/4TU9aIuwM9EYkTUII/PVyevH2WG+Gn2D46SjkdfygPAKdCwbUocjcyaDz3Pik4J45 u4Xsg7NT 9xvdgD8tlmCIfBaU+tF+tuzmessAd4Oj55O+xGojaOPmYBd00aDM81c8gkUX8U+f6igebOsyhaPoMulloFDbfGK5los3DwTmzB2bc4u/Ek3HI/TC9vDieoUp5aUfI4rVWcM+ptnPl0myCVbnjX+TfDArYt03DJ0G20nXqKQe8tQBrlFRloaBnblMMp1CvUi5LuAXK6jw7e6lJusEBiDlOVfiYCscRvDMYdJ/xMj5mWraJCURCf0nhyn553F1xMrZqWuBZsqVTvmZ4CkNE+HnXi8q+W9qmUD1xRT1Qm5iGOn/PMy/LaFWJtlGAaJKUq+0Z9WCHWk/m0LYbFDHDwFuDJKj1jAX9/LAj9tDs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that. If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache. Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it. Signed-off-by: Jens Axboe --- fs/ext4/ext4.h | 1 + fs/ext4/file.c | 2 +- fs/ext4/inline.c | 7 ++++++- fs/ext4/inode.c | 18 ++++++++++++++++-- fs/ext4/page-io.c | 28 ++++++++++++++++------------ 5 files changed, 40 insertions(+), 16 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 44b0d418143c..60dc9ffae076 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -279,6 +279,7 @@ struct ext4_system_blocks { * Flags for ext4_io_end->flags */ #define EXT4_IO_END_UNWRITTEN 0x0001 +#define EXT4_IO_UNCACHED 0x0002 struct ext4_io_end_vec { struct list_head list; /* list of io_end_vec */ diff --git a/fs/ext4/file.c b/fs/ext4/file.c index f14aed14b9cf..0ef39d738598 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -944,7 +944,7 @@ const struct file_operations ext4_file_operations = { .splice_write = iter_file_splice_write, .fallocate = ext4_fallocate, .fop_flags = FOP_MMAP_SYNC | FOP_BUFFER_RASYNC | - FOP_DIO_PARALLEL_WRITE, + FOP_DIO_PARALLEL_WRITE | FOP_UNCACHED, }; const struct inode_operations ext4_file_inode_operations = { diff --git a/fs/ext4/inline.c b/fs/ext4/inline.c index 3536ca7e4fcc..4089d0744164 100644 --- a/fs/ext4/inline.c +++ b/fs/ext4/inline.c @@ -667,6 +667,7 @@ int ext4_try_to_write_inline_data(struct address_space *mapping, handle_t *handle; struct folio *folio; struct ext4_iloc iloc; + fgf_t fgp_flags; if (pos + len > ext4_get_max_inline_size(inode)) goto convert; @@ -702,7 +703,11 @@ int ext4_try_to_write_inline_data(struct address_space *mapping, if (ret) goto out; - folio = __filemap_get_folio(mapping, 0, FGP_WRITEBEGIN | FGP_NOFS, + fgp_flags = FGP_WRITEBEGIN | FGP_NOFS; + if (*foliop == foliop_uncached) + fgp_flags |= FGP_UNCACHED; + + folio = __filemap_get_folio(mapping, 0, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) { ret = PTR_ERR(folio); diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 54bdd4884fe6..afae3ab64c9e 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, int ret, needed_blocks; handle_t *handle; int retries = 0; + fgf_t fgp_flags; struct folio *folio; pgoff_t index; unsigned from, to; @@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, return 0; } + /* + * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop contains + * foliop_uncached. That's how generic_perform_write() informs us + * that this is an uncached write. + */ + fgp_flags = FGP_WRITEBEGIN; + if (*foliop == foliop_uncached) + fgp_flags |= FGP_UNCACHED; + /* * __filemap_get_folio() can take a long time if the * system is thrashing due to memory pressure, or if the folio @@ -1172,7 +1182,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, * the folio (if needed) without using GFP_NOFS. */ retry_grab: - folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, + folio = __filemap_get_folio(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) return PTR_ERR(folio); @@ -2903,6 +2913,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping, struct folio *folio; pgoff_t index; struct inode *inode = mapping->host; + fgf_t fgp_flags; if (unlikely(ext4_forced_shutdown(inode->i_sb))) return -EIO; @@ -2926,8 +2937,11 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping, return 0; } + fgp_flags = FGP_WRITEBEGIN; + if (*foliop == foliop_uncached) + fgp_flags |= FGP_UNCACHED; retry: - folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, + folio = __filemap_get_folio(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) return PTR_ERR(folio); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index ad5543866d21..10447c3c4ff1 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -226,8 +226,6 @@ static void ext4_add_complete_io(ext4_io_end_t *io_end) unsigned long flags; /* Only reserved conversions from writeback should enter here */ - WARN_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN)); - WARN_ON(!io_end->handle && sbi->s_journal); spin_lock_irqsave(&ei->i_completed_io_lock, flags); wq = sbi->rsv_conversion_wq; if (list_empty(&ei->i_rsv_conversion_list)) @@ -252,7 +250,7 @@ static int ext4_do_flush_completed_IO(struct inode *inode, while (!list_empty(&unwritten)) { io_end = list_entry(unwritten.next, ext4_io_end_t, list); - BUG_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN)); + BUG_ON(!(io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED))); list_del_init(&io_end->list); err = ext4_end_io_end(io_end); @@ -287,14 +285,15 @@ ext4_io_end_t *ext4_init_io_end(struct inode *inode, gfp_t flags) void ext4_put_io_end_defer(ext4_io_end_t *io_end) { - if (refcount_dec_and_test(&io_end->count)) { - if (!(io_end->flag & EXT4_IO_END_UNWRITTEN) || - list_empty(&io_end->list_vec)) { - ext4_release_io_end(io_end); - return; - } - ext4_add_complete_io(io_end); + if (!refcount_dec_and_test(&io_end->count)) + return; + if ((!(io_end->flag & EXT4_IO_END_UNWRITTEN) || + list_empty(&io_end->list_vec)) && + !(io_end->flag & EXT4_IO_UNCACHED)) { + ext4_release_io_end(io_end); + return; } + ext4_add_complete_io(io_end); } int ext4_put_io_end(ext4_io_end_t *io_end) @@ -348,7 +347,7 @@ static void ext4_end_bio(struct bio *bio) blk_status_to_errno(bio->bi_status)); } - if (io_end->flag & EXT4_IO_END_UNWRITTEN) { + if (io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED)) { /* * Link bio into list hanging from io_end. We have to do it * atomically as bio completions can be racing against each @@ -417,8 +416,13 @@ static void io_submit_add_bh(struct ext4_io_submit *io, submit_and_retry: ext4_io_submit(io); } - if (io->io_bio == NULL) + if (io->io_bio == NULL) { io_submit_init_bio(io, bh); + if (folio_test_uncached(folio)) { + ext4_io_end_t *io_end = io->io_bio->bi_private; + io_end->flag |= EXT4_IO_UNCACHED; + } + } if (!bio_add_folio(io->io_bio, io_folio, bh->b_size, bh_offset(bh))) goto submit_and_retry; wbc_account_cgroup_owner(io->io_wbc, &folio->page, bh->b_size);