From patchwork Tue Feb 14 08:37:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13139675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38F5BC6379F for ; Tue, 14 Feb 2023 08:37:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1DA46B0078; Tue, 14 Feb 2023 03:37:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ACCEA280003; Tue, 14 Feb 2023 03:37:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96D49280002; Tue, 14 Feb 2023 03:37:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 822076B0078 for ; Tue, 14 Feb 2023 03:37:35 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 60BA2140DB3 for ; Tue, 14 Feb 2023 08:37:35 +0000 (UTC) X-FDA: 80465243670.23.B96C095 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 8949B120011 for ; Tue, 14 Feb 2023 08:37:33 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UU4u+f6D; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676363853; a=rsa-sha256; cv=none; b=k5UHwNLYPcIA7hu1/eFSNvpGsq2xmqjdFTv4YuMsvUzA9mdkparLr3P4UVN4Ml7ngwG7x4 nUDzGu+ruVMEIAjpYjSeA1CddI8bJiz1SkfZwKZO5yjH3h6Wdn2XU/tqHJJYc+9iR9V4dz VO7Cw5q35RsQf/QHci5cGq5+zRg7tgk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UU4u+f6D; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676363853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=49IJ7O8KaG9bUjSCVtwVGeadJYJCODKyAnH6L7XF9aY=; b=ohf0kN4sZ2OgGUo9qzGKdVwuWVSmbPE5mrP7CtZa3dHMuJt4T72+fES5tpUJi5zZTIjn/E nyiLhik+v7RTbV2U5hIqg+Lrrh9nzX/JquEIn3Vfy3mFm0+/muuOd8sFJMmj5To9gtgIif X+eba5HbIpgVuYED7zgXK4yvw3Ir2QI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676363852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=49IJ7O8KaG9bUjSCVtwVGeadJYJCODKyAnH6L7XF9aY=; b=UU4u+f6D4ATWa2BexKT//KyLUP/0/VJgt8dfBVUMbMfTM2r+a8q/bZ1FslVhlIo6Dv+1b1 0EQtgIhASXRQKfs86Oc4uZW48/OP16lsSHpFEfsqfAa2fg0teQRwt8fe4FvEVcJQzhtlOy ZZ9AU1G+UKyKr8gnws4nt+aUyD8DTD0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-208-whlcj7bIP-qL1QdHi6m4cg-1; Tue, 14 Feb 2023 03:37:31 -0500 X-MC-Unique: whlcj7bIP-qL1QdHi6m4cg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B63CC811E6E; Tue, 14 Feb 2023 08:37:30 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id C013540C945A; Tue, 14 Feb 2023 08:37:28 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v3 4/5] splice: Move filemap_read_splice() to mm/filemap.c Date: Tue, 14 Feb 2023 08:37:09 +0000 Message-Id: <20230214083710.2547248-5-dhowells@redhat.com> In-Reply-To: <20230214083710.2547248-1-dhowells@redhat.com> References: <20230214083710.2547248-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Queue-Id: 8949B120011 X-Rspamd-Server: rspam01 X-Stat-Signature: rwwtmtjenmgi6thyajtorc7bqdan7fwi X-HE-Tag: 1676363853-161492 X-HE-Meta: U2FsdGVkX18++xpqEt30LwaC8lDxVOBhLAKW5RC1NIL4DAseLmpXkeF0aUHxwbuThb9hY1dO/OTcLXz+7Yb1SflQIZs+Isbbb9KIMdZVAuRqNaGwemK8CgABrhhLXEFYD1qCc7QncNw3gF7TER0LVQ72wzJH5IVcZwNrfKDlIW75uTOhcxhc6zqE4osQEht4FLLfmKE0rDktklbzmqP+VmQYZrZiT8V5uiTYKiax0yJSXYQyRsMghFsIgTsSVqpyff7NCCbJA1XwT6AgXJ+Kss/hUMzxSYz6dVpPS7J+paiwvR6hoUJqckRTweSjP4gu2gmKZBEkMTLrCo1DeDmvgwMnHle9TvNYd/B4F6Nf08W2wYO2kv4aurlIPnUXGEbSyR9d5TsTAERRy2k8y8+hDjlmzm/YXrjX+jOCqP9o7/e2Uc+aTKWj4bBe42yTVwMR6Uns/jum/v71/4HIOprrRxlTZ5niYH5cl1bp9lWWynhbm/CIxk2GGkGIlzSEvMaizeu+U142Dzhrfh16DAtlW7XnH30gfIoL+r7kX70f47BJFrTZiLhen4D+yMymmQKRwBGQNoBrl5hDl87NXoG7tePHZMlTKfPtN9RirbOxGPHOhic4ZvkEtewY6ivYtUPjIlmLF0gB9/Qq49YvhLaAt4DPkqPY9UkTY4vYigFupnvGa05s5+wfchZnW+wXCPBNauejGTej3DB7yTFvUyWY5sgKEM17EHVneyHdQ3o60av7BYjONMY9dF+fSwPHRdor1k2NHoyo4JPhSgermh9ZGvU0KPCvYt6u9819z586bi4DWRFfKurv1oxwwohbWW4V6Qf7MHdbFTTWL1/SbQg35DLnGUckFTns33nnGfA9eAKVM98h9UciBhNJ+ON2yeGpdPy5MRFfzD6t7CNh4S8+jr9+MZJRPhIl9zrumrY6DfDFbMUlbszhavnJkdGmqWsQI1xJKsEEOlZGex83poS 3zeXVR4J tgvbKuuUoMCm8wvsAsFnyRyCkTFtYXGt9AczDRi/csbFIhxw5oLmhyX2QAJhePWUQWB4elMe82ICW83BY5r9wGEi1P7DrrV2OOyBc6Alxy0bNq/Y2MPMeDWuVBFP0ZGX3yp+zCbj0qT00gLs32syp0wltKCW2YF1zhVy5wFCmDZjW7A76miAgItBHzVE9MpaFtg9YW489jy+EFQfqKsyrj5fVRbkGtUFGRYsrsIkTbGUAVZS25LrY4phNaMrnbmRmlbpG7VVQVKuJz2G7HV9o1ZPlWpKuorhLPdHe7ukRrS3qnWUhtKB/z3vg9yBH70tLmh5Yt4JLYrkyb09uX8tkDFbT88FO0cJDZXBEEU3RZTiMeAzmL+mc8BRBAGj1Ob6LJdHOHfcNttK0IkuSY+xU/W2qRrI7X2mfeDMKtcHjOMrkfQrhUFcDLkwaVoSpSogQ8f5reMKGTc00O1lck3AVTcSsfAu6W5/MnZpnjtA5qxLvXqyCFiysYF5JcA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move filemap_read_splice() to mm/filemap.c and make filemap_get_pages() static again. Requested-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- fs/splice.c | 127 ------------------------------------- include/linux/pagemap.h | 2 - include/linux/splice.h | 4 ++ mm/filemap.c | 137 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 135 insertions(+), 135 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 4ea63d6a9040..341cd8fb47a8 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -375,133 +375,6 @@ static ssize_t direct_splice_read(struct file *in, loff_t *ppos, return ret; } -/* - * Splice subpages from a folio into a pipe. - */ -static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, - struct folio *folio, - loff_t fpos, size_t size) -{ - struct page *page; - size_t spliced = 0, offset = offset_in_folio(folio, fpos); - - page = folio_page(folio, offset / PAGE_SIZE); - size = min(size, folio_size(folio) - offset); - offset %= PAGE_SIZE; - - while (spliced < size && - !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { - struct pipe_buffer *buf = pipe_head_buf(pipe); - size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); - - *buf = (struct pipe_buffer) { - .ops = &page_cache_pipe_buf_ops, - .page = page, - .offset = offset, - .len = part, - }; - folio_get(folio); - pipe->head++; - page++; - spliced += part; - offset = 0; - } - - return spliced; -} - -/* - * Splice folios from the pagecache of a buffered (ie. non-O_DIRECT) file into - * a pipe. - */ -static ssize_t filemap_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, - size_t len, unsigned int flags) -{ - struct folio_batch fbatch; - struct kiocb iocb; - size_t total_spliced = 0, used, npages; - loff_t isize, end_offset; - bool writably_mapped; - int i, error = 0; - - init_sync_kiocb(&iocb, in); - iocb.ki_pos = *ppos; - - /* Work out how much data we can actually add into the pipe */ - used = pipe_occupancy(pipe->head, pipe->tail); - npages = max_t(ssize_t, pipe->max_usage - used, 0); - len = min_t(size_t, len, npages * PAGE_SIZE); - - folio_batch_init(&fbatch); - - do { - cond_resched(); - - if (*ppos >= i_size_read(file_inode(in))) - break; - - iocb.ki_pos = *ppos; - error = filemap_get_pages(&iocb, len, &fbatch, true); - if (error < 0) - break; - - /* - * i_size must be checked after we know the pages are Uptodate. - * - * Checking i_size after the check allows us to calculate - * the correct value for "nr", which means the zero-filled - * part of the page is not copied back to userspace (unless - * another truncate extends the file - this is desired though). - */ - isize = i_size_read(file_inode(in)); - if (unlikely(*ppos >= isize)) - break; - end_offset = min_t(loff_t, isize, *ppos + len); - - /* - * Once we start copying data, we don't want to be touching any - * cachelines that might be contended: - */ - writably_mapped = mapping_writably_mapped(in->f_mapping); - - for (i = 0; i < folio_batch_count(&fbatch); i++) { - struct folio *folio = fbatch.folios[i]; - size_t n; - - if (folio_pos(folio) >= end_offset) - goto out; - folio_mark_accessed(folio); - - /* - * If users can be writing to this folio using arbitrary - * virtual addresses, take care of potential aliasing - * before reading the folio on the kernel side. - */ - if (writably_mapped) - flush_dcache_folio(folio); - - n = splice_folio_into_pipe(pipe, folio, *ppos, len); - if (!n) - goto out; - len -= n; - total_spliced += n; - *ppos += n; - in->f_ra.prev_pos = *ppos; - if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) - goto out; - } - - folio_batch_release(&fbatch); - } while (len); - -out: - folio_batch_release(&fbatch); - file_accessed(in); - - return total_spliced ? total_spliced : error; -} - /** * generic_file_splice_read - splice data from file to a pipe * @in: file to splice from diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 3a7bdb35acff..29e1f9e76eb6 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -748,8 +748,6 @@ struct page *read_cache_page(struct address_space *, pgoff_t index, filler_t *filler, struct file *file); extern struct page * read_cache_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); -int filemap_get_pages(struct kiocb *iocb, size_t count, - struct folio_batch *fbatch, bool need_uptodate); static inline struct page *read_mapping_page(struct address_space *mapping, pgoff_t index, struct file *file) diff --git a/include/linux/splice.h b/include/linux/splice.h index a55179fd60fc..691c44ef5c0b 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -67,6 +67,10 @@ typedef int (splice_actor)(struct pipe_inode_info *, struct pipe_buffer *, typedef int (splice_direct_actor)(struct pipe_inode_info *, struct splice_desc *); +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); + extern ssize_t splice_from_pipe(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int, splice_actor *); diff --git a/mm/filemap.c b/mm/filemap.c index 6970be64a3e0..e1ee267675d2 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -42,6 +42,8 @@ #include #include #include +#include +#include #include #include #include "internal.h" @@ -2576,12 +2578,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, return 0; } -/* - * Extract some folios from the pagecache of a file, reading those pages from - * the backing store if necessary and waiting for them. - */ -int filemap_get_pages(struct kiocb *iocb, size_t count, - struct folio_batch *fbatch, bool need_uptodate) +static int filemap_get_pages(struct kiocb *iocb, size_t count, + struct folio_batch *fbatch, bool need_uptodate) { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; @@ -2845,6 +2843,133 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) } EXPORT_SYMBOL(generic_file_read_iter); +/* + * Splice subpages from a folio into a pipe. + */ +static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, + struct folio *folio, + loff_t fpos, size_t size) +{ + struct page *page; + size_t spliced = 0, offset = offset_in_folio(folio, fpos); + + page = folio_page(folio, offset / PAGE_SIZE); + size = min(size, folio_size(folio) - offset); + offset %= PAGE_SIZE; + + while (spliced < size && + !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); + + *buf = (struct pipe_buffer) { + .ops = &page_cache_pipe_buf_ops, + .page = page, + .offset = offset, + .len = part, + }; + folio_get(folio); + pipe->head++; + page++; + spliced += part; + offset = 0; + } + + return spliced; +} + +/* + * Splice folios from the pagecache of a buffered (ie. non-O_DIRECT) file into + * a pipe. + */ +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct folio_batch fbatch; + struct kiocb iocb; + size_t total_spliced = 0, used, npages; + loff_t isize, end_offset; + bool writably_mapped; + int i, error = 0; + + init_sync_kiocb(&iocb, in); + iocb.ki_pos = *ppos; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + + folio_batch_init(&fbatch); + + do { + cond_resched(); + + if (*ppos >= i_size_read(file_inode(in))) + break; + + iocb.ki_pos = *ppos; + error = filemap_get_pages(&iocb, len, &fbatch, true); + if (error < 0) + break; + + /* + * i_size must be checked after we know the pages are Uptodate. + * + * Checking i_size after the check allows us to calculate + * the correct value for "nr", which means the zero-filled + * part of the page is not copied back to userspace (unless + * another truncate extends the file - this is desired though). + */ + isize = i_size_read(file_inode(in)); + if (unlikely(*ppos >= isize)) + break; + end_offset = min_t(loff_t, isize, *ppos + len); + + /* + * Once we start copying data, we don't want to be touching any + * cachelines that might be contended: + */ + writably_mapped = mapping_writably_mapped(in->f_mapping); + + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + size_t n; + + if (folio_pos(folio) >= end_offset) + goto out; + folio_mark_accessed(folio); + + /* + * If users can be writing to this folio using arbitrary + * virtual addresses, take care of potential aliasing + * before reading the folio on the kernel side. + */ + if (writably_mapped) + flush_dcache_folio(folio); + + n = splice_folio_into_pipe(pipe, folio, *ppos, len); + if (!n) + goto out; + len -= n; + total_spliced += n; + *ppos += n; + in->f_ra.prev_pos = *ppos; + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + goto out; + } + + folio_batch_release(&fbatch); + } while (len); + +out: + folio_batch_release(&fbatch); + file_accessed(in); + + return total_spliced ? total_spliced : error; +} + static inline loff_t folio_seek_hole_data(struct xa_state *xas, struct address_space *mapping, struct folio *folio, loff_t start, loff_t end, bool seek_data)