From patchwork Thu Feb 16 21:47:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13143987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CF07C61DA4 for ; Thu, 16 Feb 2023 21:48:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 562D36B0074; Thu, 16 Feb 2023 16:48:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 537ED6B0075; Thu, 16 Feb 2023 16:48:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D8C46B0078; Thu, 16 Feb 2023 16:48:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2ADEB6B0074 for ; Thu, 16 Feb 2023 16:48:23 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0AA9B1A04C5 for ; Thu, 16 Feb 2023 21:48:23 +0000 (UTC) X-FDA: 80474494086.20.790B16A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 3E02D1C000A for ; Thu, 16 Feb 2023 21:48:20 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ViuFyVdS; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676584101; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LFENKL+WQlWvhbDc2HxenGINBq9+lQWBR5ko6ef/r4M=; b=mq0V5MCQ0cNhvSJ6ScEi/e4eeRw8nBILa0tawsDhnDC1CdXy8T2UwzrV2D0bkssPd82VdE X+bi+FhukZ1a8EG3v/AbQirxTbqoRddiriAFkq0HYzIhFQtfwFR/zTAhvkVua/L1sBRKTS LFY4vYCpa5lTJqvz4a8EPYO8jN1Qozo= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ViuFyVdS; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676584101; a=rsa-sha256; cv=none; b=ydKsFVnPOqfQP3+DDfFmVa2TW9C8b2uy9pz6Xy2PUbFy6B00qdh9qyftflxQwH/YNVHJwL t7RwoBCcWY30EdcVR1PH3wbV1Ko1KNjsnYNo9o0YuZcRgpIdjR4oBOl5UPqLFjIdGSandl /oBL99gZunK8HZokExl2esIJTHtCRpU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676584100; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LFENKL+WQlWvhbDc2HxenGINBq9+lQWBR5ko6ef/r4M=; b=ViuFyVdS6vSNIQM223c65lCRmxkaqQ2rmX/di7ts2xGlA5QW6RxFlqEQhv/MAfQ0Vh11qf 9D4cVaEywJQEd8mNjmLKQWFwBcgvPRCa2GWJ7c4INSvXctFgJ7Nv6x43Cpxf1PvutTWX13 iHtPn+MivIju6RlK1oC//akaN8+h+UY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-267-_tSm9z9QMGu8nYP2_juG9w-1; Thu, 16 Feb 2023 16:48:17 -0500 X-MC-Unique: _tSm9z9QMGu8nYP2_juG9w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 25232802D32; Thu, 16 Feb 2023 21:48:16 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id E2AC5492B15; Thu, 16 Feb 2023 21:48:13 +0000 (UTC) From: David Howells To: Steve French Cc: David Howells , Jens Axboe , Al Viro , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Christoph Hellwig , Matthew Wilcox , Jeff Layton , linux-cifs@vger.kernel.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com, Christoph Hellwig , David Hildenbrand , John Hubbard Subject: [PATCH 03/17] splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE Date: Thu, 16 Feb 2023 21:47:31 +0000 Message-Id: <20230216214745.3985496-4-dhowells@redhat.com> In-Reply-To: <20230216214745.3985496-1-dhowells@redhat.com> References: <20230216214745.3985496-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3E02D1C000A X-Stat-Signature: e4awcr4mtgdiakoc8pzg7mzcaecbmm5r X-HE-Tag: 1676584100-410250 X-HE-Meta: U2FsdGVkX18yjWzpZSzslfbrAX6CehlQh2ZXx1sICBqlH/DfL+d9whmhI4dvhyn9cn4m7pPNnPUgGKZEgj13wDPg5AQ35aD3i4XdAzDg4vxpJVDyFFNb2WyO7kX8VB2f92dqtxoiGFF/9f45ZZaBuV2BClKt40jhRnmVHhq4RFqL6Rqc0Fv2wi4dylq7M9EupzYMVXJxnWu0QI8imL51D5IBKX3ZE4GEMm/3Xm74RkqA8GdRbftOWwSIGewrKFX+zAh/EqMbEjo4KqlnmXy16jo7SIQCpH7pKVuEXRLhWhy97fgJEbadCctNL2LsVvgUkQAGf4YjdLu3jdsLbtngGrBaCk0RpK5vH7lirgg90fBaiIllO/eXZ7wlHOE9ntZ6TdXYvbA79uQSy1nfj2pGMd5L1XeNlUAAAYGQ4xGXRK1JdIp5zhV5DXB8b6GGhoqvSv7QhDFj8OALCql/OTY5bizRM4ECWzvM8j1qWuxQk4ZJXGqWItbc/C6jCiZJb87B3WgccKB3YAssprQ1dMWjNkmqncn93qSJQJkGhdZTlmGKUA6H4nPmUj7T9X4gsmX2RpHdh68ZGOlTksO/3lcmOzdWKuOrtG0Rpgtowhyq3oxMp0ThaVNQwnVnsur4ugUapk411QwIV+jYIvMjAAdd52xB5vPXY0xWB/WY98iL6vlGeAIxkebbC1OvNG4Cn7FfuyJS1vQ+x92JAZKAV31ks0lC9PzJqgocAOJbPmPE1jhfa3HGwdXbj2i9Sz3QjJc7ixuXG9XL2lvPX61hHfAovo54yodCN41J2Y27gOoTI5gSomFqGiVO90MP9toDVvt1H42pwTSaottTpR77VvXyE6zlm8N1wHAET054qqSDDy7GjPmUJCEHfzpIVpmjlCMa0TgQthlqms8EvOpr11PdgZpVPaLN8moQh93/fz2JA6og0nklHRPWJgCkmiC0+bWk5COzmREcf6VM5qxvI/Z uRNP/cm4 GocnWz2QkFfYxINnm91mbn4GFGjkfufDAdnSM8jZvLt7Is/EDap453mfYCXfI91AxBhuoefKz6RN+eSS8f1QgCnmDTyiwTaZjd13zrF5pkQYvaKgRvN2d0Q0aXCuBGJg0279eldUG0Uulqt9EqvFHkETuxhRFsxrpP2OvL3o+X9NQtaXWbD4KKQUzQxyNASeqwKR6VKjSzVhMna6NyRc/vF+wDlvWLWFiAjtgDMjv8rc66RusBNLPAFap8OKHfxteZ0Ohb6Mn/IVe5wmMMon1YAQUxa3H1stSy0esdhNev47qosWA++/Dni9QkyRMZ0r9gu1JiBdMFXpt2s1TTttK5DllYLgKMXv6aRU/X4pEBBg96VJz4UTVUdZB21GJFWVd717BVwuKBPE5d/MtN0E+u1UT6mupXltdyubl2kuTbheaf8lOQx23ai7359RHpUrS7JvJwOEIMc+AwdQb3bdalg+O3Fpj4jjaB54NEUIsXIGEbtohG9o+UnBu6g5GYz+o3g++w+kJz9lF75CA4AuIaWvoBZlXom+U5MQKn3NC0OXJusU5Xy2RYJHvaFSMqYptg81YmrJCweKzYnP6BF6EqWEtc8sRltP6/46KdgfdRHEjgTVmR6jNi24I3N/Ww41C1e6kLNDLwSUCGwdIQXYKLLEWDXC+SXYt42lHKWCFFGemeSP/CDLrY8q1rEnf/cIK59QUgssKKhZtKJqoZUyfwftItJCyypl8qdRwjICu01hlqfQl93cVcGEUlA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement a function, direct_file_splice(), that deals with this by using an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't free its buffers when reverted. The function bulk allocates all the buffers it thinks it is going to use in advance, does the read synchronously and only then trims the buffer down. The pages we did use get pushed into the pipe. This fixes a problem with the upcoming iov_iter_extract_pages() function, whereby pages extracted from a non-user-backed iterator such as ITER_PIPE aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to shorten the iterator to just the bufferage it is going to use - which has the side-effect of freeing the excess pipe buffers, even though they're attached to a bio and may get written to by DMA (thanks to Hillf Danton for spotting this[1]). This then causes memory corruption that is particularly noticable when the syzbot test[2] is run. The test boils down to: out = creat(argv[1], 0666); ftruncate(out, 0x800); lseek(out, 0x200, SEEK_SET); in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW); sendfile(out, in, NULL, 0x1dd00); run repeatedly in parallel. What I think is happening is that ftruncate() occasionally shortens the DIO read that's about to be made by sendfile's splice core by reducing i_size. This should be more efficient for DIO read by virtue of doing a bulk page allocation, but slightly less efficient by ignoring any partial page in the pipe. Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com Signed-off-by: David Howells Reviewed-by: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1] Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2] --- fs/splice.c | 92 +++++++++++++++++++++++++++++++++++++++ include/linux/fs.h | 3 ++ include/linux/pipe_fs_i.h | 20 +++++++++ lib/iov_iter.c | 6 --- 4 files changed, 115 insertions(+), 6 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 5969b7a1d353..4c6332854b63 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -282,6 +282,98 @@ void splice_shrink_spd(struct splice_pipe_desc *spd) kfree(spd->partial); } +/* + * Splice data from an O_DIRECT file into pages and then add them to the output + * pipe. + */ +ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct iov_iter to; + struct bio_vec *bv; + struct kiocb kiocb; + struct page **pages; + ssize_t ret; + size_t used, npages, chunk, remain, reclaim; + int i; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + npages = DIV_ROUND_UP(len, PAGE_SIZE); + + bv = kzalloc(array_size(npages, sizeof(bv[0])) + + array_size(npages, sizeof(struct page *)), GFP_KERNEL); + if (!bv) + return -ENOMEM; + + pages = (void *)(bv + npages); + npages = alloc_pages_bulk_array(GFP_USER, npages, pages); + if (!npages) { + kfree(bv); + return -ENOMEM; + } + + remain = len = min_t(size_t, len, npages * PAGE_SIZE); + + for (i = 0; i < npages; i++) { + chunk = min_t(size_t, PAGE_SIZE, remain); + bv[i].bv_page = pages[i]; + bv[i].bv_offset = 0; + bv[i].bv_len = chunk; + remain -= chunk; + } + + /* Do the I/O */ + iov_iter_bvec(&to, ITER_DEST, bv, npages, len); + init_sync_kiocb(&kiocb, in); + kiocb.ki_pos = *ppos; + ret = call_read_iter(in, &kiocb, &to); + + reclaim = npages * PAGE_SIZE; + remain = 0; + if (ret > 0) { + reclaim -= ret; + remain = ret; + *ppos = kiocb.ki_pos; + file_accessed(in); + } else if (ret < 0) { + /* + * callers of ->splice_read() expect -EAGAIN on + * "can't put anything in there", rather than -EFAULT. + */ + if (ret == -EFAULT) + ret = -EAGAIN; + } + + /* Free any pages that didn't get touched at all. */ + reclaim /= PAGE_SIZE; + if (reclaim) { + npages -= reclaim; + release_pages(pages + npages, reclaim); + } + + /* Push the remaining pages into the pipe. */ + for (i = 0; i < npages; i++) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + + chunk = min_t(size_t, remain, PAGE_SIZE); + *buf = (struct pipe_buffer) { + .ops = &default_pipe_buf_ops, + .page = bv[i].bv_page, + .offset = 0, + .len = chunk, + }; + pipe->head++; + remain -= chunk; + } + + kfree(bv); + return ret; +} + /** * generic_file_splice_read - splice data from file to a pipe * @in: file to splice from diff --git a/include/linux/fs.h b/include/linux/fs.h index 28743e38df91..551c9403f9b3 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3166,6 +3166,9 @@ ssize_t vfs_iocb_iter_write(struct file *file, struct kiocb *iocb, ssize_t filemap_splice_read(struct file *in, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); +ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h index 6cb65df3e3ba..d2c3f16cf6b1 100644 --- a/include/linux/pipe_fs_i.h +++ b/include/linux/pipe_fs_i.h @@ -156,6 +156,26 @@ static inline bool pipe_full(unsigned int head, unsigned int tail, return pipe_occupancy(head, tail) >= limit; } +/** + * pipe_buf - Return the pipe buffer for the specified slot in the pipe ring + * @pipe: The pipe to access + * @slot: The slot of interest + */ +static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, + unsigned int slot) +{ + return &pipe->bufs[slot & (pipe->ring_size - 1)]; +} + +/** + * pipe_head_buf - Return the pipe buffer at the head of the pipe ring + * @pipe: The pipe to access + */ +static inline struct pipe_buffer *pipe_head_buf(const struct pipe_inode_info *pipe) +{ + return pipe_buf(pipe, pipe->head); +} + /** * pipe_buf_get - get a reference to a pipe_buffer * @pipe: the pipe that the buffer belongs to diff --git a/lib/iov_iter.c b/lib/iov_iter.c index f9a3ff37ecd1..47c484551c59 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -186,12 +186,6 @@ static int copyin(void *to, const void __user *from, size_t n) return res; } -static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, - unsigned int slot) -{ - return &pipe->bufs[slot & (pipe->ring_size - 1)]; -} - #ifdef PIPE_PARANOIA static bool sanity(const struct iov_iter *i) {