From patchwork Thu Jun 29 15:54:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13297122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 973A0C0015E for ; Thu, 29 Jun 2023 15:55:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BC308D0006; Thu, 29 Jun 2023 11:55:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 246C68E0001; Thu, 29 Jun 2023 11:55:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 098228D0008; Thu, 29 Jun 2023 11:55:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DE17F8D0006 for ; Thu, 29 Jun 2023 11:55:16 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9B91BB05AB for ; Thu, 29 Jun 2023 15:55:16 +0000 (UTC) X-FDA: 80956234632.16.0FAB24A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id AB1AA180008 for ; Thu, 29 Jun 2023 15:55:14 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hw4pABhX; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688054114; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dlZE9cCK+MsXM3xyy8z+SEcTpg9WUnrn3xFqBWWAVgI=; b=dU8xzunlUnc5dGovWmJv8bw2aaYmYVmW2DBfN3COC50PN7duvs4JQyOGNT2yR8Fq7/M231 XRyHfUO1QKh91nAR8x5MbP8Q79uj/z1Dq9h6gCraoq83W+exIS2P4Duouk+tKqds/kBEj9 R0wOWNRl6PiwGMuLBOMvmjxNvEgnIAo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688054114; a=rsa-sha256; cv=none; b=G62rX/Zeo3xCjSpX6PKmA/R49Huox07+WQHQE08yrSClRs/A5+RgrfeSDS3HnMzf9NaV5i 1lqgLiU4HqfU5zhh1qbuQGRf6YX1xNivuwDP40KP2wcuMpF55sKybdnEsdQLiRYdD8SIvu bnrBjSirMlYB+l6Jld0SJc8ZxMlsCrc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hw4pABhX; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688054114; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dlZE9cCK+MsXM3xyy8z+SEcTpg9WUnrn3xFqBWWAVgI=; b=hw4pABhX6UvKgb/BkB8l4buROX18eB23zJltUCUXYFpezatFq9SacFtxa7mEzdHqbIfwTV pmVscEk8Pqkryzf5qcwuwZMQDAHERzcxXRy1P6CjlDZsSxm/5mS/dgQky2IG2l1dMgG57O SI+pJ3exWSEbQhmYTwTDxq69yYZDHf4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-631-GQN9IkHCNG6moXhhB7DL1w-1; Thu, 29 Jun 2023 11:55:05 -0400 X-MC-Unique: GQN9IkHCNG6moXhhB7DL1w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6212C8A5150; Thu, 29 Jun 2023 15:54:41 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.42.28.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id F1DA41121315; Thu, 29 Jun 2023 15:54:39 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Matthew Wilcox , Dave Chinner , Matt Whitlock , Linus Torvalds , Jens Axboe , linux-fsdevel@kvack.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig , linux-fsdevel@vger.kernel.org Subject: [RFC PATCH 2/4] splice: Make vmsplice() steal or copy Date: Thu, 29 Jun 2023 16:54:31 +0100 Message-ID: <20230629155433.4170837-3-dhowells@redhat.com> In-Reply-To: <20230629155433.4170837-1-dhowells@redhat.com> References: <20230629155433.4170837-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: AB1AA180008 X-Rspam-User: X-Stat-Signature: tktophoq8t7zn3mxixngie1xfrzssw5k X-Rspamd-Server: rspam03 X-HE-Tag: 1688054114-54916 X-HE-Meta: U2FsdGVkX19ykaXphIQbbgrDjhLEvaa/bjAsOEA5e53YPgcbT1CAnBPOaTA0panV/R78yHZzW3j+J4thzpFrvlHWVol1bYuNfUVZ5tAv29++pmYA5Wk40YFfoz3JR12Kpx1GstVDqkd5aJre7tpm7+BClXP4rZT9Mam4VAtYFc+F4gBOiIEWdSOfEplzyy9ehxD6k1JgZqCIDy54qcALCceU/wMBjti2QclOLF1lWNHZWLFpstgLPHyfVJNTw2o6RmoR6a+1XXLi1GwB7WIm9H+c3f8SPdyhuC8EINtpRlMbSsgiX6CyED0jcMXKZJKKUVS4O19naJTd+NLyDOvZB76LKFoIpM3lkjGeu4INg91f1VkZBFou5+pXmDDaVwE/PD6F89lGa1sy8nXob30mzs9rs8Sffr51hptAlV5W9zNewfbvLt3bNUAwqXyS8lvRIUqnoOowGClxQVgssfNVj5QxWZx4L6B0C9IagLCInexT3oXpsE4XDNCSgOMub6lSiEaMVFv/lU4qi6wSuhGnVjq8xD1dS9uif+ZI9NnqXl0gHkEU1EtvCFJZw4q7ateupFRs1kb7DnqAmSuaIPSTJNEFJYxy5jdhCvWSt8RJT7VL+LXasNQ11Femh2VIPwwj/Xr8LR2D7a4REBj2VhWuS761nsAvMnvGdwsO7gwmIwlHdJqgrXaTEftTTl4T1wbIuIDdvJx8w1/ZjZ3WNL6ISNTTUQlFjTGpmDrmNcm2LtRhw5vWpwhRZNTLfy9HY1s8ek/v1agGYA5ZZQ8NcWtv9F94MZc1VdjJSn2o229+12LWMn0laBMP6Yr2+fpgZci2OsrZr/y/8M/JmYjg3EIZGy4CbWcydHdqHcFXQox3EKTFjiQlYregBePHJQaEpz18nixaR7lX7wwb9kIlH53Wg7zFpEQP+In9zEefzSSlYD/swIoCHrXGMSlLurlJEymBIdBtPXVn8X6Ds9W+jQ9 RhKM+481 QeawKTAqmlkpMbswQvxeuVTFULMYdzKY4tQbjpJbMv64Ydq9E+QPqB553USuGO98lPAT1pfJQiw/x9yE6pB0wMIgfnF5R6QESbnmpO1hqDJ2DMbStq+WCRa3fw10VSxjP/0qGHM5eGi0hC6/KHf2LHmoJX8TH8Q0/WvJDMut22nFIhgyS9FaVadjDGD7XEDKqU3rko5fYFhdJtChirEWR/hB4k0uLoVsvfUuedH+BDxWdbiqreKaqwOuFHWvNJEqt+rIDYVSukiSDtq0VYYDGdySh3bpjtx74juu4Hem0m0akU9u3HWPLaSb4kHAUVJ5S3B8nKtXCtpul3kzknNT8BPO1o5HEJa1Z0DVn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make vmsplice()-to-pipe try to steal gifted data or else copy the source data immediately before adding it to the pipe. This prevents the data added to the pipe from being modified by write(), by shared-writable mmap and by fallocate(). [!] Note: I'm using unmap_mapping_folio() and remove_mapping() to steal a gifted page on behalf of vmsplice(). It works partly, but after a large batch of stealing, it will oops, but I can't tell why as it dies in the middle of a huge chunk of macro-generated interval tree code. [!] Note: I'm only allowing theft of pages with refcount <= 4. refcount == 3 would actually seem to be the right thing (one for the caller, one for the pagecache and one for our page table), but sometimes a fourth ref is held transiently (possibly deferred put from page-in). Reported-by: Matt Whitlock Link: https://lore.kernel.org/r/ec804f26-fa76-4fbe-9b1c-8fbbd829b735@mattwhitlock.name/ Signed-off-by: David Howells cc: Matthew Wilcox cc: Dave Chinner cc: Christoph Hellwig cc: Jens Axboe cc: linux-fsdevel@vger.kernel.org --- fs/splice.c | 123 +++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 10 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 004eb1c4ce31..42af642c0ff8 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -37,6 +37,7 @@ #include #include +#include "../mm/internal.h" #include "internal.h" /* @@ -1382,14 +1383,117 @@ static long __do_splice(struct file *in, loff_t __user *off_in, return ret; } +static void copy_folio_to_folio(struct folio *src, size_t src_offset, + struct folio *dst, size_t dst_offset, + size_t size) +{ + void *p, *q; + + while (size > 0) { + size_t part = min3(PAGE_SIZE - src_offset % PAGE_SIZE, + PAGE_SIZE - dst_offset % PAGE_SIZE, + size); + + p = kmap_local_folio(src, src_offset); + q = kmap_local_folio(dst, dst_offset); + memcpy(q, p, part); + kunmap_local(p); + kunmap_local(q); + src_offset += part; + dst_offset += part; + size -= part; + } +} + +static int splice_try_to_steal_page(struct pipe_inode_info *pipe, + struct page *page, size_t offset, + size_t size, unsigned int splice_flags) +{ + struct folio *folio = page_folio(page), *copy; + unsigned int flags = 0; + size_t fsize = folio_size(folio), spliced = 0; + + if (!(splice_flags & SPLICE_F_GIFT) || + fsize != PAGE_SIZE || offset != 0 || size != fsize) + goto need_copy; + + /* + * For a folio to be stealable, the caller holds a ref, the mapping + * holds a ref and the page tables hold a ref; it may or may not also + * be on the LRU. Anything else and someone else has access to it. + */ + if (folio_ref_count(folio) > 4 || folio_mapcount(folio) != 1 || + folio_maybe_dma_pinned(folio)) + goto need_copy; + + /* Try to steal. */ + folio_lock(folio); + + if (folio_ref_count(folio) > 4 || folio_mapcount(folio) != 1 || + folio_maybe_dma_pinned(folio)) + goto need_copy_unlock; + if (!folio->mapping) + goto need_copy_unlock; /* vmsplice race? */ + + /* + * Remove the folio from the process VM and then try to remove + * it from the mapping. It we can't remove it, we'll have to + * copy it instead. + */ + unmap_mapping_folio(folio); + if (remove_mapping(folio->mapping, folio)) { + folio_clear_mappedtodisk(folio); + flags |= PIPE_BUF_FLAG_LRU; + goto add_to_pipe; + } + +need_copy_unlock: + folio_unlock(folio); +need_copy: + + copy = folio_alloc(GFP_KERNEL, 0); + if (!copy) + return -ENOMEM; + + size = min(size, PAGE_SIZE - offset % PAGE_SIZE); + copy_folio_to_folio(folio, offset, copy, 0, size); + folio_mark_uptodate(copy); + folio_put(folio); + folio = copy; + offset = 0; + +add_to_pipe: + page = folio_page(folio, offset / PAGE_SIZE); + size = min(size, folio_size(folio) - offset); + offset %= PAGE_SIZE; + + while (spliced < size && + !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); + + *buf = (struct pipe_buffer) { + .ops = &default_pipe_buf_ops, + .page = page, + .offset = offset, + .len = part, + .flags = flags, + }; + folio_get(folio); + pipe->head++; + page++; + spliced += part; + offset = 0; + } + + folio_put(folio); + return spliced; +} + static int iter_to_pipe(struct iov_iter *from, struct pipe_inode_info *pipe, unsigned flags) { - struct pipe_buffer buf = { - .ops = &user_page_pipe_buf_ops, - .flags = flags - }; size_t total = 0; int ret = 0; @@ -1407,12 +1511,11 @@ static int iter_to_pipe(struct iov_iter *from, n = DIV_ROUND_UP(left + start, PAGE_SIZE); for (i = 0; i < n; i++) { - int size = min_t(int, left, PAGE_SIZE - start); + size_t part = min_t(size_t, left, + PAGE_SIZE - start % PAGE_SIZE); - buf.page = pages[i]; - buf.offset = start; - buf.len = size; - ret = add_to_pipe(pipe, &buf); + ret = splice_try_to_steal_page(pipe, pages[i], start, + part, flags); if (unlikely(ret < 0)) { iov_iter_revert(from, left); // this one got dropped by add_to_pipe() @@ -1421,7 +1524,7 @@ static int iter_to_pipe(struct iov_iter *from, goto out; } total += ret; - left -= size; + left -= part; start = 0; } }