From patchwork Wed Mar 29 14:13:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EE85C74A5B for ; Wed, 29 Mar 2023 14:14:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2E206B0074; Wed, 29 Mar 2023 10:14:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DC296B0075; Wed, 29 Mar 2023 10:14:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87DF56B0078; Wed, 29 Mar 2023 10:14:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 79F8C6B0074 for ; Wed, 29 Mar 2023 10:14:10 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 43FDC1401E5 for ; Wed, 29 Mar 2023 14:14:10 +0000 (UTC) X-FDA: 80622130260.23.13103B8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 64EC940025 for ; Wed, 29 Mar 2023 14:14:07 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=beBy+dD0; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099247; a=rsa-sha256; cv=none; b=YBV+lx8vwAg9JSpRKj9y14/0J/580cj4o8YLdRZy1uU6xpsNbngTSkuQNVifFdfVgXs3r+ O/fESI+GDcBzhaOlrUGCWvUVWcbdhLNMkLU5AEHIOVA7M6rn596c3c2+jAkUFHe7k8cYW6 XCEPcB/FFv31wg470WD41MAkWRZyalI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=beBy+dD0; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099247; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fsOLHSRSLMInNcD449MTPXsGwp1kWYvONeYqTpf/Ez4=; b=6pT30KDqJIBkblQSKqodwLluz3/mfmwgVXqgKphsg7igR9TJ18YpY3yDJbHi4vFmFI/g/i rqg/J1S0Uo7OjMTKMGCPV1YDHcYa2rrjED/Z976K7hPsWFnlmucS2PqA7Xv/UFDrdu6BI9 y4zRwIzz0lr4eliq35/wdXcCYx8wJdY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099246; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fsOLHSRSLMInNcD449MTPXsGwp1kWYvONeYqTpf/Ez4=; b=beBy+dD022NdWtdaVKpwN4DF5Zv3960mDrhNFTVvEAOGATKOBtVvvg4/ScRqkNKm5PnO1y /Mt1spWdGcjxaOd2khxraHm1drcUpTrtrNvnVpwc32UatUTTollwfO/4wOAgPlLhezjkXK 1c85G+Vadt0vdvVC6gb68f/GpPMfyQc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-516-Xdb9sur-PoeNk1trWuucgw-1; Wed, 29 Mar 2023 10:14:03 -0400 X-MC-Unique: Xdb9sur-PoeNk1trWuucgw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 080F01C06EE7; Wed, 29 Mar 2023 14:14:02 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 94AC618EC6; Wed, 29 Mar 2023 14:13:59 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steve French , Shyam Prasad N , Rohith Surabattula , linux-cachefs@redhat.com, linux-cifs@vger.kernel.org Subject: [RFC PATCH v2 01/48] netfs: Fix netfs_extract_iter_to_sg() for ITER_UBUF/IOVEC Date: Wed, 29 Mar 2023 15:13:07 +0100 Message-Id: <20230329141354.516864-2-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Queue-Id: 64EC940025 X-Rspamd-Server: rspam01 X-Stat-Signature: nqewoswmbaoz45hcn9z6aanxgygk1nu1 X-HE-Tag: 1680099247-447699 X-HE-Meta: U2FsdGVkX1+SkMJ2YyR5/S+QhvTDpJMj+tLLIMowt6vnLDzTV7quY5EFTyiL5//E7GTFM59N3ShH0qyjnpyujybHuZwQzLiEBBOyt9vp0ovT6zKE5zOlvyHjOI8yPOXzkv2ewHl4ByUC6l05KvE9v7q5VNZjvw/DcrRZbX0Cd2giCGfnkUj+KyeBnqNhvzRh23ddWdjPVg4qzZzryivsXznX4hY1wYrL5b+yEjeBt3Nc9iLhMoJ6LyvShPtpmimKz3wkVeL5NU3FmmPOClSi2JELfyOyntMuAeJySKDmTUEAQDKIuS54EPGQtYadnjK3nO9BCvUAxED8iRbV0FDEAzrRTNo3xHZnSMQSbNzsLiygw+5Jpjt3bUKsnSqcAhE0ZGpU1EsgRTWYqdUSEC69hfpNL6Qb8VqY1WFdkR6fd2r1Mf3QydosN0q4vmxiTWQbTF3u2s+GfD3P++unzUAdJAyAK9KxD3B+X+LNNOykqn1Kp6CZAlbPYfTbplQBwmVxTCHhGIEvv/Tlll8jZGEkhfUNRpq9w/GjpIo/nzHRaVmo0kuntWkHaEY+ppR6xUGKS1ijk6QbiX+hxT4QNjOQLlaUIsGRrRqIVNdBObvBej12bgSjygcrPDYxXT3cT8bWwNFhCWG/D9KDHNFymYAjEV12HWPBZyGNqiB0WBQXDHih0jbBx9cyqgJhPqRTLM7ltQ2Ba7oklBkvFBZyxlIVo8tT3Qw2YDi4tMoZ+7rQk8EQVvA18pbave5/bhj/Wqzc45Z5j3cZ+O+xeO/CRS3j42NDKwfQ25SdU0+qx5P0Az87GslFqMua8H4SS2tyeVUVwgMf7f7QrtgdWx/fTuQkWtQ3sQ5SsHVfKDwt6+j+qlEktWp+/5K3zxWksg9Fc/Kt8O+xkgU2SeL7sn4/JT5eVcLCPky2YMNJSp9mVyMcaNOYUdIjuTYHXwlE7EdxKRwHA3Ax5aqodzw5bGhFlH+ 0JIZAIs7 1JQ9JqRgqC7N7tWctwN/M2J2VAxpL/YndsWt0PLMEnjtVLEYrCS/WpZvEopuWr6H6IhL43ptDpzizrGbDaZLn8B6e00q5m4yLzhS6jWBgRUP611MzEar4fH39W7zj3MTeHng+vwkegbzxWJ2BLA9QIY1y0HTnh0iMio17HxinD6ALkHV9KOyI2/lbEpx3+IQt5144VgZw8BQdZtLqAezn8F4W92mM0Gq8P3+zQcpl7G6zvi1L1bNleYjZQriEmW6UyjlmbwbsssDOxzKpM0nNtuslkjgdieWdc9cBvGOOtIUchv8efnjWNT4XJmGrZsI+5l6RjM6J4ucg0bNy2go1Ur626bd46VMEHqf/zrqCPGKCl6Tl/aCeG7aY2LcKrmNeS1tpawkDTAkniEbapuYiU2A6Cs6pe4nDj6mzHvrrYbrkzX6M/KAOCklegjaAzHvE07ql X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix netfs_extract_iter_to_sg() for ITER_UBUF and ITER_IOVEC to set the size of the page to the part of the page extracted, not the remaining amount of data in the extracted page array at that point. This doesn't yet affect anything as cifs, the only current user, only passes in non-user-backed iterators. Fixes: 018584697533 ("netfs: Add a function to extract an iterator into a scatterlist") Signed-off-by: David Howells cc: Jeff Layton cc: Steve French cc: Shyam Prasad N cc: Rohith Surabattula cc: linux-cachefs@redhat.com cc: linux-cifs@vger.kernel.org cc: linux-fsdevel@vger.kernel.org --- fs/netfs/iterator.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/netfs/iterator.c b/fs/netfs/iterator.c index e9a45dea748a..8a4c86687429 100644 --- a/fs/netfs/iterator.c +++ b/fs/netfs/iterator.c @@ -139,7 +139,7 @@ static ssize_t netfs_extract_user_to_sg(struct iov_iter *iter, size_t seg = min_t(size_t, PAGE_SIZE - off, len); *pages++ = NULL; - sg_set_page(sg, page, len, off); + sg_set_page(sg, page, seg, off); sgtable->nents++; sg++; len -= seg; From patchwork Wed Mar 29 14:13:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15946C6FD18 for ; Wed, 29 Mar 2023 14:14:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 720136B0075; Wed, 29 Mar 2023 10:14:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 681806B0078; Wed, 29 Mar 2023 10:14:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D59B6B007B; Wed, 29 Mar 2023 10:14:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3F98F6B0075 for ; Wed, 29 Mar 2023 10:14:11 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E86D61A065A for ; Wed, 29 Mar 2023 14:14:10 +0000 (UTC) X-FDA: 80622130260.18.5D720C0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id B25592001B for ; Wed, 29 Mar 2023 14:14:07 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TniwxOA8; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099247; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s7FY6IkB7NRnaaunO8kQ3wM8dgoy+C5dVBJnz6ApgXk=; b=e0qW0C5JZJiZf2qK/JetFZ9d7tyTKqQdUmoLpC2Y80BgETqYlszs7SaZiiDleWqMHZ1Ic/ rDxvTFPbuvrr3BqxWekRdO1pai65WxzsPlbXUzHdVc4Qnxjx1RKEJ7oFvY3ifFGgGY7xQ/ eL8f7y7lcVrunHDD+3VaBLchImMSqJo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TniwxOA8; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099247; a=rsa-sha256; cv=none; b=Nt0XeRjZ4v+ZCMccJJUl94R/U88kocN/ZZbcn0p8jLlbC4FnIcZhN1umQr5iF+W/+smdUx qVDdVjlhQTLrXSi0PWZC8/4letLlh7i3XkvXrZrMAkm3UVMwvnujAHn/t356U01JTLv/KM GT/558+MQ2XRV98t3ejTuOviazejODI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s7FY6IkB7NRnaaunO8kQ3wM8dgoy+C5dVBJnz6ApgXk=; b=TniwxOA85WrnSeGbVOc8y0ccQJApSTOUZM6dP7tai2wOQTyzIZfJLCU443ZBvGzmq4TccO 3jP+z/m0vrK+TMIuQc610snRh4YYzeE41STRzbhJ6P7d86pCN9EwsAgwpUZ13mhIgo/J5j jEeuf/PswlY1cEMTRzh4vK4Wj5sAC3A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-440-h-W_J7FgPEG0aunda_tCKQ-1; Wed, 29 Mar 2023 10:14:05 -0400 X-MC-Unique: h-W_J7FgPEG0aunda_tCKQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AF299855315; Wed, 29 Mar 2023 14:14:04 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6A6F492B01; Wed, 29 Mar 2023 14:14:02 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org Subject: [RFC PATCH v2 02/48] iov_iter: Remove last_offset member Date: Wed, 29 Mar 2023 15:13:08 +0100 Message-Id: <20230329141354.516864-3-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 4g1hrfzdxfh8hgq5iq9b9mtn7fyrd46f X-Rspamd-Queue-Id: B25592001B X-HE-Tag: 1680099247-492435 X-HE-Meta: U2FsdGVkX19+d3wpYdpYA5UcUIDsap2iqUYQhzaPNBJEA+WAczkMOk4hgY11ZxqJgp3M+ZJOCU4ilbeidiER+sN+uSaF/JqMzE8p1AHacGgxvqL04clB3TqVrZgpDzqgangHshXFJ9nhriX5ZxNBHpOZoJvWLT3B/d4SlVSkK5ImwPMcRJtbZfE/e/Y2AXdNWD/gaRcGgg5WxvNTKm42XWUdET1QTiw5gGSBVLF5WM2xo7xOMD+FaHL8aMyBk3VPEi7EoU09u9zkfFf6vgReGDCQ8NJW2E4IqbtseeBoSYWhmraheFGinlIcujODaDQvFRGMAaHoGDXaxh5jAOqk1v0g6fDFpQrJdZ9nqR1h1eBzdplYeP9ypnd04cj40ApXuwgidj4y/m8I0ub6ajnt7WzPK/diKi3MaSlqIhtYDl2ZO6kXZMIw5u0oprX7YZB+1zoft4FH58QSTIp5VE/F/lg4IelgvZnFrxmxD3Ya4JXWZxuYfR8W3TaIoC3SFmvkt0plWcpWrGhodhZ/OvAR7j8Y/73awZUYwV2tFP9rEu0nlu/EvuQ4HAqDkk8zuz+okFormkPSc2iNiCkyXQijLBFTS1DSQ9tn43esPWxgkLxWFKhHSUXRF5QCxeuLz5UQXGx8gpHdu2SyGerVx/CESjRNQNGEnmxYp/8RZsH1AD7SL+oYVck1tEss+AgiRVrqU2wTDgA47KhQ9vt1Is9r6IhKMwUsI4HLrUrwlGhvDMydtBsFcV95g481iXXYBkweRNoQy5LF201Y0/opKFQ1X89pCDMtLxnaeEKgKH/f+Pu32TWrqEWsa6kLq19nMnRM29aaWqroPLvFnEjJePVXk2qX67XNUCH5ETniI6M+TCkTXnstrX3gBS3tJh94RZk6HQinhN7sO4fZCIL+rhY0u36De+sa3UgZ/Nih766Tsl7bGuZAGCuJMHxNDqmWNDG7wPWOIo4WXfxeLK9l6uY DR1pbSwq Ijc2328kqXLw00ehfRFIR3RzvDBvgO1MZGvDDDvpkitUPDZyqwuYJkcOBCeOgZDGf01kdYU7W3PqCWq0SEXzcuAvS0P4PiFpDR36NidzRyjWyVCTglb+VzR/N1Fc+rVqGy6sy9FjggB4A/4sd4/n/W94ItrMDD6UFnaNrgGSzic/uz54fHUHlwO0yMMOBLmMdOA1NnEpXF7dkbJ1OvMxGS4iyFh8/lB5kID81CpnHPdh80bvqmpacHT57jewc2b/JpatQuw2frk/uTdhWAXpqbzLdMK7ZgNNDlcP3Ftg+l1aE6+0fx9+xtv60X/7o3ppvv8AkjMPxYIyoBc3/itypg8N5GugqL7TpbKO2df3wnthK4bTgQ/WpGTy9pdoO660F8VuJsVfVe/hDtbyLka9RGOuWJUuGnvSxN2CUcFPGtU7kGAZFMk5rc3nrP3M+GnNZmgfi1m4EFF6hhd5EMrg4cElJ13GOH6UnHVX0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the removal of ITER_PIPE, the last_offset member of struct iov_iter is no longer used, so remove it and un-unionise the remaining member. Signed-off-by: David Howells cc: Jens Axboe cc: Matthew Wilcox cc: Alexander Viro cc: Jeff Layton cc: linux-nfs@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/uio.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 74598426edb4..2d8a70cb9b26 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -43,10 +43,7 @@ struct iov_iter { bool nofault; bool data_source; bool user_backed; - union { - size_t iov_offset; - int last_offset; - }; + size_t iov_offset; size_t count; union { const struct iovec *iov; From patchwork Wed Mar 29 14:13:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6915CC6FD18 for ; Wed, 29 Mar 2023 14:14:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE7CE6B007B; Wed, 29 Mar 2023 10:14:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E71336B007D; Wed, 29 Mar 2023 10:14:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEA436B007E; Wed, 29 Mar 2023 10:14:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BAE136B007B for ; Wed, 29 Mar 2023 10:14:20 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 856D580D61 for ; Wed, 29 Mar 2023 14:14:20 +0000 (UTC) X-FDA: 80622130680.06.97AB8C3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id C3A6C4001A for ; Wed, 29 Mar 2023 14:14:17 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hDlHtYeB; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YKmPIBLTzwmxTywoQlbWhVqEAglt5Diim843HzqnLOY=; b=mxxVk284vf6q0rz8LJOSWCXfxu69Tj0xUPE/rzMKTJK/i6pjtiFxfaXuMGLt5JOwaaYn0v t+6UUCZDiHY2zYO5/FeTXaUHoWKwPVyWiGnlbq8kjCm/9DQuomtjiXkdeyGbW1pqCjKhEN WvJ65QYO4qMX4v95S0PIDWwR5GJBVFY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hDlHtYeB; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099257; a=rsa-sha256; cv=none; b=R39DxarviwgIWV9EoI4JhUGFnD8ZAxBzOXQw+cbyjJy4hcld/JKjhypBusKytYnsA30KWU 7L/9i0UlH84Aob06SVevScdrcLmUlAn4vC57ZUBcqbf3GanpfCf0nk3rehrqmfffaET2o7 XYfLcTh7uFbes2TQoNUYjDl9Pr1zHoc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099257; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YKmPIBLTzwmxTywoQlbWhVqEAglt5Diim843HzqnLOY=; b=hDlHtYeBApnAtpi+hhz8bITqDsRcvpmnypOxhJHTIjM3/Pqrw0kutjresj5Fb6xy5ai46+ RhFm7THjVC1PYnwvGm4uOMmlW5hGZQvgfQS5Osw+kHvPsIer514JTYOUUXrVpXwgUaHbZ/ JPBvdyTiLuVxd3pqnETiJQhDPDYaUkE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-132-y_B6QV2XMFK1PdJabznHmQ-1; Wed, 29 Mar 2023 10:14:09 -0400 X-MC-Unique: y_B6QV2XMFK1PdJabznHmQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 93F51381459A; Wed, 29 Mar 2023 14:14:07 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4B2C718EC2; Wed, 29 Mar 2023 14:14:05 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org Subject: [RFC PATCH v2 03/48] iov_iter: Add an iterator-of-iterators Date: Wed, 29 Mar 2023 15:13:09 +0100 Message-Id: <20230329141354.516864-4-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C3A6C4001A X-Stat-Signature: xcezha1ooxudu81emtig6eqbyufnzxcr X-HE-Tag: 1680099257-307423 X-HE-Meta: U2FsdGVkX19wNt5Zum2bLXxIOlrqMuVDKKuK/WVtT3wX3BAnLdN/eUXtwaxQ4fftVeYQ/mnmWHkR7ppE4MTYuirQm/QNA3msHeM2fHymjxODVxwnRTAZtisquYMZAghGa3VK3xcgA7+I53MkOr7GGNTv2mVNVhKOoIpfts+PFgeEBqDAXJCLeeePcp/BzgoBh+nR80QzUS4MYa0klWpM5H7x/2H2JSYhrWCk/S2eSYFVxtTnJgjJcAFnVGDnCq6jhkVu9dvatVDJ31CXJxbQNV5YSF0Mb/1DCeo9Is41u4ZA+12MYs9KZJevkqIzT+brtaCtNJZg9TLJ8/2GYycBuEMneJEfd0D4e2t1fXryDRTK9jO9msXJcGEo5QxT6mTuhW9DfLe4RbIKhsgJFdLcCjqjGvJg/5Np1Fo5Dj4ukCk8YdPvKCcRMkaFOxavbNvbz9Usfp7F8+rHDjR3U/FXstnzozhu1TZ+/1SjSg/SHThWySOeNbXoMRNrYuYoj3Khg+db8FZzWtOIX7gokh4evaGPv22/qZ/NeyKd91LifJjMiEucgUi8fHuINcI8LAQgRv3A267A05Z2+dh8IHq8bgMk5YxAKF1Fn0x4Ttg7G5hfI6nupsSMtVPOBHaZ0JQ68BTjH5rruWDeydG0rrP+l04es5ietQyxGuQPAxK9148lHwi3xkJUkpew8pemyd8F00jVvfuiuDRLgtiUL99TrYAMGPoG5BTfVoFkCGfPSFSLJuzQpAsWf7RUHq2D+YZzEevNEQhrH8Gb1CvUVrpWDpg7KfMjplE335hBB29jBLuVjkomdY6xnaEICDFjDHrdxjgkgbzu7y8vQfrYIXK/Eukiu+siAeM315cxUyyt9XNF83AH8cLFKs8lXK0eNTQWkC4OuLdZqlPwI3ueWqbNWOqCO68P053RGtGQM25YnQl/MqIMBJuFftfd/+0tXWr+vsqbtprPH/EzeCW1Tj9 dJH+pVXn Uz5FDPB/xxmOlAw+y4AkkD9RuQjE2eZW/AVbkZ/otAXOhftVkaX/pTK15cNt9RpkUte57cgcYw8a1Y4f7P2i87Jo3IjSB4mXUMfrGwXaDQKFLmVzWTTqHOAZNQpfCiQc9qe0RKMkhBp1CliHaY8MlE7wvd3tUQ2v+NiaaoadDV6gOFrJScfwDsRnE5Xm1+v53CEDI+rbJUsVXhslWZ6VYjJIsiNNvsG2+YC721krVQyX5IAoG6bA+IlVrwfz4WwE0xIPooZ/RobrDdPTG6oQ3cif0+XAs/knen9n+5dBM12wBdy06FF4kSW0iBmFmPqLU0B0jKdlYw8DxSCpfUI4DMEVdafmUGsjI9INUIAJnQR6+a2u2asr501bf+697yivfyCXfiapi/YDl1yp/qu9DnHphNHGN2NZeuV5QWJH6StUVjFsGB8VIv0tknTQY382KNDW4nqTQW1F97VFOYpCxIflRTHP9ZqIUf9EYj+xKEbdFAA7N9AbF0C42u+kPohBNeyXEK+rqjBSFtLUgEThYRXSK6b4Pjq12pJCd3Ka+WAgLU2SjeF3TW6FJnEAL25cETWgBusyE/Gu3AyUQCO3mDpjCjLaBk9NWNtF6kyr/t1AInfjwDABJtP3Ry2wsGJ8ufY5xMBaDIDXg9hI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new I/O iterator type, ITER_ITERLIST, that allows iteration over a series of I/O iterators, provided the iterators are all the same direction (all ITER_SOURCE or all ITER_DEST) and none of them are themselves ITER_ITERLIST (this function is recursive). To make reversion possible, I've added an 'orig_count' member into the iov_iter struct so that reversion of an ITER_ITERLIST can know when to go move backwards through the iter list. It might make more sense to make the iterator list element, say: struct itervec { struct iov_iter iter; size_t orig_count; }; rather than expanding struct iov_iter itself and have iov_iter_iterlist() set vec[i].orig_count from vec[i].iter->count. Also, for the moment, I've only permitted its use with source iterators (eg. sendmsg). To use this, you allocate an array of iterators and point the list iterator at it, e.g.: struct iov_iter iters[3]; struct msghdr msg; iov_iter_bvec(&iters[0], ITER_SOURCE, &head_bv, 1, sizeof(marker) + head->iov_len); iov_iter_xarray(&iters[1], ITER_SOURCE, xdr->pages, xdr->page_fpos, xdr->page_len); iov_iter_kvec(&iters[2], ITER_SOURCE, &tail_kv, 1, tail->iov_len); iov_iter_iterlist(&msg.msg_iter, ITER_SOURCE, iters, 3, size); This can be used by network filesystem protocols, such as sunrpc, to glue a header and a trailer on to some data to form a message and then dump the entire message onto the socket in a single go. [!] Note: I'm not entirely sure that this is a good idea: the problem is that it's reasonably common practice to copy an iterator by direct assignment - and that works for the existing iterators... but not this one. With the iterator-of-iterators, the list of iterators has to be modified if we recurse. It's probably fine just for calling sendmsg() from network filesystems, but I'm not 100% sure of that. Suggested-by: Trond Myklebust Signed-off-by: David Howells cc: Trond Myklebust cc: Anna Schumaker cc: Chuck Lever cc: Jens Axboe cc: Matthew Wilcox cc: Alexander Viro cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-nfs@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/uio.h | 13 ++- lib/iov_iter.c | 254 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 260 insertions(+), 7 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 2d8a70cb9b26..6c75c94566b8 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -27,6 +27,7 @@ enum iter_type { ITER_XARRAY, ITER_DISCARD, ITER_UBUF, + ITER_ITERLIST, }; #define ITER_SOURCE 1 // == WRITE @@ -45,12 +46,14 @@ struct iov_iter { bool user_backed; size_t iov_offset; size_t count; + size_t orig_count; union { const struct iovec *iov; const struct kvec *kvec; const struct bio_vec *bvec; struct xarray *xarray; void __user *ubuf; + struct iov_iter *iterlist; }; union { unsigned long nr_segs; @@ -101,6 +104,11 @@ static inline bool iov_iter_is_xarray(const struct iov_iter *i) return iov_iter_type(i) == ITER_XARRAY; } +static inline bool iov_iter_is_iterlist(const struct iov_iter *i) +{ + return iov_iter_type(i) == ITER_ITERLIST; +} + static inline unsigned char iov_iter_rw(const struct iov_iter *i) { return i->data_source ? WRITE : READ; @@ -235,6 +243,8 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray, loff_t start, size_t count); +void iov_iter_iterlist(struct iov_iter *i, unsigned int direction, struct iov_iter *iterlist, + unsigned long nr_segs, size_t count); ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start, iov_iter_extraction_t extraction_flags); @@ -342,7 +352,8 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, .user_backed = true, .data_source = direction, .ubuf = buf, - .count = count + .count = count, + .orig_count = count, }; } /* Flags for iov_iter_get/extract_pages*() */ diff --git a/lib/iov_iter.c b/lib/iov_iter.c index fad95e4cf372..8a9ae4af45fc 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -282,7 +282,8 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, .iov = iov, .nr_segs = nr_segs, .iov_offset = 0, - .count = count + .count = count, + .orig_count = count, }; } EXPORT_SYMBOL(iov_iter_init); @@ -364,6 +365,26 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) if (WARN_ON_ONCE(!i->data_source)) return 0; + if (unlikely(iov_iter_is_iterlist(i))) { + size_t copied = 0; + + while (bytes && i->count) { + size_t part = min(bytes, i->iterlist->count), n; + + if (part > 0) + n = _copy_from_iter(addr, part, i->iterlist); + addr += n; + copied += n; + bytes -= n; + i->count -= n; + if (n < part || !bytes) + break; + i->iterlist++; + i->nr_segs--; + } + return copied; + } + if (user_backed_iter(i)) might_fault(); iterate_and_advance(i, bytes, base, len, off, @@ -380,6 +401,27 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) if (WARN_ON_ONCE(!i->data_source)) return 0; + if (unlikely(iov_iter_is_iterlist(i))) { + size_t copied = 0; + + while (bytes && i->count) { + size_t part = min(bytes, i->iterlist->count), n; + + if (part > 0) + n = _copy_from_iter_nocache(addr, part, + i->iterlist); + addr += n; + copied += n; + bytes -= n; + i->count -= n; + if (n < part || !bytes) + break; + i->iterlist++; + i->nr_segs--; + } + return copied; + } + iterate_and_advance(i, bytes, base, len, off, __copy_from_user_inatomic_nocache(addr + off, base, len), memcpy(addr + off, base, len) @@ -411,6 +453,27 @@ size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) if (WARN_ON_ONCE(!i->data_source)) return 0; + if (unlikely(iov_iter_is_iterlist(i))) { + size_t copied = 0; + + while (bytes && i->count) { + size_t part = min(bytes, i->iterlist->count), n; + + if (part > 0) + n = _copy_from_iter_flushcache(addr, part, + i->iterlist); + addr += n; + copied += n; + bytes -= n; + i->count -= n; + if (n < part || !bytes) + break; + i->iterlist++; + i->nr_segs--; + } + return copied; + } + iterate_and_advance(i, bytes, base, len, off, __copy_from_user_flushcache(addr + off, base, len), memcpy_flushcache(addr + off, base, len) @@ -514,7 +577,31 @@ EXPORT_SYMBOL(iov_iter_zero); size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t bytes, struct iov_iter *i) { - char *kaddr = kmap_atomic(page), *p = kaddr + offset; + char *kaddr, *p; + + if (unlikely(iov_iter_is_iterlist(i))) { + size_t copied = 0; + + while (bytes && i->count) { + size_t part = min(bytes, i->iterlist->count), n; + + if (part > 0) + n = copy_page_from_iter_atomic(page, offset, part, + i->iterlist); + offset += n; + copied += n; + bytes -= n; + i->count -= n; + if (n < part || !bytes) + break; + i->iterlist++; + i->nr_segs--; + } + return copied; + } + + kaddr = kmap_atomic(page); + p = kaddr + offset; if (!page_copy_sane(page, offset, bytes)) { kunmap_atomic(kaddr); return 0; @@ -585,19 +672,49 @@ void iov_iter_advance(struct iov_iter *i, size_t size) iov_iter_bvec_advance(i, size); } else if (iov_iter_is_discard(i)) { i->count -= size; + }else if (iov_iter_is_iterlist(i)) { + i->count -= size; + for (;;) { + size_t part = min(size, i->iterlist->count); + + if (part > 0) + iov_iter_advance(i->iterlist, part); + size -= part; + if (!size) + break; + i->iterlist++; + i->nr_segs--; + } } } EXPORT_SYMBOL(iov_iter_advance); +static void iov_iter_revert_iterlist(struct iov_iter *i, size_t unroll) +{ + for (;;) { + size_t part = min(unroll, i->iterlist->orig_count - i->iterlist->count); + + if (part > 0) + iov_iter_revert(i->iterlist, part); + unroll -= part; + if (!unroll) + break; + i->iterlist--; + i->nr_segs++; + } +} + void iov_iter_revert(struct iov_iter *i, size_t unroll) { if (!unroll) return; - if (WARN_ON(unroll > MAX_RW_COUNT)) + if (WARN_ON(unroll > i->orig_count - i->count)) return; i->count += unroll; if (unlikely(iov_iter_is_discard(i))) return; + if (unlikely(iov_iter_is_iterlist(i))) + return iov_iter_revert_iterlist(i, unroll); if (unroll <= i->iov_offset) { i->iov_offset -= unroll; return; @@ -641,6 +758,8 @@ EXPORT_SYMBOL(iov_iter_revert); */ size_t iov_iter_single_seg_count(const struct iov_iter *i) { + if (iov_iter_is_iterlist(i)) + i = i->iterlist; if (i->nr_segs > 1) { if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) return min(i->count, i->iov->iov_len - i->iov_offset); @@ -662,7 +781,8 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, .kvec = kvec, .nr_segs = nr_segs, .iov_offset = 0, - .count = count + .count = count, + .orig_count = count, }; } EXPORT_SYMBOL(iov_iter_kvec); @@ -678,7 +798,8 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction, .bvec = bvec, .nr_segs = nr_segs, .iov_offset = 0, - .count = count + .count = count, + .orig_count = count, }; } EXPORT_SYMBOL(iov_iter_bvec); @@ -706,6 +827,7 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, .xarray = xarray, .xarray_start = start, .count = count, + .orig_count = count, .iov_offset = 0 }; } @@ -727,11 +849,47 @@ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count) .iter_type = ITER_DISCARD, .data_source = false, .count = count, + .orig_count = count, .iov_offset = 0 }; } EXPORT_SYMBOL(iov_iter_discard); +/** + * iov_iter_iterlist - Initialise an I/O iterator that is a list of iterators + * @iter: The iterator to initialise. + * @direction: The direction of the transfer. + * @iterlist: The list of iterators + * @nr_segs: The number of elements in the list + * @count: The size of the I/O buffer in bytes. + * + * Set up an I/O iterator that just discards everything that's written to it. + * It's only available as a source iterator (for WRITE), all the iterators in + * the list must be the same and none of them can be ITER_ITERLIST type. + */ +void iov_iter_iterlist(struct iov_iter *iter, unsigned int direction, + struct iov_iter *iterlist, unsigned long nr_segs, + size_t count) +{ + unsigned long i; + + BUG_ON(direction != WRITE); + for (i = 0; i < nr_segs; i++) { + BUG_ON(iterlist[i].iter_type == ITER_ITERLIST); + BUG_ON(!iterlist[i].data_source); + } + + *iter = (struct iov_iter){ + .iter_type = ITER_ITERLIST, + .data_source = true, + .count = count, + .orig_count = count, + .iterlist = iterlist, + .nr_segs = nr_segs, + }; +} +EXPORT_SYMBOL(iov_iter_iterlist); + static bool iov_iter_aligned_iovec(const struct iov_iter *i, unsigned addr_mask, unsigned len_mask) { @@ -879,6 +1037,15 @@ unsigned long iov_iter_alignment(const struct iov_iter *i) if (iov_iter_is_xarray(i)) return (i->xarray_start + i->iov_offset) | i->count; + if (iov_iter_is_iterlist(i)) { + unsigned long align = 0; + unsigned int j; + + for (j = 0; j < i->nr_segs; j++) + align |= iov_iter_alignment(&i->iterlist[j]); + return align; + } + return 0; } EXPORT_SYMBOL(iov_iter_alignment); @@ -1078,6 +1245,18 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, } if (iov_iter_is_xarray(i)) return iter_xarray_get_pages(i, pages, maxsize, maxpages, start); + if (iov_iter_is_iterlist(i)) { + ssize_t size; + + while (!i->iterlist->count) { + i->iterlist++; + i->nr_segs--; + } + size = __iov_iter_get_pages_alloc(i->iterlist, pages, maxsize, maxpages, + start, extraction_flags); + i->count -= size; + return size; + } return -EFAULT; } @@ -1126,6 +1305,31 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); +static size_t csum_and_copy_from_iterlist(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + size_t copied = 0, n; + + while (i->count && i->nr_segs) { + struct iov_iter *j = i->iterlist; + + if (j->count == 0) { + i->iterlist++; + i->nr_segs--; + continue; + } + + n = csum_and_copy_from_iter(addr, bytes - copied, csum, j); + addr += n; + copied += n; + i->count -= n; + if (n == 0) + break; + } + + return copied; +} + size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { @@ -1133,6 +1337,8 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, sum = *csum; if (WARN_ON_ONCE(!i->data_source)) return 0; + if (iov_iter_is_iterlist(i)) + return csum_and_copy_from_iterlist(addr, bytes, csum, i); iterate_and_advance(i, bytes, base, len, off, ({ next = csum_and_copy_from_user(base, addr + off, len); @@ -1236,6 +1442,21 @@ static int bvec_npages(const struct iov_iter *i, int maxpages) return npages; } +static int iterlist_npages(const struct iov_iter *i, int maxpages) +{ + ssize_t size = i->count; + const struct iov_iter *p; + int npages = 0; + + for (p = i->iterlist; size; p++) { + size -= p->count; + npages += iov_iter_npages(p, maxpages - npages); + if (unlikely(npages >= maxpages)) + return maxpages; + } + return npages; +} + int iov_iter_npages(const struct iov_iter *i, int maxpages) { if (unlikely(!i->count)) @@ -1255,6 +1476,8 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages) int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE); return min(npages, maxpages); } + if (iov_iter_is_iterlist(i)) + return iterlist_npages(i, maxpages); return 0; } EXPORT_SYMBOL(iov_iter_npages); @@ -1266,11 +1489,14 @@ const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) return new->bvec = kmemdup(new->bvec, new->nr_segs * sizeof(struct bio_vec), flags); - else if (iov_iter_is_kvec(new) || iter_is_iovec(new)) + if (iov_iter_is_kvec(new) || iter_is_iovec(new)) /* iovec and kvec have identical layout */ return new->iov = kmemdup(new->iov, new->nr_segs * sizeof(struct iovec), flags); + if (WARN_ON_ONCE(iov_iter_is_iterlist(old))) + /* Don't allow dup'ing of iterlist as the cleanup is complicated */ + return NULL; return NULL; } EXPORT_SYMBOL(dup_iter); @@ -1759,6 +1985,22 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i, return iov_iter_extract_xarray_pages(i, pages, maxsize, maxpages, extraction_flags, offset0); + if (iov_iter_is_iterlist(i)) { + ssize_t size; + + while (i->nr_segs && !i->iterlist->count) { + i->iterlist++; + i->nr_segs--; + } + if (!i->nr_segs) { + WARN_ON_ONCE(i->count); + return 0; + } + size = iov_iter_extract_pages(i->iterlist, pages, maxsize, maxpages, + extraction_flags, offset0); + i->count -= size; + return size; + } return -EFAULT; } EXPORT_SYMBOL_GPL(iov_iter_extract_pages); From patchwork Wed Mar 29 14:13:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCD97C74A5B for ; Wed, 29 Mar 2023 14:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D0CB6B0078; Wed, 29 Mar 2023 10:14:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6822C6B007B; Wed, 29 Mar 2023 10:14:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5495E6B007D; Wed, 29 Mar 2023 10:14:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 452D96B0078 for ; Wed, 29 Mar 2023 10:14:19 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0C14C80D53 for ; Wed, 29 Mar 2023 14:14:19 +0000 (UTC) X-FDA: 80622130638.21.18F5CDC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 2652CC0019 for ; Wed, 29 Mar 2023 14:14:16 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYRCqrFD; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jo7xD6cW+5TTIp8oqR2hYvnkWHwWe/N8CbruF8cE9GA=; b=sXYklq6t6KjeY+64oSjE8roKnZIBeon3X4W+Xsl4QpLb+U8uCh/Cyeu5ARFz58V+0NheNJ K7tt+e6FoHe9ZXD7cvsncrrhrJQmd44GMlPNqgX/B5Z8WuhCEQjkINPTmp5xIvyn6bFNCV J9aktwbkgCDNm7k+8cqb/1lrHgq11oY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYRCqrFD; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099257; a=rsa-sha256; cv=none; b=TXKd63gNtYkLdkPpJSc+xkbItmneJ9pMtN+7Wl1rJKZLv6RPYMf5bMwYS1hypp0UJLWm+K 1EM3pXM0D93cxLABzF1U12X2FurywxodIeyJiefUDq7R32F2ppaQ7WRa3ULcq6yovy0j33 byhvfRwSg1fF/cr6+z/kc9+wVUqeDc4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099256; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jo7xD6cW+5TTIp8oqR2hYvnkWHwWe/N8CbruF8cE9GA=; b=TYRCqrFD85cZurfnlOWRFWCh2DTKGtvYgs2/9KjlfwOFT7cT5O8YT7+idKT0B1X1xyinD0 hsZ+v/yhcIJM2wN5LrKPuCZl9fRSUA7uR50f28hE5AXdpbaVa2p7PaSh+kPVilig6cvqgh 0tUPO+j8enubSaZjutRtAOcXBvaeXfs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-228-hXJAadh0MrOcoI5-BRrEXg-1; Wed, 29 Mar 2023 10:14:11 -0400 X-MC-Unique: hXJAadh0MrOcoI5-BRrEXg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 42E2B185A78B; Wed, 29 Mar 2023 14:14:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4BD8D2166B33; Wed, 29 Mar 2023 14:14:08 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [RFC PATCH v2 04/48] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Wed, 29 Mar 2023 15:13:10 +0100 Message-Id: <20230329141354.516864-5-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: fcptmfcjg8zrwwupfi86co5bfeucxnfe X-Rspamd-Queue-Id: 2652CC0019 X-HE-Tag: 1680099256-673745 X-HE-Meta: U2FsdGVkX1+jcSe8kI/Y59kH3Ux8H5mxDmRQiN2qAbp8iRPtlElG8pzONIXxDzop5kHUGclGMYn4iU/9pFNw8RltfYEG9rOyo3y+UUYAkgXJVSzilYhzg2xlR3uo4QLexFAzPln64WRRuq/er/krbN9056fEhlhBWP5gOg7YG659Zn24B7Biv/ZqiAYcVfIhHkSqIavgjXMhQuJmqg3Jf6SQTd7vMrjv9j9pXkNduwbBTfbd4+o/k3wLQWSy9p9rXDG1Du2L/6MTuQMWDuDb6KNSH/4niVTT/ztHyM9d9+4cMHyNJrJbqntUoY6itp7JXCsjh5sxOzFRnnXlH1mndFaEZldTqNuy0HqymAhxSlAclpToqoNrMMHzv+wadwbxH6vBdIxafotOCeXME6t9K+UjTyYKdRK2ASCluPzAB2aSetXKInCnAEMbZF6t5P7jDEvcDhDqiel3c9pHkn2pO3ivVCvgS7I+Ky7B1kEA9h7kqML3XZLpkj1v/zRLjnIbVQUYlMQNyFB4Rce3YMj2G2xa3Hw59ztBOh15glaCxwVRuNSkkH5cmMSoipMv8G8ohpuUHdLiBbCaKLSeJBka4rmn0mFQtRFhrvcy/buJfp6FDVD0Ay9AtA3rsQjcwZmjQQ4u9O/97YBF5iyPbndLWjuIidV16keIk84NzNjPhf3/gKs2wvlwFyn6DCnTO5Vvc9uDxQyO88TzRS8DCiV8zfspEbqJvzdMHIcMWI5GqMUgQi4pN6IIAjKN1LjS33OvhlqbqCNZzSLFPMSroZyTMa0bithMjdeUXXzoPDq52HAaud0GOAEGHA+ATzfxRBnJGSzcHzj1AIBwECBTCxB7onAn5fpAdgD/i80LubqZjtWJ1iAtQ3PkAqsiUWd16RasNRtIPCAfJistbQvgpKSIROaz9xApm+iF+JfBdZ6RZVhyNWIquZXLmLdYCOCBvPJsTCxUKPhura3Ak+4Ha7C sCBoFrQz 9ag4o0eFcp6Wz2yMLJlneDd8qhQeBadYKJ6ZGSadsx4fzLeMmFPxprgfsZu2iQGDhFsW5RoRUNJU7TZ/ZplK+O53/OYQ72QEPFNaIQqEoc9ETa2NdgyyMhVGRQD9AWkiXu7zsY5wh1JCM6vmA2bFWm2Dik+K+DqxiJycbBkka/2hd4aR20TutMk3KPq8zw4Kw53/SZxOFvdanRSjdMpm4Xc2Yrq0aXk7et0zkB+8lYqb0bowHJd8nfHg9uyla2tVzaAnWtRp6hBV/mkPbx3bQrn8iOiOfYOJYdW9/1AmNoDdWNez1pCuiIKn4Tkb1hxA3rVxkg9ouedEATKgB3q0UAQ92XTLontSY6T4JdsZZ7wgGDm0hUO0ZeHEvroMVii1VCMTAGKZymzI88lEGBwOQq3rJ3mhSnMPnnq3rEruE2H6px1W7OpVTIZTcxZvqDmEAcsE9YWDH0NkRO8VsWKmLtc03mFrQxzbcB6sEAQUGzcQ6WZo8n10NSEs7ZYrxjeiSQEZ8ZnkE0al7/9WYPJA8U0kYtw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg and recvmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/socket.h | 3 +++ net/socket.c | 7 +++++++ 2 files changed, 10 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..c2fa0f800999 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg, recvmsg, sendmmsg and recvmmsg syscalls */ +#define MSG_INTERNAL_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/net/socket.c b/net/socket.c index 6bae8ce7059e..dfb912bbed62 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2139,6 +2139,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2192,6 +2193,7 @@ int __sys_recvfrom(int fd, void __user *ubuf, size_t size, unsigned int flags, if (!sock) goto out; + flags &= ~MSG_INTERNAL_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; err = sock_recvmsg(sock, &msg, flags); @@ -2579,6 +2581,7 @@ long __sys_sendmsg(int fd, struct user_msghdr __user *msg, unsigned int flags, if (forbid_cmsg_compat && (flags & MSG_CMSG_COMPAT)) return -EINVAL; + flags &= ~MSG_INTERNAL_FLAGS; sock = sockfd_lookup_light(fd, &err, &fput_needed); if (!sock) @@ -2627,6 +2630,7 @@ int __sys_sendmmsg(int fd, struct mmsghdr __user *mmsg, unsigned int vlen, entry = mmsg; compat_entry = (struct compat_mmsghdr __user *)mmsg; err = 0; + flags &= ~MSG_INTERNAL_FLAGS; flags |= MSG_BATCH; while (datagrams < vlen) { @@ -2775,6 +2779,7 @@ long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, struct user_msghdr __user *umsg, struct sockaddr __user *uaddr, unsigned int flags) { + flags &= ~MSG_INTERNAL_FLAGS; return ____sys_recvmsg(sock, msg, umsg, uaddr, flags, 0); } @@ -2787,6 +2792,7 @@ long __sys_recvmsg(int fd, struct user_msghdr __user *msg, unsigned int flags, if (forbid_cmsg_compat && (flags & MSG_CMSG_COMPAT)) return -EINVAL; + flags &= ~MSG_INTERNAL_FLAGS; sock = sockfd_lookup_light(fd, &err, &fput_needed); if (!sock) @@ -2839,6 +2845,7 @@ static int do_recvmmsg(int fd, struct mmsghdr __user *mmsg, goto out_put; } } + flags &= ~MSG_INTERNAL_FLAGS; entry = mmsg; compat_entry = (struct compat_mmsghdr __user *)mmsg; From patchwork Wed Mar 29 14:13:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 063AAC74A5B for ; Wed, 29 Mar 2023 14:14:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 992996B007E; Wed, 29 Mar 2023 10:14:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F2EA6B0080; Wed, 29 Mar 2023 10:14:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F7696B0081; Wed, 29 Mar 2023 10:14:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 59BB66B007E for ; Wed, 29 Mar 2023 10:14:24 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0C77FAC2F3 for ; Wed, 29 Mar 2023 14:14:24 +0000 (UTC) X-FDA: 80622130848.27.A289E0A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 1FFD210001F for ; Wed, 29 Mar 2023 14:14:20 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="hCC5qfW/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zHarMfhYqdBKP5ReoH3//xqNxFuro7P2+Q66shvU368=; b=dpWXlQ/fOVpZ5DD+SMChV4/q18K0e9lBA9UeVDItnfx+KJZvMwn/ALUCNBcvSXUjJ+wQCR wcBWqJRug/WSYd/Eb5fGXf9af3O/4wMuT8lv6cSXqCCndi0H2dw/Qjy976ZTzlPhSRBQo9 fMSKsHGDMGEkmvVhS5+e3gjSyeij74E= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="hCC5qfW/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099261; a=rsa-sha256; cv=none; b=k1Mpzn8jQ7zuMwdbz02CqsO+QuAj5VhKWnZxQ/dQUQdR3r/MLnwbI+T2AD5NRaCxc7sTqg ISGIJA0X5p4p6AMn/xLmRH5xEu5w8JapSkQZ12NjZPCSt9QvYGsqKAhQ26n/mJz6NrhgM/ dN9QQrEaB/+hTA2ZPD6Ej1JDKm2s1iE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zHarMfhYqdBKP5ReoH3//xqNxFuro7P2+Q66shvU368=; b=hCC5qfW/rLHBKkSWPegSlZt5rSHyF7fHZ+kMKSX0mp43LSuZvDc6MhUTxq2NEGs2+AzKlh 9acwemvQTOZyeAFy7M/3oeTaX0+NIslB1kHV8BEeBbQRuuvlgw0nRIQV8Je1ekYVvTQmAo MZT16mMvOihpfNR8RGgvjY3Dy84JvaU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-338-kG3Hk5VCOq-MrLBrXf1dJQ-1; Wed, 29 Mar 2023 10:14:14 -0400 X-MC-Unique: kG3Hk5VCOq-MrLBrXf1dJQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2DA6A3814592; Wed, 29 Mar 2023 14:14:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id D3728492C3E; Wed, 29 Mar 2023 14:14:10 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [RFC PATCH v2 05/48] mm: Move the page fragment allocator from page_alloc.c into its own file Date: Wed, 29 Mar 2023 15:13:11 +0100 Message-Id: <20230329141354.516864-6-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1FFD210001F X-Stat-Signature: pmaaq5t6jfpx3uc1qnaxwdhbsjkidon3 X-HE-Tag: 1680099260-115710 X-HE-Meta: U2FsdGVkX18+IA6Em4SG5FricxMC9B62uikBcyNjkbqBNA9z9ynW5qplEU+wL6Wve6EBuIajaA1+QMYqPOFNq4XGXixEGjbLmIJWcV0poTDXXK1NZrJaObrGGXo83pF2+YojQwOJL5qIBVS+J5wzWelmCS+F+OjceiEnSBddjEfCilmo3ueUAzncgyZKp7uy5FEEXpdrSWRS4LatNOjrbTtgT8FyxahQ5c5CJgWIgAPqLdNvdBdBCqJCyN4FStmjpDLFeQHBVxoYrErklh3VwJmONXJXy5Iv2wYy+jyCXYHjkElyf1Q1VwVe5t6LjdzU9YcXY7ntgs1Dxj99AnY80AhKC+qo0z39gpjhdSMEKaMSjmH7FSF9T3jeLMQldvSdWNYmylzkh9uAP4ggnG4hZRFIopi+JVL0Jl22iibHcAQK1eXcV9QsU+jYTrRSPJBHbETWpNQWe1vv6U5Vwd2J5XMj50odDuSTs0x6Ewb8O8BjTqsGVUW1B/BCjSEvKvjm2/eC+1r+/Ax9EaWiTIS1IHQkZTSaGHNeEqK91/dEDvcsiEP+O++KOCIjpVko7CSB5Nm6zhvTJ+3hfDEC/Wu/QMh6wW+sdvynQgOHk3Vmaj0u+NdxUejBRnYGdclzBb5219IAMNQ1QQPhre8UMeKhG54KVD1Yp10OMwCrG0LmdJzn5wXWUBuqH7hxJhvJFBgRmjxsM04lLGQhK6RL2f3ncs2QGsEOpzIZQ8wDXADw5jLDVSgJP0ubl6mhnPOvYATTm17zaAE/hw1lfdCTlcFlYnHmsXSi0hUxfi3ldp0n9tBIHYeJOkrnPvkEWrVOtR8NHOmSYXsS2fexrdbn0+gLWU53CZR9yedUK/iyse5qSz7JZ2RAmZNbNmoYdiZKpfTv89QijJk2RbZgFEfWclM8yQTaMjkgiBFqhWhMdb8yYk9bprOfzp0SlJoZvTNgl0Vhtpop0vDvsKTZmSpncjF GNPq/fCu BgeGUdWvwfVhdCoMdshFy024te40RygzPp3UET6f8+osz23LIOu/Dz4va5dl6uPAmZJds/agf9hUMtcVquCpLdyozeSSYHynVAksagbMao+tUzDUq53KgeaBHkXWz16VI7vV5smdUm3vSJkyJu8OWMo7x7qt0AnpSxCEwCAwmgWrefvkPCL/q0q2wsverfmZNFfv5hfDAM8SkaESDh/5y7DECzV9YbEj49Z+zd1y5BfzxyQG61sqF2/VTqlMVBKGVOPw7KlK7WwqdXLh9cSQLUTmhb6Ln4fPVN3+hUR+8XwzSkcxoG6aBCm8fYOoY2RaWvXd0AbH6k//IgdSkqz906lLnEMat+zmtFmn6RLI/sQD+MOnwbk8tJ1zc4xNrFfyktPMlF5arcT/h+UsDItmL9FqlznVZobcITsdPlg9X53t7dxcVuPN8+c8wdwu897F3AmHRNO7vOSzHEAyyDR+9NBUREXbVhnk5jKhmoAXBkL3gApNmCsaBsJ3zUa053TRC6ZJosZWYPxMplJENQFaivhiOOg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the page fragment allocator from page_alloc.c into its own file preparatory to changing it. Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- mm/Makefile | 2 +- mm/page_alloc.c | 126 ----------------------------------------- mm/page_frag_alloc.c | 131 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 127 deletions(-) create mode 100644 mm/page_frag_alloc.c diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..4e6dc12b4cbd 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -52,7 +52,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ - compaction.o \ + compaction.o page_frag_alloc.o \ interval_tree.o list_lru.o workingset.o \ debug.o gup.o mmap_lock.o $(mmu-y) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ac1fc986af44..c08847308907 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5694,132 +5694,6 @@ void free_pages(unsigned long addr, unsigned int order) EXPORT_SYMBOL(free_pages); -/* - * Page Fragment: - * An arbitrary-length arbitrary-offset area of memory which resides - * within a 0 or higher order page. Multiple fragments within that page - * are individually refcounted, in the page's reference counter. - * - * The page_frag functions below provide a simple allocation framework for - * page fragments. This is used by the network stack and network device - * drivers to provide a backing region of memory for use as either an - * sk_buff->head, or to be used in the "frags" portion of skb_shared_info. - */ -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) -{ - struct page *page = NULL; - gfp_t gfp = gfp_mask; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; - - return page; -} - -void __page_frag_cache_drain(struct page *page, unsigned int count) -{ - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - - if (page_ref_sub_and_test(page, count)) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(__page_frag_cache_drain); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) -{ - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; - - if (unlikely(!nc->va)) { -refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) - return NULL; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); - - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; - } - - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (!page_ref_sub_and_test(page, nc->pagecnt_bias)) - goto refill; - - if (unlikely(nc->pfmemalloc)) { - free_the_page(page, compound_order(page)); - goto refill; - } - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); - - /* reset page count bias and offset to start of new frag */ - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - return NULL; - } - } - - nc->pagecnt_bias--; - offset &= align_mask; - nc->offset = offset; - - return nc->va + offset; -} -EXPORT_SYMBOL(page_frag_alloc_align); - -/* - * Frees a page fragment allocated out of either a compound or order 0 page. - */ -void page_frag_free(void *addr) -{ - struct page *page = virt_to_head_page(addr); - - if (unlikely(put_page_testzero(page))) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(page_frag_free); - static void *make_alloc_exact(unsigned long addr, unsigned int order, size_t size) { diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c new file mode 100644 index 000000000000..bee95824ef8f --- /dev/null +++ b/mm/page_frag_alloc.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Page fragment allocator + * + * Page Fragment: + * An arbitrary-length arbitrary-offset area of memory which resides within a + * 0 or higher order page. Multiple fragments within that page are + * individually refcounted, in the page's reference counter. + * + * The page_frag functions provide a simple allocation framework for page + * fragments. This is used by the network stack and network device drivers to + * provide a backing region of memory for use as either an sk_buff->head, or to + * be used in the "frags" portion of skb_shared_info. + */ + +#include +#include +#include + +static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) +{ + struct page *page = NULL; + gfp_t gfp = gfp_mask; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; +#endif + if (unlikely(!page)) + page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + + nc->va = page ? page_address(page) : NULL; + + return page; +} + +void __page_frag_cache_drain(struct page *page, unsigned int count) +{ + VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + + if (page_ref_sub_and_test(page, count - 1)) + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(__page_frag_cache_drain); + +void *page_frag_alloc_align(struct page_frag_cache *nc, + unsigned int fragsz, gfp_t gfp_mask, + unsigned int align_mask) +{ + unsigned int size = PAGE_SIZE; + struct page *page; + int offset; + + if (unlikely(!nc->va)) { +refill: + page = __page_frag_cache_refill(nc, gfp_mask); + if (!page) + return NULL; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* Even if we own the page, we do not use atomic_set(). + * This would break get_page_unless_zero() users. + */ + page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + + /* reset page count bias and offset to start of new frag */ + nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + nc->offset = size; + } + + offset = nc->offset - fragsz; + if (unlikely(offset < 0)) { + page = virt_to_page(nc->va); + + if (page_ref_count(page) != nc->pagecnt_bias) + goto refill; + if (unlikely(nc->pfmemalloc)) { + page_ref_sub(page, nc->pagecnt_bias - 1); + __free_pages(page, compound_order(page)); + goto refill; + } + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* OK, page count is 0, we can safely set it */ + set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + + /* reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = size - fragsz; + if (unlikely(offset < 0)) { + /* + * The caller is trying to allocate a fragment + * with fragsz > PAGE_SIZE but the cache isn't big + * enough to satisfy the request, this may + * happen in low memory conditions. + * We don't release the cache page because + * it could make memory pressure worse + * so we simply return NULL here. + */ + return NULL; + } + } + + nc->pagecnt_bias--; + offset &= align_mask; + nc->offset = offset; + + return nc->va + offset; +} +EXPORT_SYMBOL(page_frag_alloc_align); + +/* + * Frees a page fragment allocated out of either a compound or order 0 page. + */ +void page_frag_free(void *addr) +{ + struct page *page = virt_to_head_page(addr); + + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(page_frag_free); From patchwork Wed Mar 29 14:13:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BE73C761AF for ; Wed, 29 Mar 2023 14:14:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CED2E6B007D; Wed, 29 Mar 2023 10:14:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C77F46B007E; Wed, 29 Mar 2023 10:14:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC7C96B0080; Wed, 29 Mar 2023 10:14:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9E9E86B007D for ; Wed, 29 Mar 2023 10:14:21 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 720CC140D11 for ; Wed, 29 Mar 2023 14:14:21 +0000 (UTC) X-FDA: 80622130722.24.7AC346F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id A2721A0016 for ; Wed, 29 Mar 2023 14:14:19 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MzvbyZ1I; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099259; a=rsa-sha256; cv=none; b=r0K7VN+43gvrBzUxuw3zr8YNQ0dG+2qlmxDHvT7BTPRWMjlXg04dnR+NtZ66LtStaQlsmN lBqpht2KAKl99wnBtXA16LhhJLSscxpJA9tv5EDzgA6uaIdArU8s9MTbk1wEfjVwicLzgv XZv5TMOmYY/CSgIZclBBTHTBwm/TCWc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MzvbyZ1I; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w9vYmidIh5WhninpbPh1DpHK4Q4l9z+4/ehUwmRVCvg=; b=TyEoDMHw2btZfZVNlP3X3S7z5Teuzj46jxijeEIsIg/htmmfes+364VPAb68x5HnVGWaPb ze912djSzVTkUFKio9iM0BgEUiLKmD8R2kOrLtVF6RZAH1yCoSHjA6HvUe6iYlB6+eq7Yk XDCFsiLbS+7meG/ayoZTDxi9NleJT0A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w9vYmidIh5WhninpbPh1DpHK4Q4l9z+4/ehUwmRVCvg=; b=MzvbyZ1IyLe5dTKmk3AJr5BnY6XvhByZaOOlXP6WqhQWCSXAVmdfcsieqOHwgfDw8/cA14 M97KjuBaIiZcdkXbHT/FQ8sXXUqK96J8WVwxHhiY9O3REk9xk4drTFkAc0SDmUO1xvHimi 3EZIX5Om0hCyXy+LbGisFmtwPzvh4mk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-187-vEQ2qQN_Nv6Frz5PzfcuyA-1; Wed, 29 Mar 2023 10:14:16 -0400 X-MC-Unique: vEQ2qQN_Nv6Frz5PzfcuyA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C7D96801779; Wed, 29 Mar 2023 14:14:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id E3D6E4020C83; Wed, 29 Mar 2023 14:14:13 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 06/48] mm: Make the page_frag_cache allocator use multipage folios Date: Wed, 29 Mar 2023 15:13:12 +0100 Message-Id: <20230329141354.516864-7-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Queue-Id: A2721A0016 X-Rspamd-Server: rspam01 X-Stat-Signature: g8tsj57bphmnyx3ydyt13xa68eebbrct X-HE-Tag: 1680099259-868071 X-HE-Meta: U2FsdGVkX1/rpF2FgofPsql2fuA90RmadhFxEM/7u5hCnR1ORAmF0XK/ZAZbEFUcCipN7S4rguFw/q/U8erSPZIbzCqhsk0CUMteUwmTEmGVyQy7t4XzTbaSsLTXIytHYJlOzC8puAna+Bob+UTicRXGgF5YRCLWh31qlzz3oXJEvenJ/0IziwfQ4q11p7Ri8Ka+XbV/dBdNLYAj7AllT9ezSs+leSezm5RKYt72v3DPmH5uIAHRv5IBx2bKRzbDV3N/2n62dDYzw77TnOF6iBpmRHFNSu0mOI3o1fsPV8ES3L8TAT8U6pR+VYjk+w1+ik1NJAjKTwgMfROEpgh9qXW9eVd+2LmqBKq9StGImINkHLhhKgx7y6RPDrgoApVUk5bwyS/QBJVZcVgfQ/PoHNm0NKFVBJsBSHAtWhc58xgLv2MSic8rcFomwd+HZ6mCWKRgLJxFB3rluxr7LGdyqiBylWgMzEyUEts7azRCx7KpWXTWwQNOe9PbgrksUE+6QfVuYVVMcjg8ztI6wKk4tCetJ8Xx/ZSZGjxDb+U/b57EkRwVD8vqVp9zRfapb8ZneUvxD9Ua/bd+i94rm8CbaHWbL5ssFpzQGDMvFaejvsfgZWPSH6v/yYgD2cCVr5tevI6Cz4oIoM6pnh87Tk4VJC9Oe/00K21PfLgvjzjNtDP3LDRKKXpmiJX0Bj5r3/NT1i7HyTA0hsWa01e8KmdDtQwrHxGPNe2EZllk+2XaL5YBs95Ld7ogj+JRhZOu/Vfv7tsGZqBhEuBDeYeRXMPv052uMpn8/aExgZBs69YsziY75d+Gl85IgQjCHoTEPrgEi1zl/h44TCD4mKisrieXTwZJYwBMzmMfKmtTVGinl0ihAngAAJHDAf/RLNEiaMIN3OQ2PoChenwbumUPCKiDwYUCHQXgX2l3W964rH3O+h+WJksBGrxJh8G6lcfv9hKz7rMDk5j1prlP5LYX9Q3 uyjDPr0T 2lNgVcEhHUuNRnZHscCpqAR85AlMwaA4SvOXkQ6uT1WXBRvi16RnZyDxjmIPhy6F0zPhLHjaBYoL+9jCRCUmyTYR94OvJNNDhdrsFRVHQCl9j4DmVfxEicxxBrpOUhljv+Gcs5DBmMj6dMqZx3cuDoMifmUNNaMMxTBAXt5n3B29B2sxr8ZpWDZZ997GPozOOtDPOkmijInX/sVctPM+aEyAIyq1HwDCGqm4Ty7x8BHDBgceeOFJD5U9l/c/UXU15qDxa17QvbdbAw0JgllY+7+QpJH4JsrJFr98S7w5uneBKBDWmrb8jw8t4jpt3NHx6WQVxLcu7PbrRpvdpuYjcFSOw8iOHpGLimeivIyx5NYX6vS7MTgmMu3yTfVIVP9YgR36pPJOLaP2rDMBdcL42w962R5lcW9iMwL+h6DoOBhFaeqqEGhkfmha8inAUrDjAsECj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change the page_frag_cache allocator to use multipage folios rather than groups of pages. This reduces page_frag_free to just a folio_put() or put_page(). Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/mm_types.h | 13 ++---- mm/page_frag_alloc.c | 88 +++++++++++++++++++--------------------- 2 files changed, 45 insertions(+), 56 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0722859c3647..49a70b3f44a9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) } struct page_frag_cache { - void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif + struct folio *folio; + unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; - bool pfmemalloc; + unsigned int pagecnt_bias; + bool pfmemalloc; }; typedef unsigned long vm_flags_t; diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index bee95824ef8f..c3792b68ce32 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,33 +16,34 @@ #include #include -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +/* + * Allocate a new folio for the frag cache. + */ +static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) { - struct page *page = NULL; + struct folio *folio = NULL; gfp_t gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; + gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); #endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; + if (unlikely(!folio)) + folio = folio_alloc(gfp, 0); - return page; + if (folio) + nc->folio = folio; + return folio; } void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + struct folio *folio = page_folio(page); - if (page_ref_sub_and_test(page, count - 1)) - __free_pages(page, compound_order(page)); + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + + folio_put_refs(folio, count); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -50,54 +51,47 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask) { - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; + struct folio *folio = nc->folio; + size_t offset; - if (unlikely(!nc->va)) { + if (unlikely(!folio)) { refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) + folio = page_frag_cache_refill(nc, gfp_mask); + if (!folio) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pfmemalloc = folio_is_pfmemalloc(folio); nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; + nc->offset = folio_size(folio); } - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (page_ref_count(page) != nc->pagecnt_bias) + offset = nc->offset; + if (unlikely(fragsz > offset)) { + /* Reuse the folio if everyone we gave it to has finished with it. */ + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { + nc->folio = NULL; goto refill; + } + if (unlikely(nc->pfmemalloc)) { - page_ref_sub(page, nc->pagecnt_bias - 1); - __free_pages(page, compound_order(page)); + __folio_put(folio); + nc->folio = NULL; goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { + offset = folio_size(folio); + if (unlikely(fragsz > offset)) { /* * The caller is trying to allocate a fragment * with fragsz > PAGE_SIZE but the cache isn't big @@ -107,15 +101,17 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, * it could make memory pressure worse * so we simply return NULL here. */ + nc->offset = offset; return NULL; } } nc->pagecnt_bias--; + offset -= fragsz; offset &= align_mask; nc->offset = offset; - return nc->va + offset; + return folio_address(folio) + offset; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -124,8 +120,6 @@ EXPORT_SYMBOL(page_frag_alloc_align); */ void page_frag_free(void *addr) { - struct page *page = virt_to_head_page(addr); - - __free_pages(page, compound_order(page)); + folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); From patchwork Wed Mar 29 14:13:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B8A3C761AF for ; Wed, 29 Mar 2023 14:14:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06BBA6B0083; Wed, 29 Mar 2023 10:14:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F34A96B0082; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DACCC6B0083; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C40986B0081 for ; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8FDCF1A0633 for ; Wed, 29 Mar 2023 14:14:31 +0000 (UTC) X-FDA: 80622131142.29.CFC0632 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id AA9DA40004 for ; Wed, 29 Mar 2023 14:14:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P9VJwGJR; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099268; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/nU0M+T1DCbw3wZvEUGAhwFOIfw7rmUPAJ5895bBp2k=; b=oM6aZeGaBhC9mKrOFT/dB1mZv/sSVvwlgU1lmT4pCaZTIHFJVsA72Fx8qCuzoDx2KqlB5a VF2S93o6HxrbjgAxI56BtzmAa8shSRLjTTymmviWHNISyt2KRTy4cMzXWKh1LcmV89xgHn sDfNm2pgR7x/1v3Ayz1wDNaqsY/bonc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P9VJwGJR; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099268; a=rsa-sha256; cv=none; b=lF+f/gtGdFEr8OqWidoxCFWjf7VA0knzdU9XYbRy6u05AqDLdApPXfpfrOX/z+6P0KOcX+ BScor7HVETCWqgoi0Fb2abFYNBb5SZA3NgZSp9CK1ZIT2jMo1Q6BuqAYg/9RvQ4hCyPFOd rg2uH187A/ENIK8z1+FcGNa7hoEOr7c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nU0M+T1DCbw3wZvEUGAhwFOIfw7rmUPAJ5895bBp2k=; b=P9VJwGJRrobvkYsYgyUZ7LoUTDirJNWmq30VAsO2Ygi67Wo1M9s2HA1b8pCE5n6qEJER4P mG75rZWZK8UNuKLj58+zUBs2wQ+VEh3rbrNfNKiuBRy7Jn8xfrxTifYBixzYHP+VY6ofD4 fJknOk8BESzUzVDRfoFGBW+n5G2y7JI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-98-MxenIVgtPHWZWtbUZgH7HA-1; Wed, 29 Mar 2023 10:14:22 -0400 X-MC-Unique: MxenIVgtPHWZWtbUZgH7HA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C8DAB381459C; Wed, 29 Mar 2023 14:14:19 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 62C781121330; Wed, 29 Mar 2023 14:14:16 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lorenzo Bianconi , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org, linux-mediatek@lists.infradead.org Subject: [RFC PATCH v2 07/48] mm: Make the page_frag_cache allocator use per-cpu Date: Wed, 29 Mar 2023 15:13:13 +0100 Message-Id: <20230329141354.516864-8-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AA9DA40004 X-Stat-Signature: xcdcpm7taijaosisss6nfwytg441ektq X-Rspam-User: X-HE-Tag: 1680099268-584959 X-HE-Meta: U2FsdGVkX18Yp+wgj9LdLNWuXvlTQw6MIWGr2oxP76Xy0XA4byrbyvRg++pfWrc90Fz0lDk9rKIc4BU1MKQDXl30O1/bNad2BfSc6KIEsaf3UIdmlR3t8D4n6BfVDeBrAQy4uwa/njSO6rTuJ+aWCPGAoAlm37VZF/JZmmWW4XDt0ut8nft16aNZJAEOPSNL5/Z3JZ4z+x4CQh0DyN2YVksa/xmc+kHexGX6Y6opgoKRjm3ruU9Xk/o0ue++qZ27IUACiTB14rwDPfKnAx/L4U4SKH+iUKJP6U0OPWQClWsz3E1AjLcw5FlGHuuM4VZr6zL7FaZHB90uB4wFiUrAnXlU2+QJgsqGvtYNJFYCmls3oyBnzEaLpp4aURAkWX+qtD/6alHEGjg/0xZIlddZfyi6jo9XlJFgKlj9ExPR58g+zCDJVIM3j9RdTM5meRqzUJA+hTbhIgLprOHKbAaoJY4k02qcLXNlpylc7JsLqAC6HRGtCJ9MklgKvr0AIi+ySpGiecs2Ue6qxRl+vg7IZsEMnDv77ezNTgzXS/zRQsZCtyAWfyTmJbAnu7T+l0jB6U1wMjrUBQPGeomsgBKBU08EiobIq5mlET91Hit76LkgzrqbGCLd+dXFBFcmDsase6GGrWwto0aXG+KC/h4Ycfpa2PyrC0s2VsTLu1UY0LA/y0CpXbmrWiapN48YlZJdNBtFp+5V6fgL4rD51oRu+roylGYD54CRbbXM3bXcxLZI7JBilRm0mQ6mQpaoqokLviU9ohJHHOAMqSkQimn8qulDDhw73106lIN3ZM2iosdFqRHq5awG3crJ309Ta0/PEUkRmcsxCZWh6oUKvSCdjuMdopWW2eBxPMKmw6Cjck8/aWgJ6iQGxNZr5RYoJuVKhgN8SAu9b9+fqJ+enwYwonioey1oWGZ0X1trdkONOvwkHIkzwFy4uNk9ICWuUnI3GEeFXDcCKy1DfQfztBG 2P3503s4 OU8yCOsrQGab4MbO0+rG53qM3Eb3QwG9tdQEhjzNhSqkpnM3Y0q0+lFeSUnWscbzi1sQRtClDAZYGx0LvLVz1jEPNw32KMhjPVV3Utckv9G+ltQ1UqzjUGZNKOHDgPrNSdzZb2pDQLYHShyNCJjonEfTmHIrHMYJeEpRBPP8WRwTaB6gWnry0jmKQftLjHtqo3W7Eaz+Gtt7CQ5N+ssontq28k3r0uGrOhSJlrK3Hs0Gt3wMt/XLoMU4ZQDXHAd5N9PtN1aGYo4vhM7p29VwnoD+nACNU5b2rurWLnYIGQvhyF39Ir8jQNUK24+VLNi8FiqMfN+nv6AlQmXgq4Ywcwk1GnR7mMkkaoM31bWJobcEXfOC3fRuBDuh3+abwh4bQ7NR790iN4Pz+oQzCap8oqrZ3LgbJMIXHokvTNdqiJN9PBxGrgGXKhbGp9BQ6Awj3OHj89V6/k8rWe+Iz4nz++vz+uudZJyDpv+8Kzn6jr9AZu1iGv3OWG9DN6yqr/+zPEje2tbbJLaGFHydrgD5cmmKnIIyTJ7dua22Vbl1a78e4lRU8s9OhCQOIoaof8z2pVKBsF4LV4SZ0FVk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator have a separate allocation bucket for each cpu to avoid racing. This means that no lock is required, other than preempt disablement, to allocate from it, though if a softirq wants to access it, then softirq disablement will need to be added. Make the NVMe and mediatek drivers pass in NULL to page_frag_cache() and use the default allocation buckets rather than defining their own. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Lorenzo Bianconi cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Keith Busch cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-nvme@lists.infradead.org cc: linux-mediatek@lists.infradead.org --- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 19 +-- drivers/net/ethernet/mediatek/mtk_wed_wo.h | 2 - drivers/nvme/host/tcp.c | 19 +-- drivers/nvme/target/tcp.c | 22 +-- include/linux/gfp.h | 17 +- mm/page_frag_alloc.c | 182 +++++++++++++++------ net/core/skbuff.c | 32 ++-- 7 files changed, 164 insertions(+), 129 deletions(-) diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index 69fba29055e9..859f34447f2f 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -143,7 +143,7 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q, dma_addr_t addr; void *buf; - buf = page_frag_alloc(&q->cache, q->buf_size, GFP_ATOMIC); + buf = page_frag_alloc(NULL, q->buf_size, GFP_ATOMIC); if (!buf) break; @@ -286,7 +286,6 @@ mtk_wed_wo_queue_free(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) static void mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; int i; for (i = 0; i < q->n_desc; i++) { @@ -297,20 +296,11 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(entry->buf); entry->buf = NULL; } - - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); } static void mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; - for (;;) { void *buf = mtk_wed_wo_dequeue(wo, q, NULL, true); @@ -319,13 +309,6 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); } static void diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.h b/drivers/net/ethernet/mediatek/mtk_wed_wo.h index dbcf42ce9173..6f940db67fb8 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.h +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.h @@ -210,8 +210,6 @@ struct mtk_wed_wo_queue_entry { struct mtk_wed_wo_queue { struct mtk_wed_wo_queue_regs regs; - struct page_frag_cache cache; - struct mtk_wed_wo_queue_desc *desc; dma_addr_t desc_dma; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 7723a4989524..fa32969b532f 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -147,8 +147,6 @@ struct nvme_tcp_queue { __le32 exp_ddgst; __le32 recv_ddgst; - struct page_frag_cache pf_cache; - void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); @@ -470,9 +468,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx]; u8 hdgst = nvme_tcp_hdgst_len(queue); - req->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + req->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!req->pdu) return -ENOMEM; @@ -1288,9 +1285,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) struct nvme_tcp_request *async = &ctrl->async_req; u8 hdgst = nvme_tcp_hdgst_len(queue); - async->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + async->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!async->pdu) return -ENOMEM; @@ -1300,7 +1296,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) { - struct page *page; struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; unsigned int noreclaim_flag; @@ -1311,12 +1306,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } - noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 66e8f9fd0ca7..d6cc557cc539 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -143,8 +143,6 @@ struct nvmet_tcp_queue { struct nvmet_tcp_cmd connect; - struct page_frag_cache pf_cache; - void (*data_ready)(struct sock *); void (*state_change)(struct sock *); void (*write_space)(struct sock *); @@ -1312,25 +1310,25 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue, c->queue = queue; c->req.port = queue->port->nport; - c->cmd_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->cmd_pdu = page_frag_alloc(NULL, sizeof(*c->cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->cmd_pdu) return -ENOMEM; c->req.cmd = &c->cmd_pdu->cmd; - c->rsp_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->rsp_pdu = page_frag_alloc(NULL, sizeof(*c->rsp_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->rsp_pdu) goto out_free_cmd; c->req.cqe = &c->rsp_pdu->cqe; - c->data_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->data_pdu = page_frag_alloc(NULL, sizeof(*c->data_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->data_pdu) goto out_free_rsp; - c->r2t_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->r2t_pdu = page_frag_alloc(NULL, sizeof(*c->r2t_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->r2t_pdu) goto out_free_data; @@ -1438,7 +1436,6 @@ static void nvmet_tcp_free_cmd_data_in_buffers(struct nvmet_tcp_queue *queue) static void nvmet_tcp_release_queue_work(struct work_struct *w) { - struct page *page; struct nvmet_tcp_queue *queue = container_of(w, struct nvmet_tcp_queue, release_work); @@ -1460,9 +1457,6 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65a78773dcca..b208ca315882 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -304,14 +304,17 @@ extern void free_pages(unsigned long addr, unsigned int order); struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); -extern void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask); - -static inline void *page_frag_alloc(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask) +extern void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, + unsigned long align_mask); +extern void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask); + +static inline void *page_frag_alloc(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp) { - return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); + return page_frag_alloc_align(frag_cache, fragsz, gfp, ULONG_MAX); } extern void page_frag_free(void *addr); diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index c3792b68ce32..7844398afe26 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,25 +16,23 @@ #include #include +static DEFINE_PER_CPU(struct page_frag_cache, page_frag_default_allocator); + /* * Allocate a new folio for the frag cache. */ -static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +static struct folio *page_frag_cache_refill(gfp_t gfp) { - struct folio *folio = NULL; - gfp_t gfp = gfp_mask; + struct folio *folio; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); + folio = folio_alloc(gfp | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC, + PAGE_FRAG_CACHE_MAX_ORDER); + if (folio) + return folio; #endif - if (unlikely(!folio)) - folio = folio_alloc(gfp, 0); - if (folio) - nc->folio = folio; - return folio; + return folio_alloc(gfp, 0); } void __page_frag_cache_drain(struct page *page, unsigned int count) @@ -47,41 +45,68 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) } EXPORT_SYMBOL(__page_frag_cache_drain); -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) +/** + * page_frag_alloc_align - Allocate some memory for use in zerocopy + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The size of the fragment desired + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + * + * Allocate some memory for use with zerocopy where protocol bits have to be + * mixed in with spliced/zerocopied data. Unlike memory allocated from the + * slab, this memory's lifetime is purely dependent on the folio's refcount. + * + * The way it works is that a folio is allocated and fragments are broken off + * sequentially and returned to the caller with a ref until the folio no longer + * has enough spare space - at which point the allocator's ref is dropped and a + * new folio is allocated. The folio remains in existence until the last ref + * held by, say, an sk_buff is discarded and then the page is returned to the + * page allocator. + * + * Returns a pointer to the memory on success and -ENOMEM on allocation + * failure. + * + * The allocated memory should be disposed of with folio_put(). + */ +void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, unsigned long align_mask) { - struct folio *folio = nc->folio; + struct page_frag_cache *nc; + struct folio *folio, *spare = NULL; size_t offset; + void *p; - if (unlikely(!folio)) { -refill: - folio = page_frag_cache_refill(nc, gfp_mask); - if (!folio) - return NULL; - - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + if (!frag_cache) + frag_cache = &page_frag_default_allocator; + if (WARN_ON_ONCE(fragsz == 0)) + fragsz = 1; + align_mask &= ~3UL; - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = folio_is_pfmemalloc(folio); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = folio_size(folio); + nc = get_cpu_ptr(frag_cache); +reload: + folio = nc->folio; + offset = nc->offset; +try_again: + + /* Make the allocation if there's sufficient space. */ + if (fragsz <= offset) { + nc->pagecnt_bias--; + offset = (offset - fragsz) & align_mask; + nc->offset = offset; + p = folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return p; } - offset = nc->offset; - if (unlikely(fragsz > offset)) { - /* Reuse the folio if everyone we gave it to has finished with it. */ - if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { - nc->folio = NULL; + /* Insufficient space - see if we can refurbish the current folio. */ + if (folio) { + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) goto refill; - } if (unlikely(nc->pfmemalloc)) { __folio_put(folio); - nc->folio = NULL; goto refill; } @@ -91,27 +116,56 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; offset = folio_size(folio); - if (unlikely(fragsz > offset)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - nc->offset = offset; + if (unlikely(fragsz > offset)) + goto frag_too_big; + goto try_again; + } + +refill: + if (!spare) { + nc->folio = NULL; + put_cpu_ptr(frag_cache); + + spare = page_frag_cache_refill(gfp); + if (!spare) return NULL; - } + + nc = get_cpu_ptr(frag_cache); + /* We may now be on a different cpu and/or someone else may + * have refilled it + */ + nc->pfmemalloc = folio_is_pfmemalloc(spare); + if (nc->folio) + goto reload; } - nc->pagecnt_bias--; - offset -= fragsz; - offset &= align_mask; + nc->folio = spare; + folio = spare; + spare = NULL; + + /* Even if we own the page, we do not use atomic_set(). This would + * break get_page_unless_zero() users. + */ + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + + /* Reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = folio_size(folio); + goto try_again; + +frag_too_big: + /* + * The caller is trying to allocate a fragment with fragsz > PAGE_SIZE + * but the cache isn't big enough to satisfy the request, this may + * happen in low memory conditions. We don't release the cache page + * because it could make memory pressure worse so we simply return NULL + * here. + */ nc->offset = offset; - - return folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return NULL; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -123,3 +177,25 @@ void page_frag_free(void *addr) folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); + +/** + * page_frag_memdup - Allocate a page fragment and duplicate some data into it + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The amount of memory to copy (maximum 1/2 page). + * @p: The source data to copy + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + */ +void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask) +{ + void *q; + + q = page_frag_alloc_align(frag_cache, fragsz, gfp, align_mask); + if (!q) + return q; + + return memcpy(q, p, fragsz); +} +EXPORT_SYMBOL(page_frag_memdup); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index eb7d33b41e71..0506e4cf1ed9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -222,13 +222,13 @@ static void *page_frag_alloc_1k(struct page_frag_1k *nc, gfp_t gfp_mask) #endif struct napi_alloc_cache { - struct page_frag_cache page; struct page_frag_1k page_small; unsigned int skb_count; void *skb_cache[NAPI_SKB_CACHE_SIZE]; }; static DEFINE_PER_CPU(struct page_frag_cache, netdev_alloc_cache); +static DEFINE_PER_CPU(struct page_frag_cache, napi_frag_cache); static DEFINE_PER_CPU(struct napi_alloc_cache, napi_alloc_cache); /* Double check that napi_get_frags() allocates skbs with @@ -250,11 +250,9 @@ void napi_get_frags_check(struct napi_struct *napi) void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) { - struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); - fragsz = SKB_DATA_ALIGN(fragsz); - return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align_mask); } EXPORT_SYMBOL(__napi_alloc_frag_align); @@ -264,15 +262,12 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) fragsz = SKB_DATA_ALIGN(fragsz); if (in_hardirq() || irqs_disabled()) { - struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache); - - data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&netdev_alloc_cache, + fragsz, GFP_ATOMIC, align_mask); } else { - struct napi_alloc_cache *nc; - local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache); - data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&napi_frag_cache, + fragsz, GFP_ATOMIC, align_mask); local_bh_enable(); } return data; @@ -656,7 +651,6 @@ EXPORT_SYMBOL(__alloc_skb); struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_t gfp_mask) { - struct page_frag_cache *nc; struct sk_buff *skb; bool pfmemalloc; void *data; @@ -681,14 +675,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_mask |= __GFP_MEMALLOC; if (in_hardirq() || irqs_disabled()) { - nc = this_cpu_ptr(&netdev_alloc_cache); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&netdev_alloc_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } else { local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache.page); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); local_bh_enable(); } @@ -776,8 +768,8 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, } else { len = SKB_HEAD_ALIGN(len); - data = page_frag_alloc(&nc->page, len, gfp_mask); - pfmemalloc = nc->page.pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } if (unlikely(!data)) From patchwork Wed Mar 29 14:13:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0627FC6FD18 for ; Wed, 29 Mar 2023 14:14:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DEEB6B0080; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 98F0D6B0081; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8584A6B0082; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 77DA56B0080 for ; Wed, 29 Mar 2023 10:14:31 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 55231AC262 for ; Wed, 29 Mar 2023 14:14:31 +0000 (UTC) X-FDA: 80622131142.04.39E0E0D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 9D3F818000B for ; Wed, 29 Mar 2023 14:14:29 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T3xRTiP8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=62BPou+xERgiMp9fd3+/fNOBk5TLUi4yt2oNbndQPtM=; b=pN4mvTtMT+Jy/Gap5sP3t9b0qH5JGohh15XvBKP60bnqzr8ToPg6zwEr/xXlWudd8gkfWI 91SVjAKNpFuzsNHINjqxFGcSnWxNxYnDLD8Cqo0KIRUqFCFJFLWTDLsGMU/OInsG50NZF7 jQ+6XGd0rxBwKGWGQORys74SYhK/cH0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T3xRTiP8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099269; a=rsa-sha256; cv=none; b=n63SmF46uMA7oTiUEb2i3pPoB+2m2fneO2T/MpkTZSUzGhlgKeEhQtJE9RcWoeFBcdrNPW lXsE3jD82XX3CoMdmKrXj6oEAlhH4CWsjCKl6kviOKm0N3RVKzxxysmBC3894eQA6jR2DS vTStrtAhawYow1zx4bxcuwLlnNt+auA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=62BPou+xERgiMp9fd3+/fNOBk5TLUi4yt2oNbndQPtM=; b=T3xRTiP88VCgO/hN8Vl2gdVWknGmvB34LXs3Pb4YA/fumWZBr1a9Q31BvZ8iw5UBohBPfa ZqwRGIZmIJYTJfiIurIqGM/IZp/ERAGIwD8Vf3JeQgBTpWaq/V/ZUOrlZt39tN3rYpMy59 PFdN0etFLBbsRk/2d/0Zk8mfO7Bw+TY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-74-xczYM2jqMimXt-WLCPIA-g-1; Wed, 29 Mar 2023 10:14:23 -0400 X-MC-Unique: xczYM2jqMimXt-WLCPIA-g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6A6F1100DEAF; Wed, 29 Mar 2023 14:14:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 850CF14171BB; Wed, 29 Mar 2023 14:14:20 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 08/48] tcp: Support MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:14 +0100 Message-Id: <20230329141354.516864-9-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: 9D3F818000B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: b6pxsk38pk8jq846demxhrgg76pdmcej X-HE-Tag: 1680099269-729015 X-HE-Meta: U2FsdGVkX18DZDAlhbxmZIaDfJj5zFiPGrh1eKsC+m8/XjOC1hf5KcY7lqFH3WqOijycmuuxnzxlD1xaeStOLtzetGgsOz+CyhuF5b5mZMCxHJZtYTeqOTaAC1Qfa/LE3KstYSXskjJwVR0o2GM1R8fek2yxlVQbnvRRW7UjpS7WGU8mYJzXT0glDjFUP1tc49d8F9tiKDvpx5wkKwyrY8jMjYfhX7rr0M7c1fJQGi3zD4/vsQZxUmcVg8uTjyvR/L0EvrzTKBh/ayClZS7u3+mgQ/tEVzes1ivKs1YphpRhv348FRaWHQA5ojOmdzPckqun7RZdVma2YYWfy85aLM7cucdCmAwzc9Qut4ZfXKlqwjt1RcXJO5PNFQb4jgQBkTy08D/7yT3fqw0CBMMJ3LOJMAI7M1ssJT7E92nWqDzpcABW2MpguoQ8gOan48AWic5YyEZlj1x1dcdJs8ssPVmlIy6ymGffkRQf33TirtJ5yjIb+P7AwLS9SQwZmyPSWyinbx73RcSPvm4MlAJqTwl2KtJxaEJGjeVj/nAGcCZZHRXRl2jG0P/H38s7A+JpY9wcx5iGo5f7gJrtS4ojMb9LV14khm0zeBShknBflFmQZkagmHaeSzSFlW9Q11pZU9TpdqrOeIEBLeH99jSdjIVKCucXBhjLyHXOtg8mBKWA7lBvx1lDmvA38LB7YR15mELYe3NRMjVpRveb/bMIXKih3IpIO2kLmbfP0e4NjaH1X1377UJqoWhGJZjjraHy+b76TSJIdAw7cg2rlFNIjoj56EHZOI1s4IKKWyN1/GNsvn2KyisLtpjyXlR2ToiARgGHNqVHwkHZ5FgZA5BxDZ2EtaMkmq7EWo2qDqCsRgjGxk1zcUMo92PjfLftGU5hRdur5uACyjtL4QdNGPFfz4PZ5rzmMaR72pzObWTGw2WRLukvoXTjohtVfh086EeXUronPOOatFFuZhmMEKP 8c2KXar6 AV0RdT7MpTI68kPCyiElwxi58kFFgJvgnHOl/Rmt+Wg5C49clb35msFQh8e+o+pBriYLF6Kmb8/fb4/c/2u8zAAcWs3NFFLr8L/6CxfhNk7g73BSMDoRGU8Z00M+DcqILI7UlWJAQ9NkE8WhRgyE+s6pHOwJXz9Fg6j+k2v7W8FW99nx0swXjqhgpCeBePb2sx2XlUMOe7sTQO5rj0j4HBIv0a/SRrBZsQZoksbzHDCRk+OMCvNXuIULzh7gG6mKHJ1maCh9TQTLx6nxFS+FGYWTFwtQE7mxcLn3SoWnCYV78RDrG3S4KWUp/J8nbWkHzCmYS+C3KBj7JWL+dSiMjwuKclxbMPRXsm6rns7UyxPVp5Tarkjzs3Ri5xJteWpO+1hv/h3OJEt/WVjt+G1YMdS6kBrBmD7dMVbvENdccZbnpopdohhNv9inMp0XPVSsTu4cs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 67 ++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 60 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 288693981b00..910b327c236e 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1220,7 +1220,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1231,17 +1231,22 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = 2; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1304,7 +1309,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1345,7 +1350,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1390,7 +1395,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == 1) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1411,6 +1416,54 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == 2) { + /* Splice in data. */ + struct page *page = NULL, **pages = &page; + size_t off = 0, part; + bool can_coalesce; + int i = skb_shinfo(skb)->nr_frags; + + copy = iov_iter_extract_pages(&msg->msg_iter, &pages, + copy, 1, 0, &off); + if (copy <= 0) { + err = copy ?: -EIO; + goto do_error; + } + + can_coalesce = skb_can_coalesce(skb, i, page, off); + if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { + tcp_mark_push(tp, skb); + iov_iter_revert(&msg->msg_iter, copy); + goto new_segment; + } + if (tcp_downgrade_zcopy_pure(sk, skb)) { + iov_iter_revert(&msg->msg_iter, copy); + goto wait_for_space; + } + + part = tcp_wmem_schedule(sk, copy); + iov_iter_revert(&msg->msg_iter, copy - part); + if (!part) + goto wait_for_space; + copy = part; + + if (can_coalesce) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + } else { + get_page(page); + skb_fill_page_desc_noacc(skb, i, page, off, copy); + } + page = NULL; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + skb->len += copy; + skb->data_len += copy; + skb->truesize += copy; + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); + } if (!copied) From patchwork Wed Mar 29 14:13:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F836C6FD18 for ; Wed, 29 Mar 2023 14:14:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B2986B0081; Wed, 29 Mar 2023 10:14:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2627C6B0082; Wed, 29 Mar 2023 10:14:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DCA46B0085; Wed, 29 Mar 2023 10:14:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EDDBF6B0081 for ; Wed, 29 Mar 2023 10:14:34 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BF84412064D for ; Wed, 29 Mar 2023 14:14:34 +0000 (UTC) X-FDA: 80622131268.22.FBD850A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 09554A0027 for ; Wed, 29 Mar 2023 14:14:32 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Qbqg4FYF; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099273; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4TWX2aSvGOZ9Qof5Bdvg67w1lje6zmFD7Oc0vC/yB3w=; b=z7Zj92XFToEiunV0kAA2Wk/ZCDEhPZxtj9K5NS7iTPN9JED0MWOrS2c8TFrN5ZS7iqWwRC 5a0K804+yq9AOCre11tVKII6remmWbaS48k9tEYvsJ6gXh47xx8Eft8/4ZmgNagGonukeI iAZyIX+ITQNgPto+B5lrGbRwRHEsPyQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Qbqg4FYF; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099273; a=rsa-sha256; cv=none; b=OYrJ8x2UW0J1lG8hA0IBM4RIleJUFY3WRDWZf9AubdLErLyXLPM87L8uGH74YibnX8beHM mrv6tQD+CGpZq7YEGV0xKWd8KVcbTTmQIot9lOTXPy1QrNckYNXg7HgF7TKkr1RCwvHHrQ bcoJqOqmU5dvfrRu9Qhs827b1td1yr4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099272; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4TWX2aSvGOZ9Qof5Bdvg67w1lje6zmFD7Oc0vC/yB3w=; b=Qbqg4FYF+OtHNzDtrxIlbaHTkQR0xlShbAz/6Xh1X354q2unflCJXpD/KrNniDi/4P4XOY Rk06Ir68Flk9G5TQ9bWG7qKQVsU2ckOwRjudNRMRI/+1SA5EkOxPx4YqBugGdvUwGB5+Hi XsHcJzZ70r34iCTRXy4mnjGbKdVCDGs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-106-aG1NERqFMhu5XfuNhb_KOw-1; Wed, 29 Mar 2023 10:14:25 -0400 X-MC-Unique: aG1NERqFMhu5XfuNhb_KOw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E1811811E7C; Wed, 29 Mar 2023 14:14:24 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 082972027040; Wed, 29 Mar 2023 14:14:22 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 09/48] tcp: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Wed, 29 Mar 2023 15:13:15 +0100 Message-Id: <20230329141354.516864-10-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 09554A0027 X-Stat-Signature: jyxm8mgfh5oaqu69w896m74sqb34zsam X-Rspam-User: X-HE-Tag: 1680099272-431812 X-HE-Meta: U2FsdGVkX1/aLNVJpx8zsoUeWwBoiLcpfqfQo6Ff2zed9pwi3QQ3DvOduJ5ZZExMqdYpQDAqf8Men1UT9G1T+4g8rhR3SAdOUb2Oc6rTndvQJrQ4Rjfzn2SC1Qg/X31t8VjqQWH8m9jOshPuSBKSqjSdpVKZfgEqfGzg9MEVJJ4+CeinEk6HS/oEFZvYuMn1LIUsz49CTEP58I9SiJ+z7PHSnncH570beXfcIVL7wdsx3dyzI5MF4x6pAj6fV51hxjy79I+8a6qnmOELAQQ+9dVMCBuRXuZZIYgTtbHc7ALrT9/2R3QXRD1kga7mPBd02vT243EH9T28U4TAuAnDkY8rvO5A2YNXUrip28zkOaYI2ZajA7lYZgBY1DXig/fwrlbxf1SoVPnIbmmO2klCuyEmYVxu7m/C8KbvOoqnBuNoRQWeFP3YxEy1iGxnzsW8VP8Cn/zM3WgI0XIj4hRXLIUCvDbKGbJTK9caH+B3OUCEyzf54QYxYQG+8QF+vj+B8/RLc97qsW09pAKvevntGXlmZusN83skF4TadlvlGXO00TkwERFnbQF4cWJuAZPxYpudrpZmMK1dAPM7AoFmmQZ56JWsYoBgz6uxcoa2XoMyLF0Gelg/iuYosNgiYkb/5c9qoNuSyg9lTD4S+SXIT/p4ijDz+J6BfQd2QqSkN89VMfmE4/RKv75CIHCXUnp175yoIPzpj91c/SJaPQfKqMAsRJF0341ri7eF5fDJ03c6VmdVuz/SNjIAyBHhEB8JyFAUIKQqrjHSbM5Ck/VchxjEP5rYi3grsOWRG16ibScTyZpUhSnqvc5mV5Vn1XzQ2r7YJOznG0UvBOBgF1wCdbNFDG8oeqMsR+Q2P6rw0HMuTzqeZSZ3jflX9bRZO0v0rmuwIpcVwCU+urywA58UfCcmqvZ2AOZjNOiuC7dkuHImswJukgcmE27s7++/8JMx6ACGMpSg1gNz5f9iIAp 3M6EL3F9 uJrut1uOKE94yjQl3DEShzJcCjuDRqrdGwfnjzD4HaKyb+ZM1otaurYavTIMMeshR7gE+cE09e+wsYixvbolbOwnF2sOFTdaO8rM2qA+T6mErIw2oape9RYGkK+RojwHFXNh450Qg05QLIjIWpsadncMRCG+MNaGfgMBWkabtoHF/dnLgsxYmjXMp+10vICZG4I2xlLZFnwaAgCCmCWbIjoujCl9l17KBh8KQ7k0EM0XsYmSdhbsSCeHRHsSfftPcY7Eal5lkc1oYhCoR+rz3aI/qkF318mX4efX0Qo1M6mqDnJI59FfOChxDjDbzSFh7PsAvvbcZZBJYG/DnN0oKACYrCAT0vFkX6bcqxjt5L+hqaszAfiWrtcqTrxAaoxnJBazDzph+ncZkZLwAIKUwsmqjKYi3S5HpEzIaIJOx4+GjZMK96gqfEJofScl6GJ7ieymH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make tcp_sendmsg() copy it. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 910b327c236e..6ef0518eb706 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1417,10 +1417,10 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; copy = err; } else if (zc == 2) { - /* Splice in data. */ + /* Splice in data if we can; copy if we can't. */ struct page *page = NULL, **pages = &page; size_t off = 0, part; - bool can_coalesce; + bool can_coalesce, put = false; int i = skb_shinfo(skb)->nr_frags; copy = iov_iter_extract_pages(&msg->msg_iter, &pages, @@ -1447,12 +1447,34 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto wait_for_space; copy = part; + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, copy); + err = copy ?: -ENOMEM; + goto do_error; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + can_coalesce = false; + } + if (can_coalesce) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); } else { - get_page(page); + if (!put) + get_page(page); + put = false; skb_fill_page_desc_noacc(skb, i, page, off, copy); } + if (put) + put_page(page); page = NULL; if (!(flags & MSG_NO_SHARED_FRAGS)) From patchwork Wed Mar 29 14:13:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CBF7C77B6C for ; Wed, 29 Mar 2023 14:14:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C3F46B0082; Wed, 29 Mar 2023 10:14:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3746C6B0085; Wed, 29 Mar 2023 10:14:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1782F6B0087; Wed, 29 Mar 2023 10:14:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 01D986B0082 for ; Wed, 29 Mar 2023 10:14:35 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D451416067D for ; Wed, 29 Mar 2023 14:14:35 +0000 (UTC) X-FDA: 80622131310.07.2A141BF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id AC96F140003 for ; Wed, 29 Mar 2023 14:14:33 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Oo/66DB+"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099273; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lNWag+P+g8mnNn3gi6svi7x9UvjDZCb1uhedzWJMgb8=; b=6tw2oyoAeYPd+8fO3x1JZCLVPBd22mi85AQFqoh+X/hdOb0YhdP97Z7UIQj39xpoknHh8L N16peIA1cafL/rWjND2mPDI0aiH78rdr0XYDIdBOXC0BTODdLizajMQHzNSnL8Kx+CY7nT mfNiS1OO6dxh8mpKmzK2ASA6CUPBjR4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Oo/66DB+"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099273; a=rsa-sha256; cv=none; b=a38n7XnrUgJK7AHX6885AZ+Zc5obcfGOCmD1dAT0oF8sHWnWSRGBjh8jlYbjxlGEE+dL2Y 7FBq2uh6HjxykLrXD5LI9GVGREja+2iaIOykiSt6iqn8tf1OnoEZ2WC59AObRERtffqKLO f9UXTASd8ZoExwXOFSV+zgHp8Xod0bI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099272; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lNWag+P+g8mnNn3gi6svi7x9UvjDZCb1uhedzWJMgb8=; b=Oo/66DB+8joKChMy4DiNu497WzOMVGqE/8kNx8rHdtsSTz0EDL3jGtrUZjmuNQiPjhglyZ FDiPEK0dXl60OONwPC8/gWdpuR4NTIHa15M7rU/YV9/LDIxSAr0BCXiigDEMRosrG5p0Lq dXYPQUJZVIhusDF9kkwLwxqmBmTxNww= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-600-m7kGADpVPpym7xhogv7czA-1; Wed, 29 Mar 2023 10:14:28 -0400 X-MC-Unique: m7kGADpVPpym7xhogv7czA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 64174811E7C; Wed, 29 Mar 2023 14:14:27 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7E0FA1121330; Wed, 29 Mar 2023 14:14:25 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 10/48] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:16 +0100 Message-Id: <20230329141354.516864-11-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: AC96F140003 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 5tsgf3c9qcc8usmrojqqd4br5g1ew94p X-HE-Tag: 1680099273-527307 X-HE-Meta: U2FsdGVkX18IVkcAfmFLRpoCUlV3B08vCrcLnue6dKDMvtRWl3qNFJmfV7PG5ltFeLqquLz3FpeudGkyyu/H58wBZmUzSDKpPhTKlZmiaYpI8Ff6GT1AH28B9Vi82yin5P4BjyAUZ7b4N27qNcCmcpxT367Yg8HgCx7Q6tKiOBh9/L9x6AOH2AbOWCkG6glzTZbTRlGmA0jKQYwqPoXWiO7bQ9+MSXAIPGyvWK26U7GosgWG6m8IrBJnd/zqMRx26XdyvDMjKRFSw/ZwjRsVIYeF3qOS+HlRnO35+tbsgxmi+dLEvVELW5gQhT4sNUhHmh+G+svslswg6IIpYOnjCVtREJWhmyAiADpO3uaV2I2BB0WDcjZL755NO+fg5Yttd3C+ZwX8TAVYlSUE04B1TP1BIZrcOIKo7kNDM0MlNae4DBmjSOMM7Vj7YUvJ5JcvP+dVDrG0KHb5qyzBIcAph1CaSSWyyF7eK4ZlmB4q3g9y4Roxzm5UsIRgSIqnq16gVlDepTp1hTM1LCc3uY5FSGuaeThfiwHjkSVnT91ln5Et1GX+NOLW5yk/w4LMrGbQr8sLdqkLW0IdDtI1tCnMVuryyeYjIDMaqzvfAYop1C9TY0Xk3RWXdUTmJvBAUfLItLfIyE5Je1p/GIOREJuTfudcZV7DGJygxP94KiakV/mBXslGoxXxEBXKLx+5d8Fm2N9e33SciFWktpWFbCFFlfgxhYYVeI26DNJPPueCEPaCUwLnF+PGQb2NnxyXTTrTkU9720LPA4isc/q00bcSolkHwKqQDNQ3B8ozHSvEfUMc+N3LrcOCjb+OsUYAUls59JmFGbi9IQtCO4j4BsGum0peCwshoJy3f7SpmqE1kGHq4UJ7cyuoIOPraELwVlfLdHRvmqD61M2B56HZZXY2qo0Mv7egHl+CdWd5pzgDI8oNv07M1YUq4BrUySwyFFwRff6hscJ01l7G2ZdzRMV fCcAK2tX jH8IRxP8DsezW+OJO0FCV8gA7MszhO3uI+d7iZh1tdQPbYP/kcuqKLpjqLW1qIQXjMcaPan2GNKcJ6oibqAkZXxec5GvB613EYeGqmuuSsjL2Aen8+nJYkAxwf1XhIfF3jE3qSx5H2xttEob8E14Rm6HtBp26MuBrVhBCVOSR07n4zJ/0I8faUIQVJdvT9ABB3xPLsSVi5K0qd+fbvoLwi1/YakjO/mPKk09z/cclQrnIHP9nMFQsf3akSxp+s9Cy8xZzHKIVskuzte1KdXgttNvzFuWTDUtHn38ccOFwST8+yuUrllWoPULceeTG7mGx6R0rsTWnayCDrwpeKV5jTamjUOALu0ytuJfMmQjkYE3lpDTn764nMb49SfeCqjrO4M7sZrw28l6m1W7rIrfqljQ16YXg7IL4GRYOlyRsotuivOjpVKe9cZ7gir5rsBJne2Fc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 161 +++---------------------------------------------- 1 file changed, 10 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 6ef0518eb706..a1c5a6d9419c 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -971,163 +971,22 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } - - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; - - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags, + .msg_kflags = MSG_SPLICE_PAGES, + }; - mss_now = tcp_send_mss(sk, &size_goal, flags); - } + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Wed Mar 29 14:13:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46810C6FD18 for ; Wed, 29 Mar 2023 14:14:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 216296B0085; Wed, 29 Mar 2023 10:14:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C6C66B0087; Wed, 29 Mar 2023 10:14:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 090186B0088; Wed, 29 Mar 2023 10:14:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E14996B0085 for ; Wed, 29 Mar 2023 10:14:36 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B8075AC262 for ; Wed, 29 Mar 2023 14:14:36 +0000 (UTC) X-FDA: 80622131352.29.4D402E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 0B1EF1A0029 for ; Wed, 29 Mar 2023 14:14:34 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=O3xNpvbS; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V4A9+ThbnxeOTamldoFRxBPwB7/WQbFmJXJUgDasL0c=; b=IPPCq0ZNbfka5IkWtM4RvnmTVXGKvKX41N3In/k9mAn5rTxtkS/Sq4b1h26rBLjuU4WFed aMM8ygVy+guFEjyDQWWZvymBV/KAHUyO+LFgS81E2wMxB2xBbqomSz8wFaVzLm5wGqQu1a q1GNDv5z8gBnSpyx8BP4PrIBa6Yo4Xc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=O3xNpvbS; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099275; a=rsa-sha256; cv=none; b=BFeUyvQBHYI3dt1PanDf6kcSftfv3zQNJtMXw1riM7XRl/BAuDnsu1ac15wVq5P61wUWC/ v894NNMraXX2N5q+lSNUXuNWVT7YbJVoVOmEBio13L102FJZEtGW65TBA9jWx+U7hGhpH1 s0eDiaYfU+T7Ets8SaX5a3RThDSTWRw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V4A9+ThbnxeOTamldoFRxBPwB7/WQbFmJXJUgDasL0c=; b=O3xNpvbSt9lAcVr5qTCr/dsNw8z6LGMAVe9SkWoEGnQDh0tYTFxEPdy8e3S8eWUfxnMndE 0wKx8Co/0o5PWOhdSKWJEaHkEeWjBTW22lr2nngf2AiucmyUqGQDvtGV4qLesWOXvpBlpG dxm/tjpo3iiets6oh6NEeLMZ6drBWgw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-16-Xdfj3glLMsiVuDad3Y8AFA-1; Wed, 29 Mar 2023 10:14:32 -0400 X-MC-Unique: Xdfj3glLMsiVuDad3Y8AFA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4CBFB3C0E451; Wed, 29 Mar 2023 14:14:30 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 243C9492B03; Wed, 29 Mar 2023 14:14:28 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [RFC PATCH v2 11/48] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Wed, 29 Mar 2023 15:13:17 +0100 Message-Id: <20230329141354.516864-12-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0B1EF1A0029 X-Stat-Signature: 1suukhxbfgicins9bjxwwccdbwr6dbuq X-HE-Tag: 1680099274-667414 X-HE-Meta: U2FsdGVkX185S5H4J8aSDxAESUhWcJCIhnr/NwYYnlBWuiNx5L6fuoKn7p3Dv36vUJ2DRx6hW7SQ3Jl21ffQutGiEcIyV655sXGCMxzYhHdBJuxzTi8/g4OKPvCSAge0QazMUYBI/+ZbbObD5o0NG+BtO7TcrVbl98uU64Yc86Q3LRfYo8qlmWYa+J36XfoCkRQDbPSBzXqxbJMU1vcWFSJpWwO6HxCJ/kuWHX3skIxZZlz5RYKo0Mpj65r/tzklP2dgn0rot3aS7I1+ZDTwW9f1wssEnLlA+iHm9SddcbsU6LLXGFdkylZe4VOm5BwCfsVe5yRxG0/neTjFM7W4iussHxXAGzHtR7NXthumMfCMQvOQATasRVNUn5rkDuBtW0TGlD9PiWRGzRwJA58j7KIcPG1IhRzTLKuKqDtg0IckqL1u4EVHGDHqpuDpqB+dwjEIJR7TSXru6dW2Kj5Yl3z6F8Ve0scrW/jcdV2oP1KBAuqB4Ore39M1LIEcGKKisKOmOLVLrhixwcAqWpCJwcpssDV2cc5oBa+3c2P2lgJGSNvMyqak7tJZxzy/QY9bLLsfPSNTzrLdj78tJ/PJZppJNUhxcemeJ+cnkkGxvp6DwlYp5rq1jQ8gMPdnxcA0p5trL8Pv+NHyrbrtHNtThPIEBc2qMaqNhEeVTbsv1Cq0TRQqPeKFMQ9xBU5pKugHctw7UY7knFgFA7BqDCgfKHND6BLlEQzS6L452P2w61qzvMjJJEdT2YdcV8ph8LcPPPCxpseA/pF/uXFeQpDfv4YvILlpygYXnNLqcUiVwQrd40jhy33++hrOFFtA0f5UPHInUx9YO1drBWwqXXrpd1QpLilvf2OyAWQop0TwslYhv9gfec3PDEyJSJF+Xn8ZjAwGuzLoZ9t7pQ8DjLQ8aW7TT9nw4oloV9TQcrc9GmbFtRcNtCOtXl60mKbd92A3X4UrEo49S4GlNFR4ZEX XWNFBkNk UTqpjXKLnLduGRe4MBg80gPEs0B/LJauiQBvHyiCBohBFNLMWlWv0HBz1BMGNRX0JMhDnr0HLEVgA8fvspRCEFMSZcbPQQYtQDvY6RFJisrVRnAHzvYyH64jcS1a0g9KbQV0COVtnf6/pUl9csAc7/wgs9D8yWAcC6XfN/V9insl0HQz5isU8+lWZl0adCKSvsMZZgesWL+VlHfXBzpQwZVOfgZ0zznUeql8qbpW01A4s9P4EXWgaZVguH314qNvJKBOYnp+jLyvw6sPM4VIYiYsRcLVFzYthpTYCq7gzVBrINV4mjkaI4xtzC+HBPHnQ53PRdxFEixLnJW6OOLtmmU1EPdPbBWOi6kQQX/HKfbMYVBxOhyQ75bL6TMC2Pww8hYdhXoPpJFTWH5Esm1SuiGatMBXPjovDiS5+hNA77iuCXRuUjWd0tshU+O9R7GFjCVbiKsZBzNwVD9m5WbQE+3oV3C+essPN6hZyLS+a3OC5QXwmgedTEJ2/Oo+u0x/bSyp0rpaP0QmXzku7AX7fWZV3YHYKfC/Mw8MFbeUHF0gcn3Oa6Xn/kOUhCQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index cf26d65ca389..7f17134637eb 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -398,7 +402,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Wed Mar 29 14:13:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA45FC74A5B for ; Wed, 29 Mar 2023 14:14:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60DB76B0087; Wed, 29 Mar 2023 10:14:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BDFE6B0088; Wed, 29 Mar 2023 10:14:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43A046B0089; Wed, 29 Mar 2023 10:14:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3053B6B0087 for ; Wed, 29 Mar 2023 10:14:41 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 082FF160D1D for ; Wed, 29 Mar 2023 14:14:41 +0000 (UTC) X-FDA: 80622131562.15.CA861FF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 3CDD6C002A for ; Wed, 29 Mar 2023 14:14:38 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EkhqoFr4; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099278; a=rsa-sha256; cv=none; b=V17NZ4PhSeQtNyhvY5uRvSpNh6PfAM8XfRrjYiXxz6dOvjMOs3ycFEN17yP1OzgGajsd+h FgoDY6sEIhUsJ4wAagsPFgbOLVYy/z/O5QCqe+Ur2F/NkQ1wagvZ6EeW+qVZX0bdCeJ1nL G3AGwY3lSyiahUDIM+/BVTEFLyNi2Ig= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EkhqoFr4; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099278; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DjALLcCTAkJ9GFGzQm0mxszTSA2UBcCYteYf8nsJBJs=; b=7aq40AynbJAtoGaCRQ/pLMwBCpiSR+pOUpiksG0F/QBgOHGqUsl7O0rY/IvNJG/JoH4o+y E1VXtHWD8TkuPHJpJ1s0Q/WqQVo0s1Z5LBPeZA32FeMGKTQ5Fd2q1gaU+rCqv6InwSWjHO es92hDJo423ZVLRks7rjutdc2CQUL4k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DjALLcCTAkJ9GFGzQm0mxszTSA2UBcCYteYf8nsJBJs=; b=EkhqoFr4Q/AKgIncNUIt61FcbK31PU8oe5MPjt0AepOz4pL6ofZ4tAN4CwN6xFzFMDCgme 12qMSpKbZtxn0G6As8R55763879sEV9mYD96Kuj0AuYb07iw+5H8J2bf4Ll2kumdi3N8mJ BX85i9jU6Iu0IcHyXiJBQFOYLQwYfDI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-671-FOdpR13LNSGAWxRHNrnIWQ-1; Wed, 29 Mar 2023 10:14:34 -0400 X-MC-Unique: FOdpR13LNSGAWxRHNrnIWQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 14F9B185A78F; Wed, 29 Mar 2023 14:14:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DB6FCC15BA0; Wed, 29 Mar 2023 14:14:30 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [RFC PATCH v2 12/48] espintcp: Inline do_tcp_sendpages() Date: Wed, 29 Mar 2023 15:13:18 +0100 Message-Id: <20230329141354.516864-13-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Queue-Id: 3CDD6C002A X-Rspamd-Server: rspam01 X-Stat-Signature: kf6reyunnzs1tr156ka3a3ypbk5jgcji X-HE-Tag: 1680099278-256531 X-HE-Meta: U2FsdGVkX19BVKhvut4WcWuat4ZzT3Y2XPATziGaVtrjbovtIggjpLW5WgyrQNPUhuWSrE3D4gF0Mvc/6Symasgso4Wq4cZgnpFuFvaahjsIQkqMduSq2Qnk1RC5KrYf8kcVlY+KrdQwQFUF21c2TKWZWrNrer0Y+3c97t4jGzgMQuHbAiKHjgCGANgrR7ePAFOzS46boN5fE9smxouigTkq+9U3uunrjlCqfRsmyFSqvrhvLXp+pUGUewm+qDjCxxGkVucU+WnPZaxk4v51s898U9jVeZJgac4bSNzUjUG4FflQds5a+qLX0bROfa0lb6vY1022scIgO4aoWUlI9QDLzEiOG4KrCWKkPHH3rSgarn78Bpk+iIdjOnAMMh/lYbj9TBvpzTTg+nbjKOaQnooKTLZG47pbFLkUDclGiRZ/45b/DJ/LPr/+bBxen82mCQWy/5jR8HkkI4agzC4XwcrSLOGy+VQNSx+7JOKPq8lD0WxNd4Rul7nMo7deNYFBo5RlcBmVYrzve5Dc0WlQmDCl35IRJl+7BkStXThllRARlqsx9ZT7WJ6S08cO0YrqCj2si0E0SjeeVqxrOJfojd5O8HGjq5LTIbR0C7b+8OMhwZvjM/Sxd5bFHX4/UmL+saFEf6L8x97sERR1cSGjNvf20a+vMrNlEZQyLHsCzRwW852ZYmmZMTygGomREwIQQQvShVSp68yBuPmmtu9i69CnxZw3j0/PWngNDBzrs8zfcn4fz8+HyqSYhSUSIBhq8IzJBp4KUfWByDNOi2ZsChIXkVe+mcry4nuXinzPTVZWgqc8zxu41yFk1AGGp5bFdvVgDnb5t8iYy3T2k3LVa0BzQ0KfC14VR3nLMnkJZ3sB0bxL9aPrMgFv68J76yPo8VoQqhp4P0frxne59LbLpwg26RrDCwEEdre/RP91IvMGQS0O3E+hTYqoqJTgTLTZ0xMav6NEYY7r4Hh8IaK o8mhNaUL L3C/RH9aTSMLGjX/NwrvA4+xOmhWU2P7XDxvDGoOCes3BeAAZI47R4ggGtNUhGJwl27TEwL2OWFTB3Uv3qpbqhR4OKu3f4V+ZK6xOjTOuM5S6aGxSeXfgMI1AXH7IkUuwYy/TFs6XWVdNyEQUXnWCmYOdtduc9sHxXpNS1tRdvXk5JG75PXekQ++BAtN+zFcQ6tK5Ui11l9Hy3sPoGpLjamzbn1Lm7y2UEMxUgMrFy5PFHatlZCKMqIG3169wGG2YhoRImfXNa3PIYkM95aXF8awcDbycVPdV1iP9Bl0mqGKGM18JMidwiRtOScxNWVwdv+huHibhxVh9uraVaV7Og88MlXItDwGhaZrSaLxKtJEqU7wesnb6OTeECwMlSZt0p7bIkhOVe4QkBjbzNc/PN9eRdaYS3qpadR6xM2O0t9Sp7ETokYe3W4LrlO1AgozikXYFt7dYMJFT4bhDwwzk+0ZOjkJVPqpy+iHVjSLGsAlOeDXAX7z0wFiRThb8/mVDIdporPG1UfVTxz3Ud5vyTufrcxYn6EBMVAR1NeqAO66C7/bbG0iNxtLGWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Wed Mar 29 14:13:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF394C6FD18 for ; Wed, 29 Mar 2023 14:14:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 460766B0088; Wed, 29 Mar 2023 10:14:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40FCC6B0089; Wed, 29 Mar 2023 10:14:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 264C76B008A; Wed, 29 Mar 2023 10:14:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0E74A6B0088 for ; Wed, 29 Mar 2023 10:14:43 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D3F0B1C61BB for ; Wed, 29 Mar 2023 14:14:42 +0000 (UTC) X-FDA: 80622131604.22.500F009 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id D6D84120020 for ; Wed, 29 Mar 2023 14:14:40 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fcnlM7wm; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v6WcNKpTWWgexmydU39qEchX1MtJJl9VMoZGlW6ePEY=; b=0t3SLQGtoHHMj3g++55Z0ktMo2T+FA7DBS2fDaPZYADL/qwO8Sng3qEC7T0GVnqVKWetLP iZDZX5O2zBymYU6plgOOpk13UtsAP+6G84UEzXKjBBtl+LnVhD3B2f1wTBYS/rsl2yPT0+ tBPYG36T02d33YkYITQvuN9hYOpSO5U= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fcnlM7wm; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099280; a=rsa-sha256; cv=none; b=ppi9LPA2LhRs7XZuFBy/CwPRRApnWex+dMVy6OfD9gxtgvUWEYduXEVZndhu/ku0mfJzLS MNVvUCFxdzNL0yWlPuRL1N78cPrnxX19Ct/pB9Dyl9jUCy08zLZMpkuULjtit4OPYCLMXD uA5t7nBbJ+JdkxQMwBqLLB7P2MPL/DE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099280; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6WcNKpTWWgexmydU39qEchX1MtJJl9VMoZGlW6ePEY=; b=fcnlM7wmufZdjeVJ1N0qIER77Qvut65Q6/b1CUVkM3f+pzjILKvWhErozoWT7rqdGwWopf SXxfoZ3CAiFJ0J96gVxDQfj660NOqX28Wic0hsFUpGD/vHHvXyn/A5CV77s7clU8qhb6dr N2M1O9qL69lD5xg8umWJRHJ8YIzdEGc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-428-r-bTm78zPGe9KMPftn10KA-1; Wed, 29 Mar 2023 10:14:36 -0400 X-MC-Unique: r-bTm78zPGe9KMPftn10KA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D52383C0E441; Wed, 29 Mar 2023 14:14:35 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id C31801121330; Wed, 29 Mar 2023 14:14:33 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [RFC PATCH v2 13/48] tls: Inline do_tcp_sendpages() Date: Wed, 29 Mar 2023 15:13:19 +0100 Message-Id: <20230329141354.516864-14-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D6D84120020 X-Stat-Signature: 95k4gz9a5stjx8oocsw7mpc4uk19hdg8 X-Rspam-User: X-HE-Tag: 1680099280-824317 X-HE-Meta: U2FsdGVkX18uTSMgsjfXMzE3DrRoCm+xmf6CqtaueSgUDXOf4vsBSzJy+5FWtY7Wy3t2SP/xqIzmVPDFK4RkXLuNmjqINak3SbRrS1tyjGodoMg7jvkVGms+WhbAa4got5YcN6rBw9leh/3tNEn4DziN/P4/iW5fxDqgfy7hA/hEldPCTcplPMpA5dRB1D2H4JpY1pvq5ZzkrsJryfU/QztQjfpON8o5fz++GJRV3S5l6pYlsJDlnZnKCkzLZqNjNMjGQDPqTee3NWVGpPZzrPWf4CvAchWGQaXUVzAYiEThaDrNUey1qyD8DZFnb9+aZ80HnxaPn5xnIw5PRjo19JKA28bUk2cCcGzJ68ojCplKyVr1huaOCbS1sJ9bL9Rb2614Y0fneV8JsaaNelOfS3aMFyQ7q/GMgl2B/CUqpdOSpblsRI46DxOaqiOAqPM05wqvlCToZoH/2ODYT+9K3IjUGBpGpCPWWw2IrR9WmaGhj4vm3z1Pb+56fq0/YfGKzJU5ndBagxTKY9wHqbcJxEtXFDrRYvHgXVeHAXhL67S/Ow1ivFkHMbam16TxAxvQAy8nGjwcOuiDYjpfcSdTFii0dVbvOqZbaSJR8wZR3KJgOYQyBnecFYq+8b8ILB73O6p+/vD8SgIk9D5SLFbddNecJofKrC25n7XWfoJow8JACaKyPz5MphEs0o18dcuAULYw9i5SdQOuvTYrhCLfyF8MeHf+UUm4A0LT7lK2C5/b46ad3VOPgMQtO1POIIeaDPbK/TSQctR5PNrScKol2BL4jz567cRCPasIJGL8KxvcbL2UzWTGNi7FD5+xIwRe73z9BAW1BTqT5XP4hfHWa44LJarh8puxqeQFY0f2/H47Fdf5lWU7wNzUAWOxZFPmfqyudaNC//GQrhrDMOz8+7sCNenROs/DglNs1jx43phQ061QeE69vWmTT8ruNWkcEDhdVDJUWUgjU4A60aT yiQgD4yK 3TmNA+IOReFtGyZYf56CznDBbCAV15Hnprdn2cinfkwJZCBr8Eo0HlhLOsyu6R1KcgHeeyq4ovxjWGYVtWKLI+JYTzt80vtIALoWjvUdwOUVJOTmdJVNQ+Ib/cFpGfrlWhFJ5dlVirvzjSeWeB4kufvTA2m5L9fIB0gnCmPufGwBZjzB2NRW0xyp745T7L8ZK0szrCaJVGDg7KHxGr2+yxQAQf9UkRKzmZmxm1W/Qv9TLKLzwVfiG23jYl7cDjeTEDAvh8A9XG8JSO70/6Wi/IzowzI3+oeg3QQloe4jJiGVGa0KrujZgIcxFtQ9tnojJq2sdrF/TLUyHN8EQKRD/akXUpdma0mRIoXtzue3SEsusVvqvzfUo6j/pN1foDMlfcJRAHTriZoEYHsXOjxaKzxXyThyCh/YdH2XYHoaYfAzHSjaNeATJs3rg8IQ0GEPg8tmzono+ZaXR9Fnwz6McARV6rxJg6tW+YRMD853Y3SjRKOLxFJXn3HPArEG7SGaM6hEvgkgb0n3cnRsAQ4WWYPAySMQu8cMhZ9T9Mt5ky23ingnW+ZKMU/Q2YQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 25 ++++++++++++++++--------- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 154949c7b0c8..d31521c36a84 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -256,7 +256,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index 3735cb00905d..9a1e77de6a1f 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -124,7 +124,11 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SENDPAGE_NOTLAST, + .msg_kflags = MSG_SPLICE_PAGES, + }; int ret = 0; struct page *p; size_t size; @@ -133,16 +137,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -154,7 +161,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -168,7 +175,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -246,11 +253,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Wed Mar 29 14:13:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CA39C74A5B for ; Wed, 29 Mar 2023 14:14:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 990566B0089; Wed, 29 Mar 2023 10:14:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 940BD6B008A; Wed, 29 Mar 2023 10:14:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E12A6B008C; Wed, 29 Mar 2023 10:14:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6EF126B0089 for ; Wed, 29 Mar 2023 10:14:46 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 47C17AC2F4 for ; Wed, 29 Mar 2023 14:14:46 +0000 (UTC) X-FDA: 80622131772.03.B13463C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 5DAB8180017 for ; Wed, 29 Mar 2023 14:14:44 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J2E+o2T7; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099284; a=rsa-sha256; cv=none; b=wEM8AXpfS9oRn/5dbtjRPBdK6JGw+IpkT9RePQk+F6NCPdiMrU9ojVB1brrx4p1wLQ+Ptn i8R5lZ+7FHOSGVjbhNi47hIZdAE46AkB0+QMFI+e0XzqYBgJCMsEdnfwnXlX16AmnnvhVc CKEhzIWDa6SFJw06hpdRBnonLIoIing= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J2E+o2T7; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099284; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pP5V7G7GRqo/82ZWWApO+0CTJP1pdqvbg0H8r7V6l6c=; b=YuG8L6VGif+HbgLqoIa/yIErJCWIs+lDXG45Q/5W70GeHD8JXQtIDV5u4Ehn9L7ESFpm8O PGBNWtkTt6wgzxa4R0WhYYOgjoI34vFxzH3iU+SaXckWXovHoT2HqTPttmbeNPq3BW/Omo 8bb0Ak23QcQWhJ3+y2rTFVtl7Waurws= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099283; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pP5V7G7GRqo/82ZWWApO+0CTJP1pdqvbg0H8r7V6l6c=; b=J2E+o2T70kAFGxqLvoobWVTTSaagIuWVpnsUWOWlG1ZpQmKRsABLDEvZtwsaq5wjxEm6SZ YvQ4n+KOMk5B3xQgRh++Lj7vGNjC9udt2gHU8KBRjb/xAZTgbKGEgLxO8CvlTTDwMwvdm7 Pei8i+IIxXWaRWQYx2FpkLFR34N8124= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-387-i4eZEzSqMsuAkK3bbaUC_A-1; Wed, 29 Mar 2023 10:14:39 -0400 X-MC-Unique: i4eZEzSqMsuAkK3bbaUC_A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9C7CD801779; Wed, 29 Mar 2023 14:14:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 715D0C15BA0; Wed, 29 Mar 2023 14:14:36 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [RFC PATCH v2 14/48] siw: Inline do_tcp_sendpages() Date: Wed, 29 Mar 2023 15:13:20 +0100 Message-Id: <20230329141354.516864-15-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Queue-Id: 5DAB8180017 X-Rspamd-Server: rspam01 X-Stat-Signature: naz5uo43nk3xcns7itywap47m1badxip X-HE-Tag: 1680099284-741091 X-HE-Meta: U2FsdGVkX19UFKcDTG9Vh/W3er4/8II8aVizM/YyjtDSvDaDpwd0NOP10K3FCwyBcZ0onq4F//9kGgO+8fAfmM59D1AnWdYa5iIJkELlqXYETZASFFDZT6/Izq6a/p3MIoU2e+zxDzZF2uo51NEr1vN8NIX488GC42M2kx2gMg3Hb+q1Q7XzCEqD/g5hXmxUiP4TWbcPDSCEDS8TvS3hmLM9aTyZ5Jh27dFNFB7mAwYMD5syDtN4luQ6zaG0/uo+wkZg2RkPVWscUrzoY2ycJQVECvjhWa9S7Lmc47EbiEUnZWqhvc/AjH8b4aUJKonEg3C3RihV9aKAF2e6qONHHCfZnml6w71l+wERAXTyfu3cnbFB0hbhgf0rErxsDWNryzISOL4CM5tim0NC8yPwPcjzK+HqVnmVem51GYRgrMy010ggkWLBLJcDCBuJTUg8z60aHCXuuOLv/zfGc/8U5wXWwkDK3cjrF1HD2gbJcEcfHbloEgxYEvW0TZZmH7sa3AOL2mxel8U29e+cdg6dLKw8x1Z3V1OtPA0S1KevQD8visiiz3j2QRPSNOTCP1U+Bnh5s3i/dVXIG/j5mIdjPIbT7VqGoaISsMgKyYnwtbmHlMBi/QnkfsLcpcB6lKEzdkb67hv0olb2Cf2T8UI6kQAa/F9HCMwV/q/rJIkiXssTMU2kQCt2IKR3Dk8dH4ds5C2KLxhPdTlC19BHhiV1Ijdshn1vQYIIT1OrDyUOFow6Iew2SQW15wAMLYpCIwNbnf5qD7fkHVfaF8h4YSJRJY+UpVXFn/ahZRJDGOn0ZHjkP3GcD3xBlyw1ilaXXoWgbkaIpnwt0RC8N2X+dr0UgvOn2oPTtbGqb7PgF9mBQ5dC4TOmsVYPejRPm+usB4mXqSPts0pt4lW4WGUY/RyrNh3qmmwctKyJUSINtyJxKb0J0L44H6n9Q2xME02oJcoXyCRreSjdXFaZ+sgBLq1 3vtSF3Lg QrcH/ZxgmByHFejTKXfvL56fd0aFKzYqNGKZtASIc58zNdWSr/BFWCz8Jq3h1F8/OVy9SJ+Mj7KRVnxlW6XKZL4roa9RqXGwLIGDf0j9f/KDFX90ESZahQJNxpvxEu/ebSuM4zWUxQNOgtGxQaY9VFMcmajiB62t1tTIrETlIAr6nKyaX6pDTlOlM0TFX6v9VztZVR51ljUs9HWDUxIgRTTPgdq2wEaE4Gqq4jay8EJEbW7CFnaLwm8I49LAK7C80A0DusYMx3xVaJJhhmJTNJ54gpCHsG/mG4rnK54zY3oopI+iRqDtxZ6y9QOlx0Eb4AvIPgYAHAW5oPOYOt+5Vml5HnrgQKmTS0eFw+Y070WbGoebvJoL9nPB+YQgytumTtVOhAnEHQTHYqnpxyfEjNIdPAFOWnqrkopCTlG7jv5vKfsTXS3xmvZP4VS5MSgABOsiBzOQFNZsjSOKUeiI6HPMIC69dmNLBhKTFIxMOVeTd34RZxTc1LN6yAQmDoSPUttsnPyPoF+TL3MQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 05052b49107f..fa5de40d85d5 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -313,7 +313,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -324,20 +324,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags = MSG_MORE | MSG_DONTWAIT; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Wed Mar 29 14:13:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB3E8C6FD18 for ; Wed, 29 Mar 2023 14:14:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6354F6B008A; Wed, 29 Mar 2023 10:14:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BFE16B008C; Wed, 29 Mar 2023 10:14:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 410806B0092; Wed, 29 Mar 2023 10:14:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 311D86B008A for ; Wed, 29 Mar 2023 10:14:48 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D6F9C120647 for ; Wed, 29 Mar 2023 14:14:47 +0000 (UTC) X-FDA: 80622131814.10.6810DCB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 221E3180026 for ; Wed, 29 Mar 2023 14:14:45 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YD7fLFro; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099286; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bb5tuF13A7AUAmGHrtoNc4Ue+Am7P92yBHtRqoRTTps=; b=DKPtMGxKiblqjqOe+yueMQ4iPRu4InIiHUiNvNjUbxXwSEj94XN152sT1acaVT/Lo+7+6A X9Ha6v7ZGDo4HkC3LiihjJvvkxE7i78O9q80+48cuL09Ox16SxIDKq3jzsGSpuq3xoUgPy KybBBLiSxLQJORpd/FeaAoWrrUXTZKQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YD7fLFro; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099286; a=rsa-sha256; cv=none; b=dJ4zigZJf3XM32b7EwaixVFkZqb6DIF5CIj0C9ujYlLFNSyC94hRGPgfhE5EVkY0SDKr/l vyz2zGMPWa/YTA9htHNjs3lbJewvUBDJefsRSoXqPF+UKD7KsaROTEnRgFiRDtiGi6JuY5 wUoTHw7Y1jVphLIIROX7G9Lf4e+boMI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099285; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bb5tuF13A7AUAmGHrtoNc4Ue+Am7P92yBHtRqoRTTps=; b=YD7fLFroNyFqVpPQpV9bhU7cXz0UKPNui4ki5fRd6/AUX2iMP1NBRnkKUfPt+Zp6XGnudb +MxRzazGIZ1pF04wYtXwSTFFziR66D59/Ez41SVjJUs+aahmevPiu7+yXGymy9Ot2xLzHM 2eb7fOp8IUU9vXqkvBkUsrOmhfPUdX4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-558-1vALR20tPYeGhBQuRt7gLg-1; Wed, 29 Mar 2023 10:14:42 -0400 X-MC-Unique: 1vALR20tPYeGhBQuRt7gLg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3D8F81C0758E; Wed, 29 Mar 2023 14:14:41 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 59D114020C82; Wed, 29 Mar 2023 14:14:39 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 15/48] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Wed, 29 Mar 2023 15:13:21 +0100 Message-Id: <20230329141354.516864-16-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 221E3180026 X-Rspam-User: X-Stat-Signature: 5axtoqktpa5symyxt55us3kr3nwrhz5f X-HE-Tag: 1680099285-190796 X-HE-Meta: U2FsdGVkX1+OzZjapsbMVrfEsVXXkSwkkJAp4eigrz80gJN/EkpW/yUc6Cu/nK+Sdvk1uNn2/mJdcPLSskz0UPH5jU2fzBPtBeNU8Jx+4lYzTxVqlBG2YOtIoZqgEE3Hbqd51sAJxuosKWTS+gZM1fIPYfu1JZzr5I6Jl5gIXhl1B/pLja/E97uDtSyvFSKk5zO/4G6Ru4fWd1fbLdV6q67HZ5T5rd/cFqps45smfXBFkZcisCW3lkBqYiUZ3V2b1ucF0x9kfIUboPHbVDEq+ExU2Apy8pur/NxMeFCWdQ/8+tVPpCkHGgFuKRSl1CX6Mf3KGcgKPXVYWVI5nI9rOVhJ+mVjwzWGy3uorQ61AEMYde0fbmqrYXCx0R3ol+tjg67R6OHn4oM0UQKxWhqJTrDfHz44+3LdVnVqEAv52OI4qLQHD+I96PzQw5oLy67amFkR2OF3V5QiifmQnCTt1xEtZ+dMc3H/Bd0t3D0uJzZUjsS0V3W5w7unFG5GamQzS++8aNMVv52jP9QmdsgA2iflb9XNsJsU1ldf2WiiG5k+aNOdvxoSq1avAnlzWZIiTOQIy81yEbzDaf23v7ACLbE17pv+ocp8yEd7gxbxKeTD3Ev4qEOquIrnoU7diyHpJX2qtOp6MEJabd1vDwi5Koi9KFpWoTJitSKyj69UXrL0mSpFSnwPWa8BelVtYfVPVX7l6lL3IVeotvLoBr9kIL33yqiPtq74kDl91V3H3qNd6IMOrclDJZ3ehvPAIFNwYySKisYOHCBoBtNRzdMbrVrCuiS2f5Qwe3q27EqwKatMMnsbXQ7ZA0/7W+TRtcApU1bBWIYZXGe/lum0OJNt5vJUuU+xbW7DBEZhlHJQiZaZqq/z3z8ar/Vz4bveNXneTdxEveO1gy+prbVPdMNoWMsLEJ05PuJ7QTNyy9TjVg1lCxwmRkh/xYiJJgNz5kvr3eCGzet8EgWxDWkATEm Jvq1dAUs 6KFXyTjeas53no28UI9w/pS7OxY5I0empTUGvlTQhZZFqtIWdDAHVStmo4SURl/7tZNiRouPr1mmIJYu9HaMqWnIAtXtFIZx0o3Xxgp04/Hk0t+DZbsWxN5UyDZvpcCaYz3Ox0WC7RPLBX0WaczaHDsCVm9lENnqUv+sKp00IPUp6W24ZrnwT0RVPeBIb4nPxfrHUeRHfJSYSWh9oQgtPDiu3QhQBfFxj117GC1iXgtO7jOPtSVVNebeWMSAyKGnlGi/8d+QhtzshjlxhgyqEjmLiLFgxFkllmdjFh30klMwoYgAvmCTkhgk163GpmVqMMyR8Tk4B39R2fgkI07uZoDKZEQUogffGZUWwesEgOjhyzfN9/3N1thTQLSbPZzU9V+hTKdgNyL5S9wHE5byN3FEty2YaXs7AZMKRUJLK4/DoJoD0il861L2JCfdXI9seGOgr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 26 ++++++++------------------ 2 files changed, 8 insertions(+), 20 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index db9f828e9d1e..844bc8e6a714 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index a1c5a6d9419c..a8f8ccaed10e 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -971,14 +971,16 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = flags, - .msg_kflags = MSG_SPLICE_PAGES, - }; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -988,18 +990,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Wed Mar 29 14:13:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8192DC6FD18 for ; Wed, 29 Mar 2023 14:14:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28B4A6B008C; Wed, 29 Mar 2023 10:14:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23BB66B0092; Wed, 29 Mar 2023 10:14:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B4776B0093; Wed, 29 Mar 2023 10:14:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F02876B008C for ; Wed, 29 Mar 2023 10:14:52 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AE8AE1C61D8 for ; Wed, 29 Mar 2023 14:14:52 +0000 (UTC) X-FDA: 80622132024.14.42AD10C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id EFA528000F for ; Wed, 29 Mar 2023 14:14:50 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BJ86lvh0; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099291; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6eWwAqocQ917L7rRS0xkmog8pJjMU6goCC63rugY9Cw=; b=j3HocgsAuXo63rnHdDu5yOKD94iD25RjUrDD21ct9q6R7dT9qIXngGgFX8z31HYtGB2rCx X9mP28vRau4B7P65EMlenQn5QU0W6sop9V6CXBxonz5z8Py29JF+muHxpInCLDMyTYr0ot Uvb/zA4e2AT0Et8kiSWrtM3OxqaRRQA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BJ86lvh0; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099291; a=rsa-sha256; cv=none; b=FL4WhwmibIm9h1xzKxbQOkewwsQwKXHbdzOt/anuMbKChCAPa4dFyp65A7Sue91UDTqmZu 0q8zq4Lro7Y+7tC8olxY6zSJRIQH4ao2VgUBMqKLgi6SKJYYBtm+dxic/Y8DJ0X2v71Pcd IHSCEdKjp12yb2ug4OPjrDx3uUPOPsA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6eWwAqocQ917L7rRS0xkmog8pJjMU6goCC63rugY9Cw=; b=BJ86lvh002xKCUmtIF2GktHA3SrCwvaAADHjg9prqpBi3USCy3+vR1JyQOEMVJEKgHGNxh 4psEyv/R2WWDpmyhapuv6TLxF5fSz+L10F2qzUYYRhOPqCP313G6MXaVqZWX9TA6UGwyYl PJ/1fUj6RSBfrljyGTno9h+c4WnFrh0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-516-0sNPtnvnMki-Ezl96bY6_g-1; Wed, 29 Mar 2023 10:14:44 -0400 X-MC-Unique: 0sNPtnvnMki-Ezl96bY6_g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C96611C0758A; Wed, 29 Mar 2023 14:14:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CD48A40B3EDA; Wed, 29 Mar 2023 14:14:41 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [RFC PATCH v2 16/48] ip, udp: Support MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:22 +0100 Message-Id: <20230329141354.516864-17-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: kf4bara7cnjjkgiyedxrkob9qwcwiu33 X-Rspamd-Queue-Id: EFA528000F X-HE-Tag: 1680099290-613988 X-HE-Meta: U2FsdGVkX18znWJUwdPta6kBjMHoCu1XjsQXJojWzsZ3H6gRSdz0KkF7ytYuZE3a1997nr8Py7GOWBlZGu9fmv3u68UrjeN0ni6yW587Ta3EqZhstQqJiMcHzW8qNfrESBaaM7CFN4OtjvHvcI2A1kYxTlIofx0qG2KnwGJT0iH0fFJYtn4rDWA1Tf0wKuzX7fdTA/5B0mp4WDZMKhJr+hOG72Jx0Wvc9qcytK3TNymEAusZjfib5qKy/ScBBSxeA+YPRcpkT2/MEwk9J+4A+T467QwTs2ho6HyTt7+zVFf3baDo8vsHObHLfDbbsjuVMRV/evwXRttjWIyy8+wIBxuvSaDKP3noIItQlWNmrin29BYaVAf3+pfagOpLMyPMbxzDptVEf9111GXegZWAhldMbmVAuw0LPHbAXoWhLKCdcO5xkY9vR3Rev4qv3Xe2VjT4/pAzDwjusGV6GqeEwwBG0tnOzvPJJax+fUrWrdniSs2BiVyzdHY3EBjodA/PJZ+d/JSGK8zIzhD+R6FBJ5CYOWnccqtOSlE92gRVBPcDzgOtl5tN4m4aLSj/Gb1eyR6Nkhqj4Jz65d79FcQXgZWWYWoXzL9KEv9BoI+kLHQu89A1g2wsQaGwzatMOYK8m9M+eqPmMsm4YhD0X8F8Ak7VZXYlxV6/NRgx+IG1HkE+iKdFGArF5e0fFaRbO7HLOTIsdjeMgdwoXSPY5U1vF/4nps0lBEFBnmkLDLwjTEIHeo14vrGXJbQpK0wGfdml5J2DzwKhl9Y2He2bw+nSNWnN9NGumFXb7oMhc1lvwYuc23xLVyiBomEKKNujIVFMwSw2hBPwvJzRVA8q637LxlPeJ2xYD3K3bPkzQU9y6SPVTm3/qbbIIkonikX9+NRlyKdxJpEbHDo/dLMW0Ar1TV7oXakWzSu3/C8lvuDpg9i2mPsX00X2KzhSKEi6hSs/9skjr6PVul/2J+rSfTe 2AjMFExy 1px39LztpfvT8iOWwQxNj1rzUW1qEfNhMHK7e679GQwKGXlPh5B+NyuMytIMDJ3ebdJigxL66Z8vxyPIbjra65uMIikvhcW5hI8IPY5EPXIQLv/WcGXHpcq6q8SjvAkLMxRdTmDkXBgV1ee0jLxGG3MGtoVX4i7mIa/Mm6ejAntlxjBDxXe7QGBBfKDxOIVLXDIX1rtCEZjUFBMnB/vJUoB86BjUm3KXBPFdGKjLbOje1arSXJQvS7ZaXfNCGcv+W9f/P0+sWSYLerFhQEDJ5UGcLkzQKf5/bNJZJuugX934WjlRVoCanDqwfGHh2PNKmbx9gqLFoIOOcj1A+wMDNaztLe+hfRMnyVnMNlKbZD8xIoRg622hPDcUMV3XTtzk5bDGaC98zbrG3T+dPx3Zcf1ihlKdPqqtvxiWnMKwrlUWi0VfOvP8+ewR9Tuoy3dAtsvO8R9yGQHOqjZrMxHAGvTBzWvUH+m/wNiu3QIR51c1JdTiPj5Q+mnba8jANcS3AvdkvPo2z5+rnt4P0sijEZfHq3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 85 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 4 deletions(-) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 4e4e308c3230..07736da70eab 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -973,11 +973,11 @@ static int __ip_append_data(struct sock *sk, int hh_len; int exthdrlen; int mtu; - int copy; + ssize_t copy; int err; int offset = 0; bool zc = false; - unsigned int maxfraglen, fragheaderlen, maxnonfragsize; + unsigned int maxfraglen, fragheaderlen, maxnonfragsize, xlength; int csummode = CHECKSUM_NONE; struct rtable *rt = (struct rtable *)cork->dst; unsigned int wmem_alloc_delta = 0; @@ -1017,6 +1017,7 @@ static int __ip_append_data(struct sock *sk, (!exthdrlen || (rt->dst.dev->features & NETIF_F_HW_ESP_TX_CSUM))) csummode = CHECKSUM_PARTIAL; + xlength = length; if ((flags & MSG_ZEROCOPY) && length) { struct msghdr *msg = from; @@ -1047,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + struct msghdr *msg = from; + + if (inet->hdrincl) + return -EPERM; + if (!(rt->dst.dev->features & NETIF_F_SG)) + return -EOPNOTSUPP; + xlength = transhdrlen; /* We need an empty buffer to attach stuff to */ } cork->length += length; @@ -1074,6 +1083,50 @@ static int __ip_append_data(struct sock *sk, unsigned int alloclen, alloc_extra; unsigned int pagedlen; struct sk_buff *skb_prev; + + if (unlikely(flags & MSG_SPLICE_PAGES)) { + skb_prev = skb; + fraggap = skb_prev->len - maxfraglen; + + alloclen = fragheaderlen + hh_len + fraggap + 15; + skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); + if (unlikely(!skb)) { + err = -ENOBUFS; + goto error; + } + + /* + * Fill in the control structures + */ + skb->ip_summed = CHECKSUM_NONE; + skb->csum = 0; + skb_reserve(skb, hh_len); + + /* + * Find where to start putting bytes. + */ + skb_put(skb, fragheaderlen + fraggap); + skb_reset_network_header(skb); + skb->transport_header = (skb->network_header + + fragheaderlen); + if (fraggap) { + skb->csum = skb_copy_and_csum_bits( + skb_prev, maxfraglen, + skb_transport_header(skb), + fraggap); + skb_prev->csum = csum_sub(skb_prev->csum, + skb->csum); + pskb_trim_unique(skb_prev, maxfraglen); + } + + /* + * Put the packet on the pending queue. + */ + __skb_queue_tail(&sk->sk_write_queue, skb); + continue; + } + xlength = length; + alloc_new_skb: skb_prev = skb; if (skb_prev) @@ -1085,7 +1138,7 @@ static int __ip_append_data(struct sock *sk, * If remaining data exceeds the mtu, * we know we need more fragment(s). */ - datalen = length + fraggap; + datalen = xlength + fraggap; if (datalen > mtu - fragheaderlen) datalen = maxfraglen - fragheaderlen; fraglen = datalen + fragheaderlen; @@ -1099,7 +1152,7 @@ static int __ip_append_data(struct sock *sk, * because we have no idea what fragment will be * the last. */ - if (datalen == length + fraggap) + if (datalen == xlength + fraggap) alloc_extra += rt->dst.trailer_len; if ((flags & MSG_MORE) && @@ -1206,6 +1259,30 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + struct page *page = NULL, **pages = &page; + size_t off; + + copy = iov_iter_extract_pages(&msg->msg_iter, &pages, + copy, 1, 0, &off); + if (copy <= 0) { + err = copy ?: -EIO; + goto error; + } + + err = skb_append_pagefrags(skb, page, off, copy); + if (err < 0) + goto error; + + if (skb->ip_summed == CHECKSUM_NONE) { + __wsum csum; + csum = csum_page(page, off, copy); + skb->csum = csum_block_add(skb->csum, csum, skb->len); + } + + skb_len_add(skb, copy); + refcount_add(copy, &sk->sk_wmem_alloc); } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Wed Mar 29 14:13:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29215C74A5B for ; Wed, 29 Mar 2023 14:14:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7461E6B0092; Wed, 29 Mar 2023 10:14:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F6346B0093; Wed, 29 Mar 2023 10:14:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5490E6B0095; Wed, 29 Mar 2023 10:14:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3ADDB6B0092 for ; Wed, 29 Mar 2023 10:14:54 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 08334C087B for ; Wed, 29 Mar 2023 14:14:54 +0000 (UTC) X-FDA: 80622132108.23.10DF25E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 3950DC0021 for ; Wed, 29 Mar 2023 14:14:51 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FSI+KyXh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099292; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vWPvILWC7NBdevnbDDsys9XJnr3/cfzKJoNpaM2CnAc=; b=ARkd1dh95vZlSeJyKzT5Nsaq5nYc5Lf5AQCJkmlToYs1roV9vJebrIZUSn0Gp0SXZ4g0Mt 0d/s9n55IFB+7N4atrI68h6l9a+h1MYZaI3mQomPwBc3nJinL4viEF11PUyWnDGNkUvxMZ 7yc2iXdxhAN0cnF6ydr/EL5cnqCf3Cs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FSI+KyXh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099292; a=rsa-sha256; cv=none; b=t/QstaGfDbcA4S0p4lQ0YIrl+IGUcSnu04wc8heBmWkombBUODsOMOpG61qpqfutM3JWM9 XAfiY3YHvl2Oxh148tnF/aoO51T3uz3Y4pH9lkfssHQoZqNrHsZCqKMBstSN9f3sI+SnCa I8CHrSNuym9hkb072/ZcmRl/r9y1A7E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vWPvILWC7NBdevnbDDsys9XJnr3/cfzKJoNpaM2CnAc=; b=FSI+KyXh+2/YdW6/xxN7eVFCkMQANmgjZXHTwByL76B37RT5IZckHDYMGvBbemr/Av/bcI 1RARGV8IIwGWmPsl3UZpqTOTkGP0HkxG9q9Qe1YDEqolAGnDrhF9U2RltVeblcXqm+BCZQ iKN2aHNpOFJUjGVeYHj4D6LTwkWizwk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-612-2VMedOL2PkqYySq_Hlwhxg-1; Wed, 29 Mar 2023 10:14:47 -0400 X-MC-Unique: 2VMedOL2PkqYySq_Hlwhxg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 76C28185A7A9; Wed, 29 Mar 2023 14:14:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8174D2166B33; Wed, 29 Mar 2023 14:14:44 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [RFC PATCH v2 17/48] ip, udp: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Wed, 29 Mar 2023 15:13:23 +0100 Message-Id: <20230329141354.516864-18-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3950DC0021 X-Stat-Signature: mrqtjjtrberhftaqarrjzp671mt93rs6 X-HE-Tag: 1680099291-410495 X-HE-Meta: U2FsdGVkX1/ie2PkAqjkcs2l/+OYGk3bbm1aN/x3hTdJrmSMlVvKmwghLXLBaWbjRe7gNp75evQ3UtMo5ddKe9ud/7zF4gxsgPdDb1fxgGVyMlskXZknsZVs9DmcMGnfoQaTNI4CRJmxVFEWl7+Pr48mPlPUS4e0XnlmfNue2+mJ2tbwanPXa5izj0TOISlsunJXV/JY8WlvdO1CxDVii2UaKI+3nxR0an6s+X+8c1lh6QQR18IZ1j0RDnV8k1Gj+wCsjbzp+kkDOHq8Dg3cm9bFBqJ+fDepmGBEpbJlCx7KnIztckHpDveyO+AKPbYM6uOOP8G1pWM1GR99fwxac/1PyWfFNwxHSkWSEcdAxDfRO6Cpsq5sCpM6AHTGHvEzdIVgznQOb6/anpPZMD1AzJHL2G3Kb1+w35tqdrnxkf87d0c0zbwKBWteQL1Pz9hsL/KoPnhXPlSNCKNRkuc9vJNXw8QrA9TKIBzPV3FaMCAoz42I0DTh21FbcdmG80wwFpB0yGk4YmUsOGURxIuBfPtEAzVmMyrmOrrBVK36fClc+K8CxDfo9dr3kzBMrYBuGUKMztSQ2LgFufRZbuit7aUU7gxzUwewa2oJzrt0zo9cmIQfGVwqoQ5o0suBq83gJ4NzgNc8A3LC6zaAawPRKRSjLCF4zJZxOBvISzk/fb0tG2fMDfb85Ce6V2GOFVovZhQftxqlFh+hmQBhpRnTLhFhnq/KG90o5IPhCP49N2IvaZbvTyUY9fsWi6k5mlQnXYWAOuf1Ds3BjVbfd7gAbPuUYOqUS9gwgNxaAEV8ndW2dj5b1w11P0aodFlUKEHeGbpWCduR9OjRM0Mo4cMDJkX6j6RaJkamP4SbYM2Ok2yYndoHbLXJjTai8k8gcXJw+yA0tILliNM3ShqBcqLAEgQuEHmMhr4EONE9n43Im+oM5eErGt37+uaAMPNYHtCjE/lvHS/HZhzeKda8Ezt dVpUKzw4 Jj6cE94QRgCYAsn9Nbk5peYyklgFAH56EgK4Bt0G7IUdU9oT24zPBQYZ7HauSl5ZFiLggMsOL90EGPU2Y/rl1yuurRRQNz/fYcZZBBKUEw1IeY5BJLsG5EACv8Ld/oEUUFEOGD0bwkpUQpyUlQIGIa7GKaiNxu0mD7Z0vibGbCJ8nOq4Hu0xalroxHUPmJ6MeSfZ11N2buxpBX9eR0A/BxPHBQY95xtUsg/xvB86TyzvFIor+6W2nOSwex+xyCUie6RqZ9FsxDyMW2zqLz/JKK/Fu7z+kATE9Ik+tBv/T9eu+mbmCRd7EVwMU71gXBPmXIz3+N3znabPN3mncxv3MrfxDaSBeZizTXYrwF8CB63ejnaVH1swG6Dw7qLt2XkTaMFJw7/hpAACwa+PD6Xp4oQ2o8cNXFT8KxxAsvpo/JEtzI5HsjhnMNyxAvHhcOGhLbyIsF+G526xjMiEJBMuKBJTY/6FNpnTAz7QLFCbfbbQsiyEfFfWxlzkTUPeVqHxsadyoBc6Gm/kLpGyO0dHipysSng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make __ip_append_data() copy it. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 07736da70eab..e4aeaab704c8 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1263,6 +1263,7 @@ static int __ip_append_data(struct sock *sk, struct msghdr *msg = from; struct page *page = NULL, **pages = &page; size_t off; + bool put = false; copy = iov_iter_extract_pages(&msg->msg_iter, &pages, copy, 1, 0, &off); @@ -1271,7 +1272,25 @@ static int __ip_append_data(struct sock *sk, goto error; } + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + err = copy ?: -ENOMEM; + goto error; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + err = skb_append_pagefrags(skb, page, off, copy); + if (put) + put_page(page); if (err < 0) goto error; From patchwork Wed Mar 29 14:13:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DDB7C6FD18 for ; Wed, 29 Mar 2023 14:14:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 438F16B0095; Wed, 29 Mar 2023 10:14:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C2DB6B0096; Wed, 29 Mar 2023 10:14:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 289966B0098; Wed, 29 Mar 2023 10:14:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 11F8B6B0095 for ; Wed, 29 Mar 2023 10:14:57 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D96A7AB440 for ; Wed, 29 Mar 2023 14:14:56 +0000 (UTC) X-FDA: 80622132192.18.527774D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 389444001D for ; Wed, 29 Mar 2023 14:14:54 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i0U7Z07w; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099295; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5bgiBObt8qBauZcUWYsV1T0v3n3OqeoexntQtIJIREA=; b=zRIfpJA02wZ1qto1A2DSW5z9bVvLIoviD798KYLDaZ4MTyWbyBYir2aaWOcbeyXIkH+Onw 4VOvVUrSVx6L/v9uLKHGIIcnhaKTgiaUiK74jSzKMwTwzSwXUWLnNo6ZhWrZ2jYZionTxb WhzN8awIANoSQr+ZIZtr9D0UoPd60zw= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i0U7Z07w; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099295; a=rsa-sha256; cv=none; b=dV7UiyLJYit4RRSGbvCMFApzOeMzmN4zP62uXZJlswELvXrs4dUv+Zy1Twpx28G6si4lsp vJ5eQXXn/mB0ND9/1dn8tzpMZEXunR5AuFj37G4dB8DGQwlPzWUTE93nOhQctbz1ftsQ0l 3JVJCXkPV+EKnCx+BEDVXaEJsw29s5o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5bgiBObt8qBauZcUWYsV1T0v3n3OqeoexntQtIJIREA=; b=i0U7Z07wk2ChCxv2RmIKltInZv5SBbsqvC5Is0Q4FH2+oZMUErGI8UnxLpz4otURL7gclh g2T27JayIp3/2Bk1pqHafBLYpDUTBg7yaGOESRS8BM8Qx3iFj66fyy3ZaGWaWzGcVGj72W GO0iIBJlJ0UxYG7g0U1eBC7SHzjDISc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-619-sSXFfrUjOyG56ONSjKtQ0A-1; Wed, 29 Mar 2023 10:14:50 -0400 X-MC-Unique: sSXFfrUjOyG56ONSjKtQ0A-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0AB0B3C0ED6A; Wed, 29 Mar 2023 14:14:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 115CE492C3E; Wed, 29 Mar 2023 14:14:46 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [RFC PATCH v2 18/48] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:24 +0100 Message-Id: <20230329141354.516864-19-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 389444001D X-Stat-Signature: 1qo8imhckxxfnqg1ymoak4pxapua5awt X-Rspam-User: X-HE-Tag: 1680099294-503410 X-HE-Meta: U2FsdGVkX1/RQdFdG2ssbAEc2DenuQrLv+d0hiuEt1C2Ab3TkfepP1ovRrbYZnqXrpToObWaN7nHLuqp6GcTTkyeDGzqsVJDJWWQmJJ58hS1HSJrlxNPYA3S3RxwE/rtP5xbVrU2/zt1vm6zHipG8H2ztQkIcCmV39ekkEVa6FfBE3EWPaypnPn2D6wDLG/45xbVwSQXeRc7Up7CWhlIC46Jw6GD/c1jgpLLJ3re5zXSWuJqdV2KeCKNvmB1lune1ZFJGvGZoHacuuSbycx7sx8y30KQ+T+Jh2wKGQOyYxQQaqYIV96NWJpx3IWdoOp30Mjq/V4C/X0wuyb0FVzW8YBA7pnci+2DCGppTfMwgAIIrFWtIj7sLqpEbsr3DTWFaJtLF1BszVtTB0k8Oiv2o6We843h+QQ0/8lNHhjzEMPyryfDb9CuVdUVkgeHC4nzAtD53Q915h2+D7ltacRpuM1LNvVi0FUB42bAKFBHeeNVkO0mNH71vbHBMEDiWY0PKdX38mTnMizagLEZ/ndHrZk8Au4Teu0cTlVzYaUYg0gbiQKQY9w48edXeBhbzYU8RXOE6J2OW7LMTStbYQMepJkhrUqFG4wI99ga/SVOufAn/FqEXa6GipHAlWEZ6FjxH0D7Q3dcwby6hKW32bzeIvMfgRct391FnlAvoqY5Oe5lTWZrUpY3UqzGHHXffaPuMs9O6CJXS7lMbeP+RRJk8jvYDC41aiw1txNMT/66smzxTwXg9QWrdMcii9bn8NfnN9/vXESQuZDnC7EKUwCQoo+NIwuaBKHqAuAWd9Lm9PnNRpHTQR0fE+9GDxQH7iN1UwcMiLieUSlp22U87iF1CFTKExxm1bHjMWw3r96WvQZ2c+uISvq07iN1NqecSw579sxhhuxvrHPqZAtYHVEa3w8jYu9uRccawXqNv8dNpCd8eNforWbZHHdAJdsjqyEy7CMxvuejG/niPZbwsiI juXEokDH zIHsu9h74Ifn0G8onmoTZcQxdwQAMiznwDEkIGjVjnkZIVo3AiZq6a03bAKuRKx/wb7nrAwkiG3ZuslPHbnEF/+FbVZ3lOQzyH6LQ9aYf12HhWH4Eeiha8mjtTAoxav7DbxHzImrQjCBGDusMeZ2IJa2KeZggJ3taoLJ2Kuqpynjv/3vZjJPu/G8R2tIIjeZM6kY6xVAwNR4P1yPIIt4lSLKyvuUsmg5BGTxybzsUtqJzMzZc1/9zNzfuCRnhhYxS6AtoBfPSEhNClsM3q8LYDrQGN/ZOOLWn2EGD95ZijTXex8XMxQCQyPDlwedz316tl/Mz0OYs3oN/UEyKLIEY/Rfd5A4NvWhje5rMWgK/rc0RrzR/R+OkSqN2m9yEi5QBENGs1qzkiMbduKFIX/Y+jgL+31eCzOgaNJiELfhOPM7tsC21YN2IfoZXi7HVCQGwk6vXS+sqpXIIPYak/284ATuBjub4HCgYoyz4GQSTWfcq+xGK68vWHFvF1R+NS5l26U+4Txo72beXBCdc3xgMaRmJBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/udp.c | 50 +++++++++----------------------------------------- 1 file changed, 9 insertions(+), 41 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index c605d171eb2d..097feb92e215 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,52 +1332,20 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_MORE + }; int ret; - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; lock_sock(sk); - - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: + ret = udp_sendmsg(sk, &msg, size); release_sock(sk); return ret; } From patchwork Wed Mar 29 14:13:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C89AC761AF for ; Wed, 29 Mar 2023 14:15:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DB296B0072; Wed, 29 Mar 2023 10:14:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2638D6B0078; Wed, 29 Mar 2023 10:14:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B5DB6B0096; Wed, 29 Mar 2023 10:14:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F01816B0072 for ; Wed, 29 Mar 2023 10:14:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AA92D80D45 for ; Wed, 29 Mar 2023 14:14:58 +0000 (UTC) X-FDA: 80622132276.23.FE60A18 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id AE7B240016 for ; Wed, 29 Mar 2023 14:14:56 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YIZ7FzDz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099296; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NG18J76VRrzhfHMvUgJjpISKtrp6GX9XUZPxH3Nxq0I=; b=ZGC05cflF8WbpeFTKJ17iiZeaV7XidW25YbhXrI10/0QyH6EVHBTphFvCg/EZ7omUIJ6Ls LDO7RNAPOZyVx/LKxuskdfMjHEEss1AG0dt+jXJl7Kdi7HM2IdrfTFEtS2lz6JK5GOYJdj WiGNKeByhiheClJC93X0Vk2VvD6K/oY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YIZ7FzDz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099296; a=rsa-sha256; cv=none; b=TwKFf5Tm6jSwynHDWIu2v02dDacNQwoHoAfDJ36WK4RQCPFIvioL3Ty4mialxoRX4/pbAu PmhCLvOE1VeeI5EiW5h1jEaO6JGw6KsMs1bmwgmnxc5ZOzPTm9gJMQnQXLEDnKJkTDMF8Q nvyEILmg4tWtY5vkP7BREgzM49RVOrE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099296; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NG18J76VRrzhfHMvUgJjpISKtrp6GX9XUZPxH3Nxq0I=; b=YIZ7FzDz88YXGKH8ouR2Hc/ZYDNlBj8XQJCHzUy+bTAhpxz4JCwz3BeU72+qIUJ1HhtRN5 YIO9x2Uy57aEIDfCtS9uXs5dwUUpAinSECht+r3/4tU46Yr9TETDCFW/dQ1mQfhb8ZW36z Zkrq4Nh60aeCByzpDeRj2xIFapSdAkU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-310-xuaB0LLNMnyvhGqA--dfEQ-1; Wed, 29 Mar 2023 10:14:52 -0400 X-MC-Unique: xuaB0LLNMnyvhGqA--dfEQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B50B1C0758A; Wed, 29 Mar 2023 14:14:51 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA9AC2166B33; Wed, 29 Mar 2023 14:14:49 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 19/48] af_unix: Support MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:25 +0100 Message-Id: <20230329141354.516864-20-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Queue-Id: AE7B240016 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: fpkfm7m7sntabagu9wy3qia93ocggqi3 X-HE-Tag: 1680099296-535902 X-HE-Meta: U2FsdGVkX1+ca5weKoUlmUH3xrkgNp/4pWxIX5goJmI+Sg9u/yxLVaCGUncNZbbFNcHGoLgc63CiEKZQOOY4/40yqR6Ei1yyjDggcbgQpbwKxCcS6mNuaVFkNCmM2STdF9V9SksTe9AYvI7kRLVRXLjCTIEBzNm4ZeUfL3xHOVmN+sS2QQt5uxKOELyyYdYjXlKtqiITylyw8zlEm+Dd/XTl2AT7wOG6uEwIyq1grdCwgicqCJw3obTaN/rWf9qg2bcH8Fgod9Vf7Zkx4j33ZVyP2ekGEo47LruPUDNtsY9/MTAorTN+JN1pgbhmNIf8hXF1h28uXC31FsZFmKAqK43iEO3bH+InrYQKIvgc3HC6ctBY4Ros5c/GdTbQohvFxXuIhqKmN95dQ6nrdKqJEDcT3HhqsJIyyLcTuitumqosFm2EdJGkq2R8+vD4DH3hm9jI5L6swF6hMmyupNcClGfu1DnN3Vc1j0asKNk4rddoksUUe5smL/varopHaho13vaLpTjGKF4xY5v65wV7yJyJm/++JqH4jZmyTBS+nWVW22ejTqSnYNsrnQ90bsulSVZdjesdS6DRofWFH7ajQYjUvYgSa5cKnVU7pF/ZW9kvGFNTHLdc7LLclq7fsa6nRaxssF2yfJt07o41zzcgly8Nfz6KlwENq+K96+GjaITrmnJPFIYb/VrEfUj66bJ4zhprQRj+Bap8tBgJnY4clzWL7n+hbunrXohLaPh5+oqMLe9I1TbjN9OtQp0TfOtAgUNHzC3cty5Op7ybnkIg0xj03HOVvGxKjWgxm2rj+cLO9iPQkuTeFa7O9CWfLCmXr7b8aLlEG52sS3UeyHnzPnVgBpHfa7Z6BkaPAs8lhiYkxVv+gGNxcfa0H0KQ6PcJHRVA8TBHoncXE1dhjga0mPniEG0zNkev5tcc5l6oJTcRLqCzu+xjAxZSKLMPdXhtrvBkmGUf4WziyuYrDpW TzzxI3h5 gwkzmtwsywyfZtgMV2Pgi0BTge/RFC/QNyWIhLj5CL9djmNtnDIv2Z7VaTrNljEsPczE6u1YolulMpeSHD6aj7+eL9mlJmaJjEKwQ3gnOzKUATfg2CVolTtWct8skVrNDFysBogNPBG1cGjOOvMyl0/0ZNlXj9Q3KWgs/7qnrtOU7yJsx0yC5UWmiy3hZgQwH5ceoGEta0Tvd/Wjhp+pmAXJvS8q82HpcrLtbIgiooLr4Zn7YY/UJVgNi3Fu+oTBeQ4L/VIBLLNewhA5UajPHbV+xyOCp4u+EUeg035zjFE7YqW8DMpuRLI6gIg+BdxUc7zFVCgr6dTohbvMcnwkzjhyaPPJzRl72Q93seto7JvaWOJvITcTedLENsddUoxFSydJekuCvObLNWhzgv5btPiZYpmOAHAD9PvxbxKwmK8bIlTwhRUi9wmc7yGdAJmetWqQE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if given and if ITER_BVEC and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 93 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 347122c3575e..84a0d97f1aa4 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2151,6 +2151,53 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other } #endif +/* + * Extract pages from an iterator and add them to the socket buffer. + */ +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, + struct iov_iter *iter, ssize_t maxsize) +{ + struct page *pages[8], **ppages = pages; + unsigned int i, nr; + ssize_t ret = 0; + + while (iter->count > 0) { + size_t off, len; + + nr = min(MAX_SKB_FRAGS - skb_shinfo(skb)->nr_frags, + ARRAY_SIZE(pages)); + if (nr == 0) + break; + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + if (!ret) + ret = len ?: -EIO; + break; + } + + i = 0; + do { + size_t part = min(PAGE_SIZE - off, len); + + if (skb_append_pagefrags(skb, pages[i++], off, part) < 0) { + if (!ret) + ret = -EMSGSIZE; + goto out; + } + off = 0; + ret += part; + maxsize -= part; + len -= part; + } while (len > 0); + if (maxsize <= 0) + break; + } + +out: + return ret; +} + static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) { @@ -2194,19 +2241,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2218,13 +2271,21 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + skb->data_len += size; + skb->len += size; + skb->truesize += size; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Wed Mar 29 14:13:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 359EFC74A5B for ; Wed, 29 Mar 2023 14:15:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C75916B0078; Wed, 29 Mar 2023 10:15:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C25FD6B007D; Wed, 29 Mar 2023 10:15:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC6BC6B0096; Wed, 29 Mar 2023 10:15:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9D4586B0078 for ; Wed, 29 Mar 2023 10:15:03 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6EAF816067D for ; Wed, 29 Mar 2023 14:15:03 +0000 (UTC) X-FDA: 80622132486.24.1BA9FD1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 9D337120030 for ; Wed, 29 Mar 2023 14:15:01 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ikqa8NT+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DuSpRIzeQZ01nwpBuy8LkLTaj8h3sZ+W9+uqSPxJMKE=; b=I6VJPUOtHOZ42217q4jDZbFZFqJUaSeoyboHk5iiM9TtEnhjtJ3SIapzjX/0qbht3dM0xC NiE4MukQwXoc5Z/No7pKeQ2xYZOG9PjwZjiO1ozQLOUt8jL6rIkkDOBeXN2ffBKiMtrhiE n/+GXqxmb8XEx3GUUFo76CmA8Sgn8SQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ikqa8NT+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099301; a=rsa-sha256; cv=none; b=HxDrjludY7oBe+iWzHDg9IoznaUlo3iDM7ZQ91vFwl6o7zIpdlTut+MHyMoLbwNxiOPO0j wL+cHzkoZdwC27wqmg2UvzmOSOngYkeeZFV6SCfBedJ+EtrpzL5KcwTusvZ5Gjb+fcVEzw bX/DylAGbVCjote4d1/OLdvLni2KXqk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DuSpRIzeQZ01nwpBuy8LkLTaj8h3sZ+W9+uqSPxJMKE=; b=Ikqa8NT+S3ldFr78WGnoKdiOQKW7zATeeRMFipjl8QjhCht1bqpTsyNlBFLiWK+6HvNOCt L4d2XaSZVPRIVEdixQB3IjvgViZ1906LAhAzIQYiMCZ9MIm6HtKxZpFe+Q1jg8TEGemang taDF6yWCz2rQaRPev8at8UfadMIw8ms= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-306-jnNXOk_ROhmWbX02B6xDmA-1; Wed, 29 Mar 2023 10:14:55 -0400 X-MC-Unique: jnNXOk_ROhmWbX02B6xDmA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 170BE85A5B1; Wed, 29 Mar 2023 14:14:54 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 35746492C3E; Wed, 29 Mar 2023 14:14:52 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 20/48] af_unix: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Wed, 29 Mar 2023 15:13:26 +0100 Message-Id: <20230329141354.516864-21-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9D337120030 X-Stat-Signature: hdfb1oikfp5ufj9pk16g7jf8b1ezzthk X-HE-Tag: 1680099301-977577 X-HE-Meta: U2FsdGVkX195b6B+bkMvD/dAufteMgpRI4h4NUinpERiNkXaiv9UYNXeBvYCPxfe4jmeCA8x/O8BMffD5a/iJUX4SLUzivu8A76xWsvZ77ZUnUE+QQTbdbafZbD6bVnobdMhrQ+Ymna6eV79oWboKyV5arHNZHPYGK2NZomPwAxh06NoRNXd0PKVQ8IoxnUauAxObMM25NgzDvVY1+zpz2k+e2gVJBI2gOfp2WBHc/HTtGD1yWnoX9nXcksjLp6vEq0BevuPPjjGTRx4OM7Q41uETJPMaaxLufGKK9Qwsb0+r7lzcee9VLxbA87MGcD7D0KWbI2FltuF6ANiaSKBUEKeRadKxEnbK/MSmFKbw+HF1CBAsEe2Gre4rlhLIzISqIi6egxbic/TxtHI5NxZAffsY4iPq7hAGTaxjB7mHatzMmAjiCs5zFtLzOmi+chQirJw6QFMlhlAN59dcRm9WfUbDCBAVbhX++6SQxTXYPjYT8Vq2suZWmiiL99hbjkt31gM0zrfsffSqsW2k8UWNXQyMqagpCrZrPqCThJ59quvDSCuu4u9mbdN+3Exs1WkMQZVa+2MuEcNTlB1x7tSO05dv5G5diIPN8p1zv0SQgmY1y8WrOluzy8YKpdkYXelSTFx5rNd2P78KhJ6edqxOO1PZyKneKDMV/jXqUmiLn2oddaZudZSrpc7LG5PXaRb4a993uWpx5vLcNeJ/HQEEq1BtQGZsnqTJoMLmy2We8levpGT/JXxAc023p2WnsiIse9d3FaSumj6doLeEKPB/TJV47kmo3iR/+LYv6o95/ucGCpd9dE9TVR1cBljofDEnYQiiuOqktmCWZfVZB0+jA8fsK7ZB1cDQu+JJmY1kTJGm26GKGgtonBDSVbTGB359oC5dWbwkVgtvxVXYKMit6ex+p6Lu/srTjWoTOQFFLfUDcU3aTVNLqyI4ZkpV8AWjmr2k4b9tGfZoc9s8d+ z71I5/Gh J2Ot+VXRf5JrA8QedfHIHyiAA6oFRJU0AwU+XNgwgYIGAqpDGkTbULQqRs9ue0UC1vCaAyBrag23tIAueiV0MhIU378jkUHCNOK8lKpzhFRn1zavtW7VGFqOI7ydsbFf/UJx1cqEfc6Md0+VFtMTeE0yYM1NbUzLPXHcSWs0ShF7DTdHzItYzRSMPZ7+eHrWoOkdXJWjiuhHPzOKJg3p2W+jUESztExQkRE5lRBD/vFDyfLtGMe7dwvmx9OOxS56uygDlXLTmPqCprAg8xaVneeliatq33MzCXUZOqCNKEo/LrPYdCdOgFR0bidXdHFHtrdbWHG70++K5rszt63RymDru+EQUx0lXf4OEh0dAR+9blbxlo8F42E5b1JRcKc31cqrhIXQMyV6PomD7sBx22BHICpJ97At4soLb64zLI42OVuHnmXqI/ewirPpvhn72pFcOaQJWOdMQmEhd5TWb1PB82AgyS/Oia8ulrXO8yXEBirY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make unix_extract_bvec_to_skb() copy it. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 44 +++++++++++++++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 84a0d97f1aa4..b4b27a652ef0 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2154,12 +2154,12 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other /* * Extract pages from an iterator and add them to the socket buffer. */ -static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, - struct iov_iter *iter, ssize_t maxsize) +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) { struct page *pages[8], **ppages = pages; unsigned int i, nr; - ssize_t ret = 0; + ssize_t spliced = 0, ret = 0; while (iter->count > 0) { size_t off, len; @@ -2171,31 +2171,52 @@ static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); if (len <= 0) { - if (!ret) - ret = len ?: -EIO; + ret = len ?: -EIO; break; } i = 0; do { + struct page *page = pages[i++]; size_t part = min(PAGE_SIZE - off, len); + bool put = false; + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, gfp, + ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(iter, len); + ret = -ENOMEM; + goto out; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } - if (skb_append_pagefrags(skb, pages[i++], off, part) < 0) { - if (!ret) - ret = -EMSGSIZE; + ret = skb_append_pagefrags(skb, page, off, part); + if (put) + put_page(page); + if (ret < 0) { + iov_iter_revert(iter, len); goto out; } off = 0; - ret += part; + spliced += part; maxsize -= part; len -= part; } while (len > 0); + if (maxsize <= 0) break; } out: - return ret; + return spliced ?: ret; } static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, @@ -2272,7 +2293,8 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, fds_sent = true; if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { - size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size, + sk->sk_allocation); skb->data_len += size; skb->len += size; skb->truesize += size; From patchwork Wed Mar 29 14:13:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0046DC761AF for ; Wed, 29 Mar 2023 14:15:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C27C36B0096; Wed, 29 Mar 2023 10:15:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD8126B0098; Wed, 29 Mar 2023 10:15:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2B006B0099; Wed, 29 Mar 2023 10:15:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 91D456B0096 for ; Wed, 29 Mar 2023 10:15:04 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2C0FB40CAB for ; Wed, 29 Mar 2023 14:15:04 +0000 (UTC) X-FDA: 80622132528.15.688E269 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id B08624002A for ; Wed, 29 Mar 2023 14:15:01 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LepUYK+k; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yzXpQhTxTxrOyN4QfDXBN8fe/pZE0zPIgWzTS/9cMoE=; b=nVu5lVdll2ahtO6f2JdWrNnPmXiq0cNiX004z77IazzF4H0g1ru+jJCx6zTaMKV3Yj8ukC j/9mf8Z0xHeIKafKpdZC17WA6GL/YYXBB6wtNkAI4AWkdjLV8oEimSgzD1063Gr2OhtpK7 JJGSEq4sycJXmrpsjQn6Y9qwefPgMXs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LepUYK+k; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099301; a=rsa-sha256; cv=none; b=oE9y1WQcbk3uf7NVnuFMSjQbW1XSJYWQrUatIQucZEp+5WIXb8KCJgLzUuHHEvTin5T4gb D9I5EKHg+9iyGszViWI3wBp3OKzIRM1lDU6VWZhAd4ACLIWjIeI1R+wST2ZJmE7iXNOBgy 0Vy8keHEMjqRY9cx7s4zF8G7sQmtl7k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yzXpQhTxTxrOyN4QfDXBN8fe/pZE0zPIgWzTS/9cMoE=; b=LepUYK+kPQDXpRhkKeiZRAWqf+5w3cbZsmcWeIiJJ2cqw3zaMYmbfI53q+bjS8baUB+LUk 17SNmNnBjc9AAUHnfju3fnJ2V5Z8mbJ31Ukc7DcKzlAlkrifMyrv5pm0ieBmoqJukW4YD9 JwrQr2pVW6Ra4ejJ8TXRvZBXwRQ5L6o= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-445-ZVu3CVMuPAy5XfxUPzoW4A-1; Wed, 29 Mar 2023 10:14:58 -0400 X-MC-Unique: ZVu3CVMuPAy5XfxUPzoW4A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B9DA11C07588; Wed, 29 Mar 2023 14:14:56 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A725B1121331; Wed, 29 Mar 2023 14:14:54 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 21/48] crypto: af_alg: Pin pages rather than ref'ing if appropriate Date: Wed, 29 Mar 2023 15:13:27 +0100 Message-Id: <20230329141354.516864-22-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B08624002A X-Rspam-User: X-Stat-Signature: bghkcs3gcjjtz9r5mt1xpdjakuemizrd X-HE-Tag: 1680099301-905799 X-HE-Meta: U2FsdGVkX1/g3RvSsyuIYxIOBfgJ0HBKZCmUun2sDO/lgCHTvVpcoBHqPAtXjbgWrHEj33xTucee538A0qOGjaLBovVmlDCHEPaz88l9u+5M43UM+s5f0CCSqkds6oIbeM8D50NNe5o2OOmoXPq2kGoZk7nJZ4HizhkKVYSCBD0BBXeWKvQbtVKAR3RPkzHp1acmCLoFCyepZ5+Bpl0dqnYnWLBqcLlzg9LjzSXCdXnNq+ya5FXJMjmapgiSvIw57sZwydzIiFYS7GUiwkPoaMVcXcPhyPcv3ZACoMaP0r51JyZG7QCJsztKt1mV3BEuqe3K/pFJwpJE2Ie3voKhZuIhtLNSean4MNYGo3J7q6QAH9H4jlEF4mbohmYYsl1prdiL9Dn34Br4pOcfJ/eEFwhtb49TyhqegKHzZAalXJq30ctYfZ+U7oXL9a2u5qrgXgpEWAuzxN7dTpBssBGNC5xLfDhfmcklbZK/6ZxygIK/qmZ9kzxfMfIl6YTSahenpiYruDe3arQTkgVIwRauNkNhiQidQM7k1oJi6hFD6zrJHOija4RiRHgxmNAyOXYgcpoNMbICvAgTBKOJzPYH07WPPocw8KlIb2bOTtQQau3M9QsgJJXpGOo6bGo2l/uPjO+n/osb6NyHqqyd3/jTX0epeeCsBItn7Aokv3mt67cT24Cx0yJKg/oiFrwCIgXTGQw1FzewOHiEOti1rS8Bx/Ufhx9ewFeCF2rZDUriaDEklQsd0Sc+cH6u9/9VbQ2lfCMJr5xuuRBqxUqDfZp/DYojzyXjyEatu47r40hZUeXlVUQtC73sB/K+wtQm2QMhdDMiPzJDqa3NLXMw8eo8VEKvjOAPF292qYt+jMvlM/B7T8cFSOuH8HjHv7b2ORwsZNcncbBAzxRHcebmgWoySLvP9fqwTTgXU2sozLAzt0UURquJ7aVtmL1plSMrjljEalHRl0pcreTaiG6cHIA ZDEUfQri 5H1gd9lLTh140QYIagnPWlehLOI6azqRObAWFokSDaYxcczFYL02dKF+jZeZTsvJPPWU9TScnEMywlAQSWgHjE/3f3mywP4i5fOHgOsxMJLdxnH+kwKh6AHDfm97o2clJwACW446B2AX68z/ZcQsFbWX+ow7uobay97naG2mGdtRlDYe9EQzc/rqoQIRpt0imbY/5gjjOldg4HkN/lCCcK7skoXzuHwv9K8thGLUcE++rMFOIBeOLnH4ngV+UReuFpiVf+1rRXPfpRfV5L3XVMXZgsnFCiIK053BpET7h7JgiWQJ0nWdDhP41vhW1uJSKiXPy8vlQ4TYB3G/harXRYa+UxokNV0jIYmpeldE+pnIat/I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert AF_ALG to use iov_iter_extract_pages() instead of iov_iter_get_pages(). This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO-read rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O would otherwise end up only visible to the child process and not the parent). Signed-off-by: David Howells cc: Herbert Xu cc: linux-crypto@vger.kernel.org --- crypto/af_alg.c | 10 +++++++--- include/crypto/if_alg.h | 1 + 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 5f7252a5b7b4..7caff10df643 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -533,14 +533,17 @@ static const struct net_proto_family alg_family = { int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len) { + struct page **pages = sgl->pages; size_t off; ssize_t n; int npages, i; - n = iov_iter_get_pages2(iter, sgl->pages, len, ALG_MAX_PAGES, &off); + n = iov_iter_extract_pages(iter, &pages, len, ALG_MAX_PAGES, 0, &off); if (n < 0) return n; + sgl->need_unpin = iov_iter_extract_will_pin(iter); + npages = DIV_ROUND_UP(off + n, PAGE_SIZE); if (WARN_ON(npages == 0)) return -EINVAL; @@ -573,8 +576,9 @@ void af_alg_free_sg(struct af_alg_sgl *sgl) { int i; - for (i = 0; i < sgl->npages; i++) - put_page(sgl->pages[i]); + if (sgl->need_unpin) + for (i = 0; i < sgl->npages; i++) + unpin_user_page(sgl->pages[i]); } EXPORT_SYMBOL_GPL(af_alg_free_sg); diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h index 7e76623f9ec3..46494b33f5bc 100644 --- a/include/crypto/if_alg.h +++ b/include/crypto/if_alg.h @@ -59,6 +59,7 @@ struct af_alg_sgl { struct scatterlist sg[ALG_MAX_PAGES + 1]; struct page *pages[ALG_MAX_PAGES]; unsigned int npages; + bool need_unpin; }; /* TX SGL entry */ From patchwork Wed Mar 29 14:13:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C3EAC74A5B for ; Wed, 29 Mar 2023 14:15:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E23DB6B0099; Wed, 29 Mar 2023 10:15:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD3936B009A; Wed, 29 Mar 2023 10:15:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9BF4900002; Wed, 29 Mar 2023 10:15:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B32006B0099 for ; Wed, 29 Mar 2023 10:15:07 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5263BC0B59 for ; Wed, 29 Mar 2023 14:15:07 +0000 (UTC) X-FDA: 80622132654.26.4EFE851 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 9846820020 for ; Wed, 29 Mar 2023 14:15:05 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q86dNfQ4; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099305; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j9S4Bu113H4v2EoDiT0rFcd+jOn0KUYsdPxs77CpfgI=; b=Rpc1naVNE5CLOAee0y4u9DEAq66aRZ4vBtplhtXFNqnyFJ6A67MgscHgIDMFJ2EQT6TYgt mBXQrhOxItOcPTqYPGvCNmHhZYUkce7ap4BJO1RRpUh2EBDydj0Gpv93XbarGQUOwWbdRK P9jwR0zLv/LV6ArRJjjwRYfNPOt5MZQ= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q86dNfQ4; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099305; a=rsa-sha256; cv=none; b=yQKwAEz4/P8dATLUwL95JLNUbIuFwIFRwOqxTxnoosgIqfT+etZIjeV7dCPoCDjLIW6m5j 5vhtvAEOKEL/xpupOdAHsmJKyz+en4mGzlFir7qC8mrYzul2jvyfM6BFiJO3umlPW7p3Mk gwsxZeTS2hwWIBJakMqzt1Tj401rr0Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j9S4Bu113H4v2EoDiT0rFcd+jOn0KUYsdPxs77CpfgI=; b=Q86dNfQ42JY74ZD/92xSieZdTtrxJ2owrDBd3+i1p/QNge9ll/l1lWiER11CTCaxA1hQJ5 bFTTXJQLq6uwmUl+4hS+FFokg7wVPH2xM0r/iT1DYTpv+1qZuTJZYsiehj8pQR3JZtB1Gt YRFC1mJijJbg1XAm56jhnJB21juwRgw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-3-Iw9_nRRqMKG9XRVw14kxqw-1; Wed, 29 Mar 2023 10:15:02 -0400 X-MC-Unique: Iw9_nRRqMKG9XRVw14kxqw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6BDD18028B2; Wed, 29 Mar 2023 14:14:59 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5B9152166B34; Wed, 29 Mar 2023 14:14:57 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 22/48] crypto: af_alg: Use netfs_extract_iter_to_sg() to create scatterlists Date: Wed, 29 Mar 2023 15:13:28 +0100 Message-Id: <20230329141354.516864-23-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9846820020 X-Rspam-User: X-Stat-Signature: qq8qmxz447db38hg9grckxkscc4tn9ey X-HE-Tag: 1680099305-891529 X-HE-Meta: U2FsdGVkX18rVcFw25MdqlyK7eNIAbkYOhgACG+heH7Uacisp+8uOsSxVGFqswCwjpNwr9YFzERPejRq0mfoYf2f+2ax9RE5/9oeTZN5x5TeqS1XnVNRmn3l7/QmoKtoWRa83uAy9yN8dC9t9p7o/gQ16WbeWOrKRRvScqTbXzbk5X16snoSOk7wJ2TX2yhYLZEp0R3VHpX00+v+k81inMAw0J5QV66e8IuhSrWF+je4Xy49fpaJ3962f8sIyRP/IMvpqWhQ+TCMI15hIGSAbcrPIh/DCcJ4/zTTQrLLzJSxZqF7EZ1bC73R4jn40+fDPnyurFbCMQhnxcDH0m5EqTZ+HTYNp9foqsMW9eAtMCWYIKbnNAHThOG/lzuQmAfqmaOpx5O4TR842ZIkwKtKpaMC84/vr6d/Tim2kjUdtBGHSNr2+wB52KO+AT99riw7LdI3nu3CsuFCfJc+hV5pMDOWJ3g955G/pKqCawurNNOBGhJfYsUl+0IZAN/eCiExdsMzsqSSUzHdRaeafT6CrGG/VfPJHQlcggFWlTIKoqNK1MS9F302SiZGBdu2Qo57JC4lOU2lSujYw4mk7OzdwlGcN9xyRcigfAPlICt9ozI04cp8JcUFnfLJyn/eExvRKKtMQno3NYwlCmi/3efNPivjK0ibjgyslsP4/9e5Fw+grCDdcatoRx2WIhlLKcxrxOfpI/WGgfRkiBJCH3FnOBXFal+XLn7tzbwFylvkb+J9YbQEHrYk2Ve82WyUdYWXIvLWN+vRzDvQgFJ3oRFHeX8xVnClAfKVuST0m9yfpHTucFMVGyWO/c05YgULKF2t4y6xtIq1tngToW1fatLR6HOL0atUoukeNSsQ09Za+mz6fR+zI8voU7LlBHaN45YGLNLuXKgJbMNaJT8MGf+n+096yfVr0U1BQBNIl4/xgdG4r0rOmDJruEYppBqy2XuHbZqeEl4yxQLyf5PpbCI Mj645OhQ Mh5J0Fp8FxlRMUTdX6I2zk4e9hIv3lC+Dp2eu+G8pUCtgzpIXL9Lv61EeFXF7gzz1SSwP9s6k8HjT6rtEHOOmoKn+yrA2mqOd+MG+TLjN11ucnuBXBk1I58N1qGtMkH9aDraHgP9QJ33AlTcXgcZr6QXPkho3sQGPFWlJtbnkZftkUqUPAqilfwNikMNBjfw2JkjvVA49MiKmJUMl9NneDvb/peCTJqs7S5EjKbvff/AZAbgc4/bTjZ7kNnCK0gX/IPcI3lTsgclYjg5cb3taw+/LhRzaPrzei/5PZDqIdjpIYyG/yhdRcmy5+d+ubm4pU8uV/2U9aMFbaPcl2UlYTbxaX42FHB40nNtQuvKGR0KZMSo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use netfs_extract_iter_to_sg() to decant the destination iterator into a scatterlist in af_alg_get_rsgl(). af_alg_make_sg() can then be removed. [!] Note that if this fits, netfs_extract_iter_to_sg() should move to core code. Signed-off-by: David Howells cc: Herbert Xu cc: linux-crypto@vger.kernel.org --- crypto/af_alg.c | 55 ++++++++++------------------------------- crypto/algif_aead.c | 12 ++++----- crypto/algif_hash.c | 18 ++++++++++---- crypto/algif_skcipher.c | 2 +- include/crypto/if_alg.h | 6 ++--- 5 files changed, 35 insertions(+), 58 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 7caff10df643..1dafd088ad45 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -531,45 +532,11 @@ static const struct net_proto_family alg_family = { .owner = THIS_MODULE, }; -int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len) -{ - struct page **pages = sgl->pages; - size_t off; - ssize_t n; - int npages, i; - - n = iov_iter_extract_pages(iter, &pages, len, ALG_MAX_PAGES, 0, &off); - if (n < 0) - return n; - - sgl->need_unpin = iov_iter_extract_will_pin(iter); - - npages = DIV_ROUND_UP(off + n, PAGE_SIZE); - if (WARN_ON(npages == 0)) - return -EINVAL; - /* Add one extra for linking */ - sg_init_table(sgl->sg, npages + 1); - - for (i = 0, len = n; i < npages; i++) { - int plen = min_t(int, len, PAGE_SIZE - off); - - sg_set_page(sgl->sg + i, sgl->pages[i], plen, off); - - off = 0; - len -= plen; - } - sg_mark_end(sgl->sg + npages - 1); - sgl->npages = npages; - - return n; -} -EXPORT_SYMBOL_GPL(af_alg_make_sg); - static void af_alg_link_sg(struct af_alg_sgl *sgl_prev, struct af_alg_sgl *sgl_new) { - sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1); - sg_chain(sgl_prev->sg, sgl_prev->npages + 1, sgl_new->sg); + sg_unmark_end(sgl_prev->sgt.sgl + sgl_prev->sgt.nents - 1); + sg_chain(sgl_prev->sgt.sgl, sgl_prev->sgt.nents + 1, sgl_new->sgt.sgl); } void af_alg_free_sg(struct af_alg_sgl *sgl) @@ -577,8 +544,8 @@ void af_alg_free_sg(struct af_alg_sgl *sgl) int i; if (sgl->need_unpin) - for (i = 0; i < sgl->npages; i++) - unpin_user_page(sgl->pages[i]); + for (i = 0; i < sgl->sgt.nents; i++) + unpin_user_page(sg_page(&sgl->sgt.sgl[i])); } EXPORT_SYMBOL_GPL(af_alg_free_sg); @@ -1292,8 +1259,8 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, while (maxsize > len && msg_data_left(msg)) { struct af_alg_rsgl *rsgl; + ssize_t err; size_t seglen; - int err; /* limit the amount of readable buffers */ if (!af_alg_readable(sk)) @@ -1310,16 +1277,20 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, return -ENOMEM; } - rsgl->sgl.npages = 0; + rsgl->sgl.sgt.sgl = rsgl->sgl.sgl; + rsgl->sgl.sgt.nents = 0; + rsgl->sgl.sgt.orig_nents = 0; list_add_tail(&rsgl->list, &areq->rsgl_list); - /* make one iovec available as scatterlist */ - err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen); + err = netfs_extract_iter_to_sg(&msg->msg_iter, seglen, + &rsgl->sgl.sgt, ALG_MAX_PAGES, 0); if (err < 0) { rsgl->sg_num_bytes = 0; return err; } + rsgl->sgl.need_unpin = iov_iter_extract_will_pin(&msg->msg_iter); + /* chain the new scatterlist with previous one */ if (areq->last_rsgl) af_alg_link_sg(&areq->last_rsgl->sgl, &rsgl->sgl); diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c index 42493b4d8ce4..f6aa3856d8d5 100644 --- a/crypto/algif_aead.c +++ b/crypto/algif_aead.c @@ -210,7 +210,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, */ /* Use the RX SGL as source (and destination) for crypto op. */ - rsgl_src = areq->first_rsgl.sgl.sg; + rsgl_src = areq->first_rsgl.sgl.sgt.sgl; if (ctx->enc) { /* @@ -224,7 +224,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, * RX SGL: AAD || PT || Tag */ err = crypto_aead_copy_sgl(null_tfm, tsgl_src, - areq->first_rsgl.sgl.sg, processed); + areq->first_rsgl.sgl.sgt.sgl, processed); if (err) goto free; af_alg_pull_tsgl(sk, processed, NULL, 0); @@ -242,7 +242,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, /* Copy AAD || CT to RX SGL buffer for in-place operation. */ err = crypto_aead_copy_sgl(null_tfm, tsgl_src, - areq->first_rsgl.sgl.sg, outlen); + areq->first_rsgl.sgl.sgt.sgl, outlen); if (err) goto free; @@ -268,8 +268,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, /* RX SGL present */ struct af_alg_sgl *sgl_prev = &areq->last_rsgl->sgl; - sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1); - sg_chain(sgl_prev->sg, sgl_prev->npages + 1, + sg_unmark_end(sgl_prev->sgt.sgl + sgl_prev->sgt.nents - 1); + sg_chain(sgl_prev->sgt.sgl, sgl_prev->sgt.nents + 1, areq->tsgl); } else /* no RX SGL present (e.g. authentication only) */ @@ -278,7 +278,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, /* Initialize the crypto operation */ aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src, - areq->first_rsgl.sgl.sg, used, ctx->iv); + areq->first_rsgl.sgl.sgt.sgl, used, ctx->iv); aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen); aead_request_set_tfm(&areq->cra_u.aead_req, tfm); diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index 1d017ec5c63c..f051fa624bd7 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -14,6 +14,7 @@ #include #include #include +#include #include struct hash_ctx { @@ -91,13 +92,20 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, if (len > limit) len = limit; - len = af_alg_make_sg(&ctx->sgl, &msg->msg_iter, len); + ctx->sgl.sgt.sgl = ctx->sgl.sgl; + ctx->sgl.sgt.nents = 0; + ctx->sgl.sgt.orig_nents = 0; + + len = netfs_extract_iter_to_sg(&msg->msg_iter, len, + &ctx->sgl.sgt, ALG_MAX_PAGES, 0); if (len < 0) { err = copied ? 0 : len; goto unlock; } - ahash_request_set_crypt(&ctx->req, ctx->sgl.sg, NULL, len); + ctx->sgl.need_unpin = iov_iter_extract_will_pin(&msg->msg_iter); + + ahash_request_set_crypt(&ctx->req, ctx->sgl.sgt.sgl, NULL, len); err = crypto_wait_req(crypto_ahash_update(&ctx->req), &ctx->wait); @@ -141,8 +149,8 @@ static ssize_t hash_sendpage(struct socket *sock, struct page *page, flags |= MSG_MORE; lock_sock(sk); - sg_init_table(ctx->sgl.sg, 1); - sg_set_page(ctx->sgl.sg, page, size, offset); + sg_init_table(ctx->sgl.sgl, 1); + sg_set_page(ctx->sgl.sgl, page, size, offset); if (!(flags & MSG_MORE)) { err = hash_alloc_result(sk, ctx); @@ -151,7 +159,7 @@ static ssize_t hash_sendpage(struct socket *sock, struct page *page, } else if (!ctx->more) hash_free_result(sk, ctx); - ahash_request_set_crypt(&ctx->req, ctx->sgl.sg, ctx->result, size); + ahash_request_set_crypt(&ctx->req, ctx->sgl.sgl, ctx->result, size); if (!(flags & MSG_MORE)) { if (ctx->more) diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index ee8890ee8f33..a251cd6bd5b9 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -105,7 +105,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, /* Initialize the crypto operation */ skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm); skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sgt.sgl, len, ctx->iv); if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) { /* AIO operation */ diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h index 46494b33f5bc..34224e77f5a2 100644 --- a/include/crypto/if_alg.h +++ b/include/crypto/if_alg.h @@ -56,9 +56,8 @@ struct af_alg_type { }; struct af_alg_sgl { - struct scatterlist sg[ALG_MAX_PAGES + 1]; - struct page *pages[ALG_MAX_PAGES]; - unsigned int npages; + struct sg_table sgt; + struct scatterlist sgl[ALG_MAX_PAGES + 1]; bool need_unpin; }; @@ -164,7 +163,6 @@ int af_alg_release(struct socket *sock); void af_alg_release_parent(struct sock *sk); int af_alg_accept(struct sock *sk, struct socket *newsock, bool kern); -int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len); void af_alg_free_sg(struct af_alg_sgl *sgl); static inline struct alg_sock *alg_sk(struct sock *sk) From patchwork Wed Mar 29 14:13:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AECFC6FD18 for ; Wed, 29 Mar 2023 14:15:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC51D6B009B; Wed, 29 Mar 2023 10:15:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C73B76B009C; Wed, 29 Mar 2023 10:15:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEDC56B009D; Wed, 29 Mar 2023 10:15:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9F1116B009B for ; Wed, 29 Mar 2023 10:15:10 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 68D38160D38 for ; Wed, 29 Mar 2023 14:15:10 +0000 (UTC) X-FDA: 80622132780.09.8EEB2D6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 96CEF100025 for ; Wed, 29 Mar 2023 14:15:08 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IiJWyflK; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099308; a=rsa-sha256; cv=none; b=Sftad5ppjXYG1GJLzGbV05mlMti4IDSyBPedJF2tG8Qqldt/VKWlmoD1sYl1ggkIk8lY9O +7JJGV+B6JIlOGV6MJ3+xUjLd6z7UiqtCijG6645rU1Q+xnydICIiLv155C2z4i9KycbYM y+yDyE13BQ7v86ncVvz54zfrWb7tgmQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IiJWyflK; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099308; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cnsl2PBKcn/WeJcDUlvQg+kDRV/xI85q/f39rakA9vk=; b=lRinxnfR2CrkanCdZBdlyBAMTbljzDPokd5rDAxWqs5S+VZE2Dk+X67ujXrfByBpFRHVP6 tuYliQujz+08AG14xJHVSi+oigYi5W7l8LwLpjwY9qxzhqcEFD2jVaSUDXV1bPaxgB94Ba YvNnnF+1I6RAnHcMlyxCXW6q3YuYfK0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099307; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cnsl2PBKcn/WeJcDUlvQg+kDRV/xI85q/f39rakA9vk=; b=IiJWyflK47eMCUoMXyaGoXeIODB06BJ3L+Rx7P0pJWqXyNOfcXthtDaFtUeb3XosIanZgz BKGfNaZ1RNcG0iml8UOsUzhBc/Ta9GeUl1W7SyaCZka/V06twyFg2hFijsdoIAOMGdYvSl aan++Dh7yMAT/9sokCDnNJEyXR9g0Do= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-354-ZXES39PtP7W3LCkby0Z-qw-1; Wed, 29 Mar 2023 10:15:03 -0400 X-MC-Unique: ZXES39PtP7W3LCkby0Z-qw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 19257858F09; Wed, 29 Mar 2023 14:15:02 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 062E318EC2; Wed, 29 Mar 2023 14:14:59 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 23/48] crypto: af_alg: Indent the loop in af_alg_sendmsg() Date: Wed, 29 Mar 2023 15:13:29 +0100 Message-Id: <20230329141354.516864-24-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Queue-Id: 96CEF100025 X-Rspamd-Server: rspam01 X-Stat-Signature: 775b5kwdj959sbbt3y7ykipq1i3gjwr4 X-HE-Tag: 1680099308-617724 X-HE-Meta: U2FsdGVkX18++2Gt1bfTnw3Ayh4Y7cWw/jtYCNYicfo0b/Mu0o9yGH9dKXrjNbOzfpCv9wh+AznmFmWCHycjMkVcNhPXTpZDlXFrHl8RyU5XxKZzmBmnnPJt9xJ7w1lBHkcgy2ELGwakUzVCsVs1Pk+5tDeRTWoZzTafLiELMY9iUkUcq1EZ84rsuQ0NV2OZdE0evp3W9xMeo+szMSnjXSIH4Vwi4wIXGRzdzE7I8PRKVu+RFB3dhzNCB1bXTsxodIeKnpblsqGMBWXmIzQFlip8AQj8SQjudJVsRHbG27wRDXn85Vx6B2EIQMjzB0hrIgzxrn0UiHPUSS8IP7ZYZqj8V7RHT3SgjyLEiGEAjkm/T8Fhnp5hNtOxvvdVjgT6ilQQvaCBwW1PaOtbNKt3mHfN5+igMhVWI8Fxv89gZsgUMnGZORoBDOHcKCYhyPKkz3BZgZeFFdgWs8WITVBtUBqjibdNnNbWaOW781q4AdKVmDU+u8AFbAL+U5rtDYs2siS6MYQa9qWew31D1HXt/eCT9RTyEasQ5X4gpER+IkPqDLdAW+RqKyES1jOBG/ealYXtxt0HZwQEqzrOLhhLX4hika08x/t9qRyIpCT/pr6gNBQHBozSi8jCDK6HGt/U3CYsBNwSw79GrE9pcTkYfp+GOaaOs+KRl3RJd+S92zltNwhmId7t436tPnTYLNM9RNXE6mHRnQpI+ido3DbsHnpJ4TV0bq90/jZqz14IEB+R/jufR5hb2xQe1KDrVVHIB9d9nCziBbi9vNomD6Ao3AgR9Q7ER1OHO7R0PALK2RmMo9j2BgW2oOmMBbECxXDMWMhkwycH9gdCElKv7s/LyiBsU7YOJZiFU0S/sj/wA6dXBQuf18kaAwmr1sZptW0JYooCHZCt1XrP2NHSewNZ9maniXvOrNWduNWRoFxMuduwsZ/6FGOgohko9ra/gz26nz5gGEjiNhXDdkG+3nz mMqNMBKE OrWcif02g+O+Vwr35Bmh7LWGyZrlR7f5OqAN1Dl/PCGnmB7DLQJ9eWDj3dJA+OuDx9+bFiQJIIznyTIgogQnRNgP0HM9RFNlB1NI9H0XSMD9iIWUkC6oETSz2OgnNrkM2Ks/r0+cC463QVSOM6QABRQIGCKJ9BvD8eW415ycLAb0DxjB/f1VXKWMIHHQgnaIndPU7qRi5Xz/nqe8ClIvBuW9YUF9/3Xd1jgmVOn15YCnaMT/ry8pvjHSlcQmM+c2bvuGTJnRZ7uaIKDf5rMOkYYa1Tl8D2LVqF4wWi253WTCYfAxKH+j9S6FbE0Xpk+X77AgubJFgVyx/6rE8gQTj42zKy5zuKfOcCQ/0wzLq/9aU2h6/eSF97WzU1I2M2a1fh4CANztNE654mN9JwsOc6mooUQ9s2eWRK57q5sKZfHpdIyCb5EeNV9y2A9bwZQUYfRwD88OqH4eBaIlgewiMPLpyFrLdhKKJQ9p6DbODUpNNccDegCKEjiXUaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put the loop in af_alg_sendmsg() into an if-statement to indent it to make the next patch easier to review as that will add another branch to handle MSG_SPLICE_PAGES to the if-statement. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 50 +++++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 1dafd088ad45..483821e310e9 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1031,35 +1031,37 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, if (sgl->cur) sg_unmark_end(sg + sgl->cur - 1); - do { - struct page *pg; - unsigned int i = sgl->cur; + if (1 /* TODO check MSG_SPLICE_PAGES */) { + do { + struct page *pg; + unsigned int i = sgl->cur; - plen = min_t(size_t, len, PAGE_SIZE); + plen = min_t(size_t, len, PAGE_SIZE); - pg = alloc_page(GFP_KERNEL); - if (!pg) { - err = -ENOMEM; - goto unlock; - } + pg = alloc_page(GFP_KERNEL); + if (!pg) { + err = -ENOMEM; + goto unlock; + } - sg_assign_page(sg + i, pg); + sg_assign_page(sg + i, pg); - err = memcpy_from_msg(page_address(sg_page(sg + i)), - msg, plen); - if (err) { - __free_page(sg_page(sg + i)); - sg_assign_page(sg + i, NULL); - goto unlock; - } + err = memcpy_from_msg(page_address(sg_page(sg + i)), + msg, plen); + if (err) { + __free_page(sg_page(sg + i)); + sg_assign_page(sg + i, NULL); + goto unlock; + } - sg[i].length = plen; - len -= plen; - ctx->used += plen; - copied += plen; - size -= plen; - sgl->cur++; - } while (len && sgl->cur < MAX_SGL_ENTS); + sg[i].length = plen; + len -= plen; + ctx->used += plen; + copied += plen; + size -= plen; + sgl->cur++; + } while (len && sgl->cur < MAX_SGL_ENTS); + } if (!size) sg_mark_end(sg + sgl->cur - 1); From patchwork Wed Mar 29 14:13:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF7C5C74A5B for ; Wed, 29 Mar 2023 14:15:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 523116B009D; Wed, 29 Mar 2023 10:15:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D33C6B009E; Wed, 29 Mar 2023 10:15:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 326AF900002; Wed, 29 Mar 2023 10:15:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 227A46B009D for ; Wed, 29 Mar 2023 10:15:13 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EB972A0641 for ; Wed, 29 Mar 2023 14:15:12 +0000 (UTC) X-FDA: 80622132864.27.CD3EE7C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id E0BED4001E for ; Wed, 29 Mar 2023 14:15:10 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XauJj7TJ; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=85afYybs4YSPWlJ5qYhdk1qdo4c8pZli94KqFHLqWDY=; b=vH1QB7Pgsklj88ImSgyPEDlUyh4wJKBIgvrtoF0LDzftupYttERgY0W/tu7GiS5xEhEe5E VsLjrkrI68SELZ10BogCUPqbsy5gNzl4xAPcDgn/7DmiZwnIajtBf8m9Lc85tuvPrmO2EH aqici8OEVjY5kpw2RhAO1ErOyM//hjE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XauJj7TJ; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099311; a=rsa-sha256; cv=none; b=aK4zI2z9s6YRk6MfVoEiyeDKXwqQ+KtT8+m3ZGMQR+jDcfK5zYD1W3maIDf6WXCTkRsDjQ Ymn/FRE8yArZex9YiDC6DtC2g5SQQHff58YFCY0as9uEEexds6535YiRg02BTA9YiZXzFG JPpIK+cHzux8+uNJYepDGKG44A/G6PI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=85afYybs4YSPWlJ5qYhdk1qdo4c8pZli94KqFHLqWDY=; b=XauJj7TJ9A0GX/n9ia52gA7pwy/1G2hxP/XS0GEty7NKfYgE70lCpSsu5gFGAlh6EKisA/ roQx8yujuR8+j3fsD3psjZgHg1xjd5CzTNzIIQ2B+Dc2TvXXy2QadZPAKpysIGSVnwa7He /5FYQ/vUj9MWStNCwQ5RmfmVXXllC9w= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-GHZurxPfN1C-QlBwmC150g-1; Wed, 29 Mar 2023 10:15:05 -0400 X-MC-Unique: GHZurxPfN1C-QlBwmC150g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EAA273C0ED65; Wed, 29 Mar 2023 14:15:04 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CEF4F202701F; Wed, 29 Mar 2023 14:15:02 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 24/48] crypto: af_alg: Support MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:30 +0100 Message-Id: <20230329141354.516864-25-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E0BED4001E X-Stat-Signature: xd78rhmbfhccot4zbswczg98oq55oai4 X-Rspam-User: X-HE-Tag: 1680099310-992886 X-HE-Meta: U2FsdGVkX18SLyZqDlh4cuvmnXYKAtfNZEnNV5bAhxdCyOi9dnR/09tHib51AItu8LrfRp+NKzRG08IPjVw/PK1koHUiKMGMfPOToFS2gJqyMXL/ieHq80eJopDyS//e61XFOkmkSB0usNeX2V98jUVdDLtMFekPWVIvgdoUrNab2n3RAziwZNymKi7hGbHbOc6SZQD3Mq9M6E8qgXQz8d9xzj/l5/VutRsjjnEvak903rYZ8IF4xubhcDqSRXzzDih4m2yVED+ygTiXskmnhpGsgIf5vVfkuc7yi6U8/2yaWf+vqggVTs84HSUOua/Z80u0H0FpK893DLUOWHQXjsuZGYszwwJylVipvXOg8UW6+zV7YbMbpfnqWFyC59hr5cD5enh3Flp1BQtkNb16aoyw130S+CHHRwhXmmHw6DJ6j6t6Ib2ZPNkbxJoF+Ure5OzTZt6odfy28TXZTj+5g1JSBfFpUeKx2zdzRo6DXBolL8iRdUlfWaeQvCxiNhHRUCD9Oq6T8fm9StkdDonFQvDY79xWWknD7v1y5Odvymz+Fi8nWUgJr/OZQfgwf1VQf8iMO53eIkrSupi+nbuB8ZJRyVvDNYV8mtyMyLdEKOTUwMBFDiKvI8HsSOzNWHrcDNPtRImDXn6TUjQsc9DxoU7SPH6odocD7HRF01bdrgUXXGuvUI6PkoWCsyW+JfLtWlxBI3FebFV+fQOL+e3KUsf1cNzSFM10NVIcj9iQRun/YQ9C/o7ADm1xO9EtDtHBTY3Mip1OrSClqza8tG1jNXR1IXlN2cKAS7TdtJMZ5+lm1wicVAiZ2P9zxb1Ypijdhh6pNqkA+BHQYx3+RFd/vpipNrEH0jyRL55k16h7KNYrcY1su1m2ixjN08hJi0j0mLoAhuXv5WyjgSuU12VF2tqYk5f4Lrk0glXWsSKjLvJ3KrBOpKcJNopekFdEyBQIBYH1I75d+PaK7a3DvUI 0LWApBqY 2kBT7zrtFYlIRuryaKHRZrv6cqnQr16Si1NqlOZyPeRPgIJdJyu9hdDz9e/2ZTHeqr2u/wiehe+zyuwsPTaqVtrNc0YEFuuQZQH423smsaB1S74VspevaUQ9DCIkS8RDfrhLsKpUv1pq7YKSHWLHkiljyjBnSUBiDYd9Iaes6xdlB+cg6lT43Fk67usPSPOuYugtc7fz9q70yuFKNrspMMYxNX/JCgVKMtxFpgoxJ1MfqSqEJ9NnMwSm/SQHDrbqEvV0UeS2n0FWS+LrYES5vSzie3oGDrBYs0MYPLwghd/OpshfW8MKXdmpKeZm5yZLsxi+9PQgoJhd6rCQJ0NpecEDh+KdGlTNHTCImX3WwXIJHU6guy/AvKR9Utl0fd6ej3BpL9ObOEwchtmMMmf/zH/SBRzyeblM0nW3tAkGmeEv4VY06PZq5weFvq5UxDiHfO44opGeh8prCt9ZMwYq/U3Fgdpx2/b1YQ9lUBPsUfhIfa1BnkInaskye7w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_ALG sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/Kconfig | 1 + crypto/af_alg.c | 28 ++++++++++++++++++++++++++-- crypto/algif_aead.c | 22 +++++++++++----------- crypto/algif_skcipher.c | 8 ++++---- 4 files changed, 42 insertions(+), 17 deletions(-) diff --git a/crypto/Kconfig b/crypto/Kconfig index 9c86f7045157..8c04ecbb4395 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1297,6 +1297,7 @@ menu "Userspace interface" config CRYPTO_USER_API tristate + select NETFS_SUPPORT # for netfs_extract_iter_to_sg() config CRYPTO_USER_API_HASH tristate "Hash algorithms" diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 483821e310e9..3088ab298632 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -941,6 +941,10 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, bool init = false; int err = 0; + if ((msg->msg_flags & MSG_SPLICE_PAGES) && + !iov_iter_is_bvec(&msg->msg_iter)) + return -EINVAL; + if (msg->msg_controllen) { err = af_alg_cmsg_send(msg, &con); if (err) @@ -986,7 +990,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, while (size) { struct scatterlist *sg; size_t len = size; - size_t plen; + ssize_t plen; /* use the existing memory in an allocated page */ if (ctx->merge) { @@ -1031,7 +1035,27 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, if (sgl->cur) sg_unmark_end(sg + sgl->cur - 1); - if (1 /* TODO check MSG_SPLICE_PAGES */) { + if (msg->msg_flags & MSG_SPLICE_PAGES) { + struct sg_table sgtable = { + .sgl = sg, + .nents = sgl->cur, + .orig_nents = sgl->cur, + }; + + plen = netfs_extract_iter_to_sg(&msg->msg_iter, len, + &sgtable, MAX_SGL_ENTS, 0); + if (plen < 0) { + err = plen; + goto unlock; + } + + for (; sgl->cur < sgtable.nents; sgl->cur++) + get_page(sg_page(&sg[sgl->cur])); + len -= plen; + ctx->used += plen; + copied += plen; + size -= plen; + } else { do { struct page *pg; unsigned int i = sgl->cur; diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c index f6aa3856d8d5..b16111a3025a 100644 --- a/crypto/algif_aead.c +++ b/crypto/algif_aead.c @@ -9,8 +9,8 @@ * The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is - * filled by user space with the data submitted via sendpage/sendmsg. Filling - * up the TX SGL does not cause a crypto operation -- the data will only be + * filled by user space with the data submitted via sendpage. Filling up + * the TX SGL does not cause a crypto operation -- the data will only be * tracked by the kernel. Upon receipt of one recvmsg call, the caller must * provide a buffer which is tracked with the RX SGL. * @@ -113,19 +113,19 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, } /* - * Data length provided by caller via sendmsg/sendpage that has not - * yet been processed. + * Data length provided by caller via sendmsg that has not yet been + * processed. */ used = ctx->used; /* - * Make sure sufficient data is present -- note, the same check is - * also present in sendmsg/sendpage. The checks in sendpage/sendmsg - * shall provide an information to the data sender that something is - * wrong, but they are irrelevant to maintain the kernel integrity. - * We need this check here too in case user space decides to not honor - * the error message in sendmsg/sendpage and still call recvmsg. This - * check here protects the kernel integrity. + * Make sure sufficient data is present -- note, the same check is also + * present in sendmsg. The checks in sendmsg shall provide an + * information to the data sender that something is wrong, but they are + * irrelevant to maintain the kernel integrity. We need this check + * here too in case user space decides to not honor the error message + * in sendmsg and still call recvmsg. This check here protects the + * kernel integrity. */ if (!aead_sufficient_data(sk)) return -EINVAL; diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index a251cd6bd5b9..b1f321b9f846 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -9,10 +9,10 @@ * The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is - * filled by user space with the data submitted via sendpage/sendmsg. Filling - * up the TX SGL does not cause a crypto operation -- the data will only be - * tracked by the kernel. Upon receipt of one recvmsg call, the caller must - * provide a buffer which is tracked with the RX SGL. + * filled by user space with the data submitted via sendmsg. Filling up the TX + * SGL does not cause a crypto operation -- the data will only be tracked by + * the kernel. Upon receipt of one recvmsg call, the caller must provide a + * buffer which is tracked with the RX SGL. * * During the processing of the recvmsg operation, the cipher request is * allocated and prepared. As part of the recvmsg operation, the processed From patchwork Wed Mar 29 14:13:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D921C6FD18 for ; Wed, 29 Mar 2023 14:15:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC0176B0080; Wed, 29 Mar 2023 10:15:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E497D6B009F; Wed, 29 Mar 2023 10:15:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4DCB6B00A0; Wed, 29 Mar 2023 10:15:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B29616B0080 for ; Wed, 29 Mar 2023 10:15:15 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9012C1C6197 for ; Wed, 29 Mar 2023 14:15:15 +0000 (UTC) X-FDA: 80622132990.18.F1291A2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 853F04001C for ; Wed, 29 Mar 2023 14:15:13 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EOIXLoHs; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mERLnHIZ7Ufhp5QAaSwmDcCxGjZ2CWUtcWDMeYUTupY=; b=dOAe97sZlRFm+McTrTfqARF7DPdzbE/Sm9J8dOJlY/L0KS4sOQywXd6TVsoMDG4W6EXKvN ciXAnIJ0/7V6u+Jwi+866ZJWtJovVhxi1ye8CAowc8FovH8Qp7/7vFq/uEDujzT3Wd4W41 QY3IHbJ3YCH0eTYTd8kQqhuIXxsHwVQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EOIXLoHs; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099313; a=rsa-sha256; cv=none; b=SPGt0sH71FR8S3HBtP7Wlr4SQ6R77nR0NT5r9P3KwySRDtj7m+56Ksfs246X++5onv8smk w1xyhT31W0RXt2sSIYyaWSnPwn6lGc18bVbGjCc9RXYBMDyk6n1U9yq98KqKm7+pPAahA9 UlFJrbvjO8za/BSynuNct1saWWb3SwM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099312; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mERLnHIZ7Ufhp5QAaSwmDcCxGjZ2CWUtcWDMeYUTupY=; b=EOIXLoHspO8cxkJf0cT9nDBiYNB1tdqEe2xLsnf7leZsX79PAQY8T4PEl9hNhU7DClmB1h i2m44o7iUP7c7qsvd2fCJQfxqqF/CZkyf+LLOWb9S0RdOAXWr0hg9b7PU6MM/L/mmQxcb9 Q9O6knMaAe+6BhFAX3QWyofz5yDvDio= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-590-z9dsdsf9NKCR0l94rUYaTw-1; Wed, 29 Mar 2023 10:15:08 -0400 X-MC-Unique: z9dsdsf9NKCR0l94rUYaTw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 98FAE885625; Wed, 29 Mar 2023 14:15:07 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 85B182027040; Wed, 29 Mar 2023 14:15:05 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 25/48] crypto: af_alg: Convert af_alg_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:31 +0100 Message-Id: <20230329141354.516864-26-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: 757fewszbrr413mjypnnsg3wdoumibek X-Rspam-User: X-Rspamd-Queue-Id: 853F04001C X-Rspamd-Server: rspam06 X-HE-Tag: 1680099313-131729 X-HE-Meta: U2FsdGVkX19K82PIy7CXc2lfbidngC8ZF779Df8bYcNaUC58LwJxFdO5w/9y0aFjDf/piE2Y9ANeKGuBfw86UJw3l7cVNLYPysV/konrq9ckQQQ8Q4RQtFFUCUxeEwdU8aLwKf7CeQeW9zjqKm8Dk54A0EGYWiZicMQ7rDeyRpkeEC3+QvzYlHs/f91A3sA4PGyOj2O3bzrO0WMEvNzXvZF82QvsGnQlF2XHs/fXrWrrMyGA3xn9NDtukHSTCvCU4FR6bV1usJ9BNR++P2TPHltxJd89H7AvznGLFJHf5sJDDnsQUj/hePRb4m3Xz/NXbAI2aazaoUM9+dBp0u5LSmv4uynGknhL1wdUNPwrxphFTk1DbalyyIEN2LJUAOoRLfU04gJzTfnaT0bdu3jRuspS2GPz18kMPy5gcp3jmV7usn/OXJXvFXPnc2hppEHn4Tnjn6rIyPTFZmB8MpoCifv2ik3VheSlgGgyZbuBoTCNdk5snM3YcDcGKIjAE+EH2CepkuH1oN9cMcwxErgbM4wYjYkh8Zgenpy7iW0yR+8hoI811ekdXTLNNN5ob6Ws1lR5iSmiYDTqiLEKpQ2XRlTEp0aPOTXvIMg1lJjiL3XgUZvxSzbsOwvB1hvsHGC8zaUZgYFXHUTEv28LRfyxRUrYubaYt3RDZj9DTuODRcGEz3X+XAl9Iv6hJnHHRdsGoG/azBPIryWJqLI+zsM9Dcoaxey++TmGo+z8rqPVRueS0XnfhMYbUpW8u0auGcrERj5ErKwB6OqSlUXNvZ3nYx7taebrDQphyGd/IkG9V0bXaou9AKwyuDCudj2A/AirGH6JQEW4BhmZoGTgcvfPmpZ4cI90wGZzMA/MqVyXYczA0j89daOAGhtb2dx/JNi4e5e6USy/yit79NV9oAvxyu7KfCKV27Pq0EZAHD5v0ouJT6Br50k3H36XxpGsQIgLdCCDnKoD0+S9JsWO2/n rcUFrqKM kO+3xBbozfmnFQyzIiSDiraXUqmPpbKUOf316dCd8LYZMMtKlHk3QJmw+hevw62lrdWJZV+As+TT8dVwkntQVX4reaVF8uVXiL2WDxzY1p67jpDhC86kxRu5Lw8HBeKg1mTVWDPdvUrVq5fvqSg8EBm8bhsDbYIujhzDXlyzuWDHJoJxtXt8hoCx/NtAJm0QxrRzWjeBTj05ukOLM3QtUp6LFzbUuPlbqvbBY1JjTHW3OghqtHcg70SasuYFzmRh49wqbPVgTVV4a+VlmZAwJ7JNKacOfQ4QiHleEfR4sug876PW7EmA23SJPEtU3EzSWjxCsoelbxUZK5BNYJDlEWWDxQt9FsuRIxOHXCjzrtbUWaPUEsDMbmlCej49jYJZIidA3v8lNlaWcnKT20ouEHry9VsgzoyQdwWMWVEYIHTKVkURqFjelfNr5x9aE3gLQ6aCM2hSrUUgC823W1d/oP2536+HHzAmYrmjmWH4LFUkYrEteBAHzXBMLLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert af_alg_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 53 +++++++++---------------------------------------- 1 file changed, 9 insertions(+), 44 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 3088ab298632..7fe8c8db6bb5 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1118,53 +1118,18 @@ EXPORT_SYMBOL_GPL(af_alg_sendmsg); ssize_t af_alg_sendpage(struct socket *sock, struct page *page, int offset, size_t size, int flags) { - struct sock *sk = sock->sk; - struct alg_sock *ask = alg_sk(sk); - struct af_alg_ctx *ctx = ask->private; - struct af_alg_tsgl *sgl; - int err = -EINVAL; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - lock_sock(sk); - if (!ctx->more && ctx->used) - goto unlock; - - if (!size) - goto done; - - if (!af_alg_writable(sk)) { - err = af_alg_wait_for_wmem(sk, flags); - if (err) - goto unlock; - } - - err = af_alg_alloc_tsgl(sk); - if (err) - goto unlock; - - ctx->merge = 0; - sgl = list_entry(ctx->tsgl_list.prev, struct af_alg_tsgl, list); - - if (sgl->cur) - sg_unmark_end(sgl->sg + sgl->cur - 1); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - sg_mark_end(sgl->sg + sgl->cur); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - get_page(page); - sg_set_page(sgl->sg + sgl->cur, page, size, offset); - sgl->cur++; - ctx->used += size; - -done: - ctx->more = flags & MSG_MORE; - -unlock: - af_alg_data_wakeup(sk); - release_sock(sk); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return err ?: size; + return sock_sendmsg(sock, &msg); } EXPORT_SYMBOL_GPL(af_alg_sendpage); From patchwork Wed Mar 29 14:13:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85A55C6FD18 for ; Wed, 29 Mar 2023 14:15:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C77F6B009F; Wed, 29 Mar 2023 10:15:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 178CA6B00A0; Wed, 29 Mar 2023 10:15:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0D1A6B00A1; Wed, 29 Mar 2023 10:15:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E0C786B009F for ; Wed, 29 Mar 2023 10:15:18 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9939140CD4 for ; Wed, 29 Mar 2023 14:15:18 +0000 (UTC) X-FDA: 80622133116.02.088F1EE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 7A765180021 for ; Wed, 29 Mar 2023 14:15:16 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Dk//I4lx"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M5DaLi1tovZrH11oTI95mtlHRv/XZcPSFvwL/aUxUh4=; b=M2oP5Z8d/ozTZw8iAkug1jFMjHxpMchS5ih/ENv5oqDwIcsYc9iowd2lSHk6yUmpnDeRR3 okZmczSyY6yCDxGE0iJNkJFvXegz1O3boWtY6jfoE2pxzm81iStYn2azDtoXajj8qzHrNp S88IuLAMpb0gBUHErfgjDvtau4GxBf4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Dk//I4lx"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099316; a=rsa-sha256; cv=none; b=QfwugTElvxd9sOo5OU++44shuNGUGWbtSBKbI6EhMkVryrt8VpJ9DuGLP024iekjTHagE1 vYEQYALpo5/fVZuZdmqNd1RiZTvjG2b8GrYij3wvLf4NXd5P8xHHggpJDjxvdLFyzaksA3 l9VQnye7FeX2L654vvvNMdNq73qyhWE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M5DaLi1tovZrH11oTI95mtlHRv/XZcPSFvwL/aUxUh4=; b=Dk//I4lx7BOnYxkfIz9av1ekycqK22HPXeNxO1cvCjF8u68qCSuXVWi021S1bxx9ueIwEZ Kv55SWUKxhqolj0IODlih/ilHJqUdr8G7gqd+wnEe8ObkCiRPl1Dh76AL+LY1pK0xaIWGa 1XcSQIcbrOceMyZqQ55U01F8cDhiYWQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-612-sHDpKiIEOqiGBf3DQmh3WA-1; Wed, 29 Mar 2023 10:15:11 -0400 X-MC-Unique: sHDpKiIEOqiGBf3DQmh3WA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6673388562B; Wed, 29 Mar 2023 14:15:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 548E32027041; Wed, 29 Mar 2023 14:15:08 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 26/48] crypto: af_alg/hash: Support MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:32 +0100 Message-Id: <20230329141354.516864-27-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7A765180021 X-Stat-Signature: n4xtxws6a35nctgmbxh53nxia69qhwn8 X-HE-Tag: 1680099316-923156 X-HE-Meta: U2FsdGVkX19Nh6vcEQUnSW9QHX0ZedsmEUuFxmJJE+q3uuA2hYN7v49aVYTQz6aNmlw6TBBoj74Qcymn4LL0DC9y290GtoW+vKwq1dFMPb2pNvAeHuNOEcDT76h4pCnOvS3MotXF7/OdKrvergGtINHJH0d1G6MMBHcOBbUphkHkx5kQLzEQMWl6j/XgzqWXwkb5Cwx54Ao58spxlqtHPbrKwRjSFR8TEJSknq0RpjVjrjL/VmacawJuPoZjP/fqFRYaJKXCQVKu54PjG5guKM9yxJk/qJmukccdM69zZnav8dqmSAazfQIfE4BJ1D1OON9d8Rtov3Jo/EmrEctdmam3cuwvDFoNYnq0PBrIkIvPYx4JxZisnpmKcg1+yRsez6zPG223HqXIGtLf/WDA8PrSNBC26AoBVMzgMbTUoSjhHVg4Ds50DSvAQzgPyCON003IvOZ7RXkLG5R38SAgDx/kK5ztd5weAWlzx6RpiDFRChYXwKmuBAi3Od0a+rG07UrkDLpohf3A1Da9lDEVyGfW7307pEVDDzi2xp+LllhxM05/XLJE4IDHPkxZ9g/6kXSrBX6Apjyit6xh8lq3ufqd7URP6bQjKpJkwpN8jiBb9MKb/vYKIe7B7/BKmFwfP7LxOj1upKZWlCrn0wn3vZ1KqezrDsp3uHRs+TMoGn03UIfnY5Zx/sQ91FO1F1ArnVjGfWHW86Ui4XRPThMXJlLaMVA+Imo+9ahR8LG7QqDQIWOZ9imaUtUeDT4k9/bW0Cn5+HHV5bv7b6DD55AgM1zBNr4x2kSRS1tAbTvOPm8qdziBsM55InG3kgyKGwp0DaJx/Pv4p4a1qxpZmGxntC8xXI+dEsYnWtu2IfoNftvOjBBVv9SyYxi4BPaAKSYczRwopc6wUvlndvH+W5yYzmX5Bv9rwS6F5x2NgyhTg/gcG8ac2SO+LzyYU7BCmz7p4CcwN6gHZUjqoXGfYCT X09Q4yyM sypFlniaLwjKVqZ82ZhJLo6ulZF3RVPWKdBkNhGSpLI6LdPWxvG6+XSN7cnHqFsIiaAZhVrPtE4E4tZZu+3tEhKwp6aGWvjPfkrh/LK/PNXpqHoEEiCHyZIWNBJi7f8jdh3tY5H1OcRMS+cAi5nmoEoaKF/M02Ci52oLB7hR6gQh3sAPFt6smfFNubFVXVsthXhbqDAqYE3um7/pC0SFl4Nr5yXuF+pEzDkaAGD4pZCbS4O0CkSGHe+1RV9xjvbBGdABMv1JOXZEHIf5atk1gS1l+SEBLzJAx6KBFPe1NlxCwaY24lTUObfQHwMQhCIfFShAZm0i6qREG67T2MauXu25eoZdLRkAvMDlPqzUDZyD1GyWLCbOHVNdlkx2Pf570aJXFmJ7j6fE1wZ3wfhNZhiIqVVF0zgXW8xGrbrQwzLCsdlFhwxXYOYln99HJd/jqf+lvUzrLra53AG5taJSKE4cIri9Haucc32gEFuoL6CeXoT77fihi+ZhZmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_ALG sendmsg() support MSG_SPLICE_PAGES in the hashing code. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 11 +++-- crypto/algif_hash.c | 99 ++++++++++++++++++++++++++++----------------- 2 files changed, 70 insertions(+), 40 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 7fe8c8db6bb5..686610a4986f 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -543,9 +543,14 @@ void af_alg_free_sg(struct af_alg_sgl *sgl) { int i; - if (sgl->need_unpin) - for (i = 0; i < sgl->sgt.nents; i++) - unpin_user_page(sg_page(&sgl->sgt.sgl[i])); + if (sgl->sgt.sgl) { + if (sgl->need_unpin) + for (i = 0; i < sgl->sgt.nents; i++) + unpin_user_page(sg_page(&sgl->sgt.sgl[i])); + if (sgl->sgt.sgl != sgl->sgl) + kvfree(sgl->sgt.sgl); + sgl->sgt.sgl = NULL; + } } EXPORT_SYMBOL_GPL(af_alg_free_sg); diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index f051fa624bd7..b89c2c50cecc 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -64,77 +64,102 @@ static void hash_free_result(struct sock *sk, struct hash_ctx *ctx) static int hash_sendmsg(struct socket *sock, struct msghdr *msg, size_t ignored) { - int limit = ALG_MAX_PAGES * PAGE_SIZE; struct sock *sk = sock->sk; struct alg_sock *ask = alg_sk(sk); struct hash_ctx *ctx = ask->private; - long copied = 0; + ssize_t copied = 0; + size_t len, max_pages = ALG_MAX_PAGES, npages; + bool continuing = ctx->more, need_init = false; int err; - if (limit > sk->sk_sndbuf) - limit = sk->sk_sndbuf; + /* Don't limit to ALG_MAX_PAGES if the pages are all already pinned. */ + if (!user_backed_iter(&msg->msg_iter)) + max_pages = INT_MAX; + else + max_pages = min_t(size_t, max_pages, + DIV_ROUND_UP(sk->sk_sndbuf, PAGE_SIZE)); lock_sock(sk); - if (!ctx->more) { + if (!continuing) { if ((msg->msg_flags & MSG_MORE)) hash_free_result(sk, ctx); - - err = crypto_wait_req(crypto_ahash_init(&ctx->req), &ctx->wait); - if (err) - goto unlock; + need_init = true; } ctx->more = false; while (msg_data_left(msg)) { - int len = msg_data_left(msg); - - if (len > limit) - len = limit; - ctx->sgl.sgt.sgl = ctx->sgl.sgl; ctx->sgl.sgt.nents = 0; ctx->sgl.sgt.orig_nents = 0; - len = netfs_extract_iter_to_sg(&msg->msg_iter, len, - &ctx->sgl.sgt, ALG_MAX_PAGES, 0); - if (len < 0) { - err = copied ? 0 : len; - goto unlock; + err = -EIO; + npages = iov_iter_npages(&msg->msg_iter, max_pages); + if (npages == 0) + goto unlock_free; + + if (npages > ARRAY_SIZE(ctx->sgl.sgl)) { + err = -ENOMEM; + ctx->sgl.sgt.sgl = + kvmalloc(array_size(npages, sizeof(*ctx->sgl.sgt.sgl)), + GFP_KERNEL); + if (!ctx->sgl.sgt.sgl) + goto unlock_free; } + sg_init_table(ctx->sgl.sgl, npages); ctx->sgl.need_unpin = iov_iter_extract_will_pin(&msg->msg_iter); - ahash_request_set_crypt(&ctx->req, ctx->sgl.sgt.sgl, NULL, len); + err = netfs_extract_iter_to_sg(&msg->msg_iter, LONG_MAX, + &ctx->sgl.sgt, npages, 0); + if (err < 0) + goto unlock_free; + len = err; + sg_mark_end(ctx->sgl.sgt.sgl + ctx->sgl.sgt.nents - 1); - err = crypto_wait_req(crypto_ahash_update(&ctx->req), - &ctx->wait); - af_alg_free_sg(&ctx->sgl); - if (err) { - iov_iter_revert(&msg->msg_iter, len); - goto unlock; + if (!msg_data_left(msg)) { + err = hash_alloc_result(sk, ctx); + if (err) + goto unlock_free; } - copied += len; - } + ahash_request_set_crypt(&ctx->req, ctx->sgl.sgt.sgl, ctx->result, len); - err = 0; + if (!msg_data_left(msg) && !continuing && !(msg->msg_flags & MSG_MORE)) { + err = crypto_ahash_digest(&ctx->req); + } else { + if (need_init) { + err = crypto_wait_req(crypto_ahash_init(&ctx->req), + &ctx->wait); + if (err) + goto unlock_free; + need_init = false; + } + + if (msg_data_left(msg) || (msg->msg_flags & MSG_MORE)) + err = crypto_ahash_update(&ctx->req); + else + err = crypto_ahash_finup(&ctx->req); + continuing = true; + } - ctx->more = msg->msg_flags & MSG_MORE; - if (!ctx->more) { - err = hash_alloc_result(sk, ctx); + err = crypto_wait_req(err, &ctx->wait); if (err) - goto unlock; + goto unlock_free; - ahash_request_set_crypt(&ctx->req, NULL, ctx->result, 0); - err = crypto_wait_req(crypto_ahash_final(&ctx->req), - &ctx->wait); + copied += len; + af_alg_free_sg(&ctx->sgl); } + ctx->more = msg->msg_flags & MSG_MORE; + err = 0; unlock: release_sock(sk); + return copied ?: err; - return err ?: copied; +unlock_free: + af_alg_free_sg(&ctx->sgl); + goto unlock; } static ssize_t hash_sendpage(struct socket *sock, struct page *page, From patchwork Wed Mar 29 14:13:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B56AC74A5B for ; Wed, 29 Mar 2023 14:15:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0933B6B00A0; Wed, 29 Mar 2023 10:15:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0401A6B00A1; Wed, 29 Mar 2023 10:15:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD3B26B00A2; Wed, 29 Mar 2023 10:15:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C1A3C6B00A0 for ; Wed, 29 Mar 2023 10:15:20 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6A4BEA0677 for ; Wed, 29 Mar 2023 14:15:20 +0000 (UTC) X-FDA: 80622133200.03.F33C606 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 70EA01C0006 for ; Wed, 29 Mar 2023 14:15:18 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FSA3Hbcl; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I/S0tdHFgU2wYUbqRS8y7qqDbwCo5raKog1zVTXoTHk=; b=HXahDIjT/dVRXC/8002oavIajmpLiQ5g8tW+OCvnQCcUL3LOWyx75dF/UMIkvIzfie056y lQGSnngQIkfEKpqcc8JhJ4p/maC5UE51hZBsk1+Y2j/tsNT6I7sq6KvAA9ajXEj9K5H1JC 6vgF9+mXZguySNv9NcRNXnjNydht8X4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FSA3Hbcl; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099318; a=rsa-sha256; cv=none; b=AT4M6rv7s9YfpEvBEM0/GBX+DI+45Gc51bzy8ZGt2FKv+QrW23Glm1nCaU01Patq8qPRJo UIN5H74ix5U92GGt01ewfsXmOraJGW2+8eozlxq375ArXjmt77TM7Z9MHIeGFyhNi+qIbe S1tkUsZffY+HQDK9BJgb994ObNTAt8k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I/S0tdHFgU2wYUbqRS8y7qqDbwCo5raKog1zVTXoTHk=; b=FSA3HbclTG/VUq/4IdV1ol7z1lsRggUH8Ffx/3rAGIpyfJ5SgzEnXNlZKIya4Ye4/d1x7r tzanLLn1mDy6UF/PdYemBcqRZ85CwaP4bw2AaK9jVa6DajEz91qejUu7/5o33IyXJIaelY gLa336/tBv2/PDdVtZl4kJzaT51WN/4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-96-tJwSygBLMUeAppTVp53tcA-1; Wed, 29 Mar 2023 10:15:14 -0400 X-MC-Unique: tJwSygBLMUeAppTVp53tcA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E7F23C0ED4B; Wed, 29 Mar 2023 14:15:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 151BA492B01; Wed, 29 Mar 2023 14:15:10 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 27/48] splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() Date: Wed, 29 Mar 2023 15:13:33 +0100 Message-Id: <20230329141354.516864-28-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 70EA01C0006 X-Stat-Signature: 5zmxxj1ererctr4of166qyyxyn95i78z X-Rspam-User: X-HE-Tag: 1680099318-251017 X-HE-Meta: U2FsdGVkX1/yvtMDG/kPICK4E8na70dw+f6C3Wyq8nGtYwPv+/IgSxaUwZcyOjGZpBYKD/oigVeeE8mL68YtuMLSq+gqgc7QTXu1pm8D8EIsrV/CxF32ZVv3M6dovFicTS8X4i+CGlwGBp+XcrPqAUW9QDdd6AyYO2g0lKPBKFb4izK00cnzISjql4JrTK818a5lOBN8aEujttd9zi0gLsyJNkuC86c/OwncOHFqfIfQsQux7A2XfXVR8XSck3Tl0xZX72BlgoArdEBj5ZT8tMxskA/ka3kB65SlRoFrvZOJ5byZjJLMPn1wJFcUXSY9df5bx1DexXEilPc8wafYmWbbLy1PRdOX0KpasQ/eJ24fmFaV44CKU5Va43jrusY58MQovNy3AediCgxAh6C3VZjr+gVZ23m+KGkYIP7NRuKhexCvzxZv4SoY1y542OA7IWQ0Oz+dvDUpm/omAvWNgLCgkmbnfHqPmYG4BnS3ZeM+MLrrwA6o5VDomdpMzjdWxVrvBc5Hj4whhP8QULiTYyxRHi64LoFy7lnEesh+qhfaAuRqO//l+zUncTwJcyzHUZgG7CHR5x7x1ge+ylf4Eu16B32GZN2AT7DUHbC1u+qp8E4yiB3br9JduMaXwEigtwxKA5zwSIj9vDBSrm1K/cmQrRwlphFg9uuDGVM3sSMWJfli1SBfSQCytOvFcD7gunNugNiXIN5zi1YNbabtOUpdsRDzaXV0BWrUI075sZTyj+CZuJaij6IXhGOl6xkD3vwEi0Xo7qvPt2RXKEwCdF2jAfl+Eb5tXI3M7xqkp0unnZIc69U6p60OpoUus+/IvNRv7GRZ4Wf3RAGeq+Guh3ca3EOQIg3eU6C1eUcPZe9RFLqlk4wkHBr90Pgj9WYoC8FL0JbD4bQEcQrFBwrRdpWg7XPetABksh8+uH8wxQ7i/j93/AXK+0hONnGjKWknZ4797Vy0Z/5YQ0m0MSN FH71OS7S IQt9ExYdwiKxL4sqCNXVc9GfMlLU7oPj0gOxbRaorzuIg82GNOGt2Jua5wPUbWt46rT2U/e4gtWoZc9SSmpps0GdVCiIzrt6HjHGwk6OYG57/WCaSJQGOj7M2RlqZbuV3ZGfDzmi0wdDfHaYGuSJ5+yT7goSbdeQ0GakKaktSiFv0ayQnRJyvrCHpsxiazu0Ap8kAqCNskbxES81UtQNBptG7KThCK50ZHCpU3v3CLbSUdKPWVwbhAaFVLWqLqm3vNcnj2i8ALLGpUHANi4IhaFNxdBeFHPZxtWe0hdfjZWEXPozIzMwPzbwCQi4aldhqSP1BEFrDQKE0WZvSGC7iMPcbMAQ4AdWAbx/mkdT8dLEu3Dc0dtcnpdTqSUzjI9h6HQtDpsAZTFZD2mXkru6z5su8/Py67KoJsEpYcajAxhmqKe283S/6t81DR3kqj1HMKFpc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() to splice data from a pipe to a socket. This paves the way for passing in multiple pages at once from a pipe and the handling of multipage folios. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- fs/splice.c | 42 +++++++++++++++++++++++------------------- include/linux/fs.h | 2 -- include/linux/splice.h | 2 ++ net/socket.c | 26 ++------------------------ 4 files changed, 27 insertions(+), 45 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index f46dd1fb367b..23ead122d631 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -410,29 +411,32 @@ const struct pipe_buf_operations nosteal_pipe_buf_ops = { }; EXPORT_SYMBOL(nosteal_pipe_buf_ops); +#ifdef CONFIG_NET /* * Send 'sd->len' bytes to socket from 'sd->file' at position 'sd->pos' * using sendpage(). Return the number of bytes sent. */ -static int pipe_to_sendpage(struct pipe_inode_info *pipe, - struct pipe_buffer *buf, struct splice_desc *sd) +static int pipe_to_sendmsg(struct pipe_inode_info *pipe, + struct pipe_buffer *buf, struct splice_desc *sd) { - struct file *file = sd->u.file; - loff_t pos = sd->pos; - int more; - - if (!likely(file->f_op->sendpage)) - return -EINVAL; + struct socket *sock = sock_from_file(sd->u.file); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES, + }; - more = (sd->flags & SPLICE_F_MORE) ? MSG_MORE : 0; + if (sd->flags & SPLICE_F_MORE) + msg.msg_flags |= MSG_MORE; if (sd->len < sd->total_len && pipe_occupancy(pipe->head, pipe->tail) > 1) - more |= MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; - return file->f_op->sendpage(file, buf->page, buf->offset, - sd->len, &pos, more); + bvec_set_page(&bvec, buf->page, sd->len, buf->offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, sd->len); + return sock_sendmsg(sock, &msg); } +#endif static void wakeup_pipe_writers(struct pipe_inode_info *pipe) { @@ -614,7 +618,7 @@ static void splice_from_pipe_end(struct pipe_inode_info *pipe, struct splice_des * Description: * This function does little more than loop over the pipe and call * @actor to do the actual moving of a single struct pipe_buffer to - * the desired destination. See pipe_to_file, pipe_to_sendpage, or + * the desired destination. See pipe_to_file, pipe_to_sendmsg, or * pipe_to_user. * */ @@ -795,8 +799,9 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, EXPORT_SYMBOL(iter_file_splice_write); +#ifdef CONFIG_NET /** - * generic_splice_sendpage - splice data from a pipe to a socket + * splice_to_socket - splice data from a pipe to a socket * @pipe: pipe to splice from * @out: socket to write to * @ppos: position in @out @@ -808,13 +813,12 @@ EXPORT_SYMBOL(iter_file_splice_write); * is involved. * */ -ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, struct file *out, - loff_t *ppos, size_t len, unsigned int flags) +ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags) { - return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendpage); + return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendmsg); } - -EXPORT_SYMBOL(generic_splice_sendpage); +#endif static int warn_unsupported(struct file *file, const char *op) { diff --git a/include/linux/fs.h b/include/linux/fs.h index c85916e9f7db..f3ccc243851e 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2740,8 +2740,6 @@ extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); -extern ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, - struct file *out, loff_t *, size_t len, unsigned int flags); extern long do_splice_direct(struct file *in, loff_t *ppos, struct file *out, loff_t *opos, size_t len, unsigned int flags); diff --git a/include/linux/splice.h b/include/linux/splice.h index 8f052c3dae95..e6153feda86c 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -87,6 +87,8 @@ extern long do_splice(struct file *in, loff_t *off_in, extern long do_tee(struct file *in, struct file *out, size_t len, unsigned int flags); +extern ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags); /* * for dynamic pipe sizing diff --git a/net/socket.c b/net/socket.c index dfb912bbed62..2cd5c2bcdde8 100644 --- a/net/socket.c +++ b/net/socket.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -126,8 +127,6 @@ static long compat_sock_ioctl(struct file *file, unsigned int cmd, unsigned long arg); #endif static int sock_fasync(int fd, struct file *filp, int on); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more); static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); @@ -162,8 +161,7 @@ static const struct file_operations socket_file_ops = { .mmap = sock_mmap, .release = sock_close, .fasync = sock_fasync, - .sendpage = sock_sendpage, - .splice_write = generic_splice_sendpage, + .splice_write = splice_to_socket, .splice_read = sock_splice_read, .show_fdinfo = sock_show_fdinfo, }; @@ -1062,26 +1060,6 @@ int kernel_recvmsg(struct socket *sock, struct msghdr *msg, } EXPORT_SYMBOL(kernel_recvmsg); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more) -{ - struct socket *sock; - int flags; - int ret; - - sock = file->private_data; - - flags = (file->f_flags & O_NONBLOCK) ? MSG_DONTWAIT : 0; - /* more is a combination of MSG_MORE and MSG_SENDPAGE_NOTLAST */ - flags |= more; - - ret = kernel_sendpage(sock, page, offset, size, flags); - - if (trace_sock_send_length_enabled()) - call_trace_sock_send_length(sock->sk, ret, 0); - return ret; -} - static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) From patchwork Wed Mar 29 14:13:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A98CC6FD18 for ; Wed, 29 Mar 2023 14:15:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B8C66B00A8; Wed, 29 Mar 2023 10:15:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0494C6B00AA; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C26E16B00A9; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 96D596B00A7 for ; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4FAC9C0BF6 for ; Wed, 29 Mar 2023 14:15:47 +0000 (UTC) X-FDA: 80622134334.09.FED6268 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 96CF914001D for ; Wed, 29 Mar 2023 14:15:45 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JAP6ggSG; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Zy7MVQlzWaHdY2cAkrhIxWsi1s53LxwKb8sGz4UC6E=; b=idE0OXIL0dIF/HMPMogNnDwAhTI9znX14PdmHZtjS22HPTPHJn4K4tMgnsZn6r8rCRIDzU qpVbJU+gXsfErzjeFFDjPOANJyG1FfaGSHlWj9SqwnsLsc+SyxTwDX7g6ZBWx8NQvZ62tY ON5+SvsGIT6xRunNKDvZrPakX80L9hc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JAP6ggSG; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099345; a=rsa-sha256; cv=none; b=zugYvb6LgW+rQ9nca6OlFogd5eI81ih3betBDnFphTHkjPQeKF/WBhH9JeIUXevPgqbxhG S2VmK+k0UKjMPA3XAFpv8XmeGLIbmbAjGxydo97fggCwiZJI9naTEoi9gsQ5Ma+TyXLKW7 0KKe094E2M4+AVO2lBiNJwf8Fb+1tYY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Zy7MVQlzWaHdY2cAkrhIxWsi1s53LxwKb8sGz4UC6E=; b=JAP6ggSGHe+a4Dua/CKDo1wpHLQhekTjrLwTXQ+fQArLRgmc/Byk5i6PJR4Nq2cGPFiQ6m Qamh/0IS9Wt1Yhk/7riKyLbICsO97y6vb4O1XtD2ev8Crvd5zv8xSXoy30JMSLTZn9G5VL ZVzNwpf8EwPR3IJfmjmixb+tmMzcZUQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-441-MbBkqL9VMYWp7A_VXkoNTw-1; Wed, 29 Mar 2023 10:15:36 -0400 X-MC-Unique: MbBkqL9VMYWp7A_VXkoNTw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9C09D299E75B; Wed, 29 Mar 2023 14:15:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B961E492C3E; Wed, 29 Mar 2023 14:15:13 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 28/48] splice: Reimplement splice_to_socket() to pass multiple bufs to sendmsg() Date: Wed, 29 Mar 2023 15:13:34 +0100 Message-Id: <20230329141354.516864-29-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Queue-Id: 96CF914001D X-Stat-Signature: hs1aqwqedsgw6j7rz55oa4znx91jaf3q X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680099345-574998 X-HE-Meta: U2FsdGVkX18NPIohYZZVqRXloJzIpzbrZayZcLkPid+UG54rT9qiN6adXk4+R3H25HdXT7OOCnTUfF6Y22U350Nr4vMQU8aLHSEYv3iHqq8kbtNZW8ygNJhcGKY5itkMN72CPg9ihcAMXCcoE0I38Ph3NjZJhwdRc1tvCNySkcmzUmm+G6KDiCAF0EH0lpE+7A+i7aJC+bGwgfnrp3gvnmOkgOWlfJYNZH7yV8+uqLFHeflFJB4exYUL8IMbdhAcfWDTxzyEi5IpqQODDFyzfi21ecQ+ZnV1CdP9vki4L4brrUtPw7imRFlpNl07Jb32wHw66uPyOEkOY3cLBG7AJu2D+yA/zbHeuBOLVN6f/AaWOMYGrxok0acdQUtGGQ1ZehA+Dty+yZtArhA/+OmXK58PbBtD5DYAeRqr7Os80cLNx3EYEBtrA1WJk42cNZyLndOJdVzTHpeIuAEPZV9TSWqwE7lCaMwxrXbfpa3AOmVSDtMwDQKntU9a14q7byskHj55Mgk3wXFqC5pTc0wT+hGF3eQSEl1uSJiIP1A9NXzoqZJS4V66y5pST4pkIRZSpYGq7oJUvM+WSeWBx/+y335XY6xGmJqSP1Yg5/FFHgEbFVLedeKZoHcE9+JkLCdcXAolz6fQVVIhqqNbrQxnJUHcKcw22I6ZXqCg8W2ywNHJB9BeTwFRgj/VG1swcRJfs78P1pQkNeVH2Vo+J5pg8502TmgRkGuq3UK6/F++tt2hSfctnUAlhOqLMlWa5Kh5eo7J013jb0XA79/dR0di10vQJz24vnxlYpzZfZPLdDW/T7b0U87idC5QKsIBoeiAU4qFOl6fpTN/7Pa8BlJHnVJO4mOKCD5J7M95kgx21hGuQt70yzOkeYXhZEYjw+mN/eb9evBg4zOue1CMSZPt/Bg8LnkkYSTFptfs5ayhEkpjOPKPyu0NEQm39k0v55K/xuGiaH/zyGpv4HN1gco n8YPOOP8 0k+Ft03g3XwJor+inyLGSUOjPbX7zNaTgKA6Uxe/2M+lBQ8YFKtv4npV5MqJHZdNfjvbw+WaO4oKQkysgl5YTLiUu/B0g2wG7jXd3BeV0iexzA6ajG1ikxs7k7mArzgkfPFsDemjVZ19RGporw3mIzHj59XHUq9XTjnmIVwQCgb+wGqixflncSf21To5IRJXIbhAI5uOuNSCGDa0QeSnFsqs9pKg6Foz8xC3876YZhYJPWC4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reimplement splice_to_socket() so that it can pass multiple pipe buffer pages to sendmsg() in a single go. Signed-off-by: David Howells --- fs/splice.c | 148 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 120 insertions(+), 28 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 23ead122d631..d5bc28b59720 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -411,33 +411,6 @@ const struct pipe_buf_operations nosteal_pipe_buf_ops = { }; EXPORT_SYMBOL(nosteal_pipe_buf_ops); -#ifdef CONFIG_NET -/* - * Send 'sd->len' bytes to socket from 'sd->file' at position 'sd->pos' - * using sendpage(). Return the number of bytes sent. - */ -static int pipe_to_sendmsg(struct pipe_inode_info *pipe, - struct pipe_buffer *buf, struct splice_desc *sd) -{ - struct socket *sock = sock_from_file(sd->u.file); - struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = MSG_SPLICE_PAGES, - }; - - if (sd->flags & SPLICE_F_MORE) - msg.msg_flags |= MSG_MORE; - - if (sd->len < sd->total_len && - pipe_occupancy(pipe->head, pipe->tail) > 1) - msg.msg_flags |= MSG_MORE; - - bvec_set_page(&bvec, buf->page, sd->len, buf->offset); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, sd->len); - return sock_sendmsg(sock, &msg); -} -#endif - static void wakeup_pipe_writers(struct pipe_inode_info *pipe) { smp_mb(); @@ -816,7 +789,126 @@ EXPORT_SYMBOL(iter_file_splice_write); ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, loff_t *ppos, size_t len, unsigned int flags) { - return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendmsg); + struct socket *sock = sock_from_file(out); + struct bio_vec bvec[16]; + struct msghdr msg = {}; + ssize_t ret; + size_t spliced = 0; + bool need_wakeup = false; + + pipe_lock(pipe); + + while (len > 0) { + unsigned int head, tail, mask, bc = 0; + size_t remain = len; + + /* + * Check for signal early to make process killable when there + * are always buffers available + */ + ret = -ERESTARTSYS; + if (signal_pending(current)) + break; + + while (pipe_empty(pipe->head, pipe->tail)) { + ret = 0; + if (!pipe->writers) + goto out; + + if (spliced) + goto out; + + ret = -EAGAIN; + if (flags & SPLICE_F_NONBLOCK) + goto out; + + ret = -ERESTARTSYS; + if (signal_pending(current)) + goto out; + + if (need_wakeup) { + wakeup_pipe_writers(pipe); + need_wakeup = false; + } + + pipe_wait_readable(pipe); + } + + head = pipe->head; + tail = pipe->tail; + mask = pipe->ring_size - 1; + + while (!pipe_empty(head, tail)) { + struct pipe_buffer *buf = &pipe->bufs[tail & mask]; + size_t seg; + + if (!buf->len) { + tail++; + continue; + } + + seg = min_t(size_t, remain, buf->len); + seg = min_t(size_t, seg, PAGE_SIZE); + + ret = pipe_buf_confirm(pipe, buf); + if (unlikely(ret)) { + if (ret == -ENODATA) + ret = 0; + break; + } + + bvec_set_page(&bvec[bc++], buf->page, seg, buf->offset); + remain -= seg; + if (seg >= buf->len) + tail++; + if (bc >= ARRAY_SIZE(bvec)) + break; + } + + if (!bc) + break; + + msg.msg_flags = 0; + if (flags & SPLICE_F_MORE) + msg.msg_flags = MSG_MORE; + if (remain && pipe_occupancy(pipe->head, tail) > 0) + msg.msg_flags = MSG_MORE; + msg.msg_flags |= MSG_SPLICE_PAGES; + + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, bc, len - remain); + ret = sock_sendmsg(sock, &msg); + if (ret <= 0) + break; + + spliced += ret; + len -= ret; + tail = pipe->tail; + while (ret > 0) { + struct pipe_buffer *buf = &pipe->bufs[tail & mask]; + size_t seg = min_t(size_t, ret, buf->len); + + buf->offset += seg; + buf->len -= seg; + ret -= seg; + + if (!buf->len) { + pipe_buf_release(pipe, buf); + tail++; + } + } + + if (tail != pipe->tail) { + pipe->tail = tail; + if (pipe->files) + need_wakeup = true; + } + } + +out: + pipe_unlock(pipe); + if (need_wakeup) + wakeup_pipe_writers(pipe); + return spliced ?: ret; } #endif From patchwork Wed Mar 29 14:13:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC445C6FD18 for ; Wed, 29 Mar 2023 14:15:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 090306B00A1; Wed, 29 Mar 2023 10:15:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F34506B00A2; Wed, 29 Mar 2023 10:15:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAF196B00A3; Wed, 29 Mar 2023 10:15:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C9BAF6B00A1 for ; Wed, 29 Mar 2023 10:15:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8AD61160D57 for ; Wed, 29 Mar 2023 14:15:23 +0000 (UTC) X-FDA: 80622133326.19.3D3C00D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 85B2140016 for ; Wed, 29 Mar 2023 14:15:21 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QbxP+zm1; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BwLwJU8AwjD+1uYy2mnm7V08kEOLBE2Z8d7v3yKcDiA=; b=vmgbibNHON54PStYtOLkkXyjlAQLXlXRRMcpxwnTPLcwYM71s7ICuU+40bzTbvEdDHsZMt R7bSeVhhNwvKwGIeHMJ/N9zh6Mxba+Jva2UZNZUIHRQsFQoIR8ShkQJktNiZXhwxkoWYvU fS8BCqaq0N0H+4ScIP5nDgc5ja4XL9M= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QbxP+zm1; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099321; a=rsa-sha256; cv=none; b=v3IAsg4dMSvThIKh6n79jZS802vixgq2yjSffrW3Qq+62vx6wMhgYAQVSoL4KrVtHAAG/g HfQIlNNnjFUk0Yq/p5t2C8Lhk2RvsM7snBjtz7pRcHw5pMasDvK6poHHatc89g4/7yYimQ htJxaZzfKzV8FUaEidWIEofeJYN38xI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099320; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BwLwJU8AwjD+1uYy2mnm7V08kEOLBE2Z8d7v3yKcDiA=; b=QbxP+zm1WppsMQjBQth+98g/WeZk+j8ibWz+G7veLPLMcIrmhM+FlI1Ooco3E0FktGBe3r PvZmDNA5Mq8xopMV39607enNsTJGGM+sAxv1HWByYcQ/DhTaPGRTYkZvEil45+0C/21vCR DKJwTVzEK7IXWPaj5ZL6ZLTKnPZnu+E= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-1-P4yHE0I5OgCg_PbyvAV0fw-1; Wed, 29 Mar 2023 10:15:19 -0400 X-MC-Unique: P4yHE0I5OgCg_PbyvAV0fw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 23AE23814951; Wed, 29 Mar 2023 14:15:18 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3CE2E2027040; Wed, 29 Mar 2023 14:15:16 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 29/48] Remove file->f_op->sendpage Date: Wed, 29 Mar 2023 15:13:35 +0100 Message-Id: <20230329141354.516864-30-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 85B2140016 X-Stat-Signature: zjjjpi8cckcm5psmq7pbm9pnuxqq66dn X-HE-Tag: 1680099321-776024 X-HE-Meta: U2FsdGVkX1+cRCQjLoyUWFryV8Bdo18DnEQiA0rHO/dTnEg42g9lD7xFuvsKzvc48jWwaxMwNQR051R4C5nNuWxLGQvddYLtSKBhKR+LMj3K5/DIBToxtUltQLsnFh5oQGwc0Zfm+dYPvhzovmo590sr5Bjf7JJ8vz3ZrKD3wahB0lPJSRmeAuadDoD2Mb9WZ/XHi038M9VoNdxq2Sia+LPkHQupBvjJMpaDtvT35UOZBMjkvvPKbFd98bM5S0sYSlMc/4KOUo8X0ZW0ywGdpod4XlL0F1/H+SDXIx0vl8oetuqShz/xAaiO19KjXTG+F2Ru4B8ckIfqt55w0e7r/vebjWdSndVeocOi3MwPwsYn+cDU2sDgirxxLtMWdN28hC+YYLRx9ZSKq4Eqiqi32YN5UpVvsyIde2L5vSqD2zIk2MPw65PG7vcJPKp6HKSdiXFBCE2ekG2iqP9Z505tQRmt9uHMxDffuN40TMU33RnkFyV8hwNMXobtXGVaRWhtvmZsGCYOtqvSOHfBvRa3voe7vDbfESpnn4X3aj43HzO0KZVoaWadoo26C7irdJbfSjCItKjW/QGRDV2puH6vn3K21IDSWq+eo0tWCOA8owt3atFETOZEyz6anNqnJQb/zlA3QzyxgNqTvY5NxTbNcfqP9Hl+UMSQYMHKsg9mDXBtLbmBcMzOW52LVGbFi8rxPtfLNxIP1tXwOnyWvnocoZ2nR+6LCLNun8wqHhUm6bfqQWbgc/aIy2esCEQaOPjp34Yt4klFl7GG8eDGP4oEme2/7KJYU5f5iAboPRXyrdPJpXWdDe0XnMgMXbrResh77wEttyM3B50aoRj1k3884CF1CwskXsrA0xOR1dKPQrlk/2pLA1V5vETX3X4QovgMPOQAodp//Vq3BmhI1KnsL/BjyOp9lq0Qr41GTLho8aK8lVUkHcChgLw3RDLEnPI5+o4KdD4kPBVXFxIykAs CEbH57RM EYF5pcJ0tbNdUPP/dE/0ri48ZjrN7F1FZ63NMBIh7nUNDeFgtpaRk0udeR57jN8r9hpB0NoVzqVBS1ywa0dPvOfxZICogzcCZwQHkqTH+gy5TEHghC1JYUkdYOiECiDg6FTNhaILrHVwl156nhyZDuLrGXxoeG1OTZ5xZzqQLaJLU/MwjI3gKavMhCEIVDkfQLuzK/gCg9UVC+qu19IToc2Jxl5fxiHLHHYso4O4OIk7iCtCgLPT1LTPz5j3jddYUt4uD1mtYouWs9ZdjzxBAcla2ki/qZ2OmrKFXfPmD15UVPpGl+QdOxJZV83AMIjkz6pCxCYQmclb7O1JokPP+j3jGcVEPxRTLN5YVaDnyyfJ08qhp6F4P8B+zT62QFuZKUHfQpuVW5JdooTySn9WLNGfhUtH58FhkBAnIiZK7cwnfJSgyiPYO1O88Ym9jPt0qOwJV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove file->f_op->sendpage as splicing to a socket now calls sendmsg rather than sendpage. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/fs.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index f3ccc243851e..a9f1b2543d2c 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1773,7 +1773,6 @@ struct file_operations { int (*fsync) (struct file *, loff_t, loff_t, int datasync); int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); - ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); int (*check_flags)(int); int (*flock) (struct file *, int, struct file_lock *); From patchwork Wed Mar 29 14:13:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A10E7C74A5B for ; Wed, 29 Mar 2023 14:15:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C4856B0082; Wed, 29 Mar 2023 10:15:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 375836B00A2; Wed, 29 Mar 2023 10:15:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C57A900002; Wed, 29 Mar 2023 10:15:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EEECC6B0082 for ; Wed, 29 Mar 2023 10:15:26 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B8FF41C61EF for ; Wed, 29 Mar 2023 14:15:26 +0000 (UTC) X-FDA: 80622133452.29.A2A63E3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id E598D140029 for ; Wed, 29 Mar 2023 14:15:24 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VazfIy0l; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uOfuvlKLCw3PwI6FlP4CG+iWtw5a+KyQbWE9E7aqyN8=; b=rv8xNsxWqrPqq317LVBwPV0ZpugmTzugCVolUl7ljgDIflSeTZvwDcn5r74aUR7u1nus33 jJ4rE37CytpPfHGdtbAXAqewoq/MYy/Yt+e+IB0oz+HDDKf5O49ffdFpH3UHqYOeZDGq1w OEM/gYhMNEbUSqMGG333urJqVbcCOrA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VazfIy0l; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099325; a=rsa-sha256; cv=none; b=yFGuTlMt1l1HSntvJmdQDOINy2DT51PMph+3t8YtrJ66pfp+LmkEeUwmRXLXbv2Pg+yLWZ RHXDPR6can+sdP4N3wjcslWjHqxjbbGhmfEX5BHeMBC7T7lTBbWAFVlRKJzpG8E3eTvWZK N4vnDJ+0cMXRq0DMsH3xJlHVwZc4ESg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uOfuvlKLCw3PwI6FlP4CG+iWtw5a+KyQbWE9E7aqyN8=; b=VazfIy0lN/Cc9yHvozVU3HQuHLf8JeMjHnkkbY6vwneF7cHcrc0BiUcZcp3UPQLUWEw7Wh mkQqm7sEkMIdHcG3ku+UKEidntosDzq3fXN40a7b7kjJ+uUox/t4XZ9pUGR/6UTyqPQ5Lk shlqGcQayFT+RZMX10E4bJQyDS8wAQU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-70-Ff7TQ3BGOAynU7ztrXIohw-1; Wed, 29 Mar 2023 10:15:22 -0400 X-MC-Unique: Ff7TQ3BGOAynU7ztrXIohw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E073A1C0758E; Wed, 29 Mar 2023 14:15:20 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B3D6F2027042; Wed, 29 Mar 2023 14:15:18 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [RFC PATCH v2 30/48] siw: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage to transmit Date: Wed, 29 Mar 2023 15:13:36 +0100 Message-Id: <20230329141354.516864-31-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 1acjc3qq7ku7jqk53cpqzfjrktmzgbh7 X-Rspamd-Queue-Id: E598D140029 X-HE-Tag: 1680099324-515447 X-HE-Meta: U2FsdGVkX188jBapd3pUjnpy8D9VA9ynObXPzXQQCHhvFPWX8MUlFibOgeZKBWXZrpuWam06mgzpaJrnklxtmtEqXzuEroAbJZK9LTdrF5Gt+ZobeFNfu1/oN89E+OxiQTY9W7Jl1UAnJd3V5engJ8cZE2sZiQtWlJmtQW8TaZrcdstfcbBjkSxS3rAUK/dwjGGd25ljJY3DfZIuO42VZh7axCqDsbpG6dgfKSXUlH0qDC1+u+Y948xmFWRTCZrgVuxpmiDkAykE0YZAh0XHd6Apffk0HnvFBhZzGTTaN5VGHsd/mPU6jBeUADUYcwjbYtnoFANihcfGOrF6YntmlZIWBNrviTZRje4gith8PuYMZ6vM6+gh5DwEx/emI38jfejgv6lIJLHoW1gWDfiL+LQSb9UZYiBlCNQ5tsLNNYBMaDASThBmCsaNm/e+oYhzQDizcaSxgV7pO1JI3kujYJYChrHHVdW1kfOjHLKCynTqcbs/3hdNh4TDoz+GAeG5hXiOCa+zoAL0M7O45r8M0pzctxz4JAqkM8hUwaZ4/i1QSzfOjHOazWcjCCxT0nTdKf5Bw8eBZ7zDEbQqodw1kOHT6nbEB8KDrknfAgNB83RnysrYED0srCLXTUmQgb2Rybe1ILWeGV0stSEs5y2DChOjCrHvP30dtvosA1tDFm+f0gmicF39Wg2xKRb2wKMWsNw8OziuNQ115uzXSTGZVbMnI6i3jlDyYbJibkp1YdFjWNf6IQgEbfi2dK+Kk9m8LGTIRFwPJWF2+XJz78LDpkNBvxzpPYxdDAToSI63yTcqfAT2skFVQrW9BXWzoP+/wtwTIwRVgRPHC8Vc8UKVAB5R57JGDXljgXRD8O8w2+j4tOQkEaAcKQBdO0BZR6D+h4cPBShOPfxqBay8hltCcSR3t4+GoXSEvXpmfONjt/CtgiANywDIvy3fj3663ln5U4N1m9PWHy2ox2wSqA2 QFZX5Bjx DF/Zn3i00F/d4tlf05oXdCYyLZhNkgRFLO8EOksMHfJYAU9wz2kQJr4cX6p50+KW6KE7dtHqfPvrd8Vlyraj9ArVyd0BLtMBu2RlhmW2jKg29WnDgIZZp5zP8mJiUNC4p2ffZzrZ40x1bYETzPZe2II2FZst7Lv+0LrE+mwhDPTNM3ZMJ+er21vEH5qIjclWIM14qVU44ADgNstOmG/Z16GbayNcKAdU4uLJQSR/z2lHQVf2DfonMh4mJ2YYFtvsrHLTM/fD/cu96jzo0NVaras34liPZG+f8GEY8UWYq74cmPWDOYz7rKoN6/azSEJB3XxnwTcsr6UKT+JHiprqb4f6ghWYCA6CWa0KxQPI9ATVZFVCJi98QQ2mavi590ODYz+Ip1pTuaI6kKaswuVC0PYlLjTcjguFos+Etj6L4zqSEnTnEZ63Izbc2Mgs1/lf0MdtN2R1713e0/U9kaT0zLJwWlV/BsJMwz3Q+kfYRSWWY7oRwKHWx3FCaVlHuGkmbv32++I6OmRFE43z9M7QN/3DpCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer (if present) are copied into page fragments that can be freed with put_page(). Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 234 ++++++-------------------- 1 file changed, 48 insertions(+), 186 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index fa5de40d85d5..fbe80c06d0ca 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,114 +312,8 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, return rv; } -/* - * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. - * - * Using sendpage to push page by page appears to be less efficient - * than using sendmsg, even if data are copied. - * - * A general performance limitation might be the extra four bytes - * trailer checksum segment to be pushed after user data. - */ -static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, - size_t size) -{ - struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | - MSG_SPLICE_PAGES), - }; - struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0; - - while (size) { - size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); - - if (size + offset <= PAGE_SIZE) - msg.msg_flags = MSG_MORE | MSG_DONTWAIT; - - tcp_rate_check_app_limited(sk); - bvec_set_page(&bvec, page[i], bytes, offset); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - -try_page_again: - lock_sock(sk); - rv = tcp_sendmsg_locked(sk, &msg, size); - release_sock(sk); - - if (rv > 0) { - size -= rv; - sent += rv; - if (rv != bytes) { - offset += rv; - bytes -= rv; - goto try_page_again; - } - offset = 0; - } else { - if (rv == -EAGAIN || rv == 0) - break; - return rv; - } - i++; - } - return sent; -} - -/* - * siw_0copy_tx() - * - * Pushes list of pages to TCP socket. If pages from multiple - * SGE's, all referenced pages of each SGE are pushed in one - * shot. - */ -static int siw_0copy_tx(struct socket *s, struct page **page, - struct siw_sge *sge, unsigned int offset, - unsigned int size) -{ - int i = 0, sent = 0, rv; - int sge_bytes = min(sge->length - offset, size); - - offset = (sge->laddr + offset) & ~PAGE_MASK; - - while (sent != size) { - rv = siw_tcp_sendpages(s, &page[i], offset, sge_bytes); - if (rv >= 0) { - sent += rv; - if (size == sent || sge_bytes > rv) - break; - - i += PAGE_ALIGN(sge_bytes + offset) >> PAGE_SHIFT; - sge++; - sge_bytes = min(sge->length, size - sent); - offset = sge->laddr & ~PAGE_MASK; - } else { - sent = rv; - break; - } - } - return sent; -} - #define MAX_TRAILER (MPA_CRC_SIZE + 4) -static void siw_unmap_pages(struct kvec *iov, unsigned long kmap_mask, int len) -{ - int i; - - /* - * Work backwards through the array to honor the kmap_local_page() - * ordering requirements. - */ - for (i = (len-1); i >= 0; i--) { - if (kmap_mask & BIT(i)) { - unsigned long addr = (unsigned long)iov[i].iov_base; - - kunmap_local((void *)(addr & PAGE_MASK)); - } - } -} - /* * siw_tx_hdt() tries to push a complete packet to TCP where all * packet fragments are referenced by the elements of one iovec. @@ -439,15 +333,14 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) { struct siw_wqe *wqe = &c_tx->wqe_active; struct siw_sge *sge = &wqe->sqe.sge[c_tx->sge_idx]; - struct kvec iov[MAX_ARRAY]; - struct page *page_array[MAX_ARRAY]; + struct bio_vec bvec[MAX_ARRAY]; struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_EOR }; + void *trl, *t; int seg = 0, do_crc = c_tx->do_crc, is_kva = 0, rv; unsigned int data_len = c_tx->bytes_unsent, hdr_len = 0, trl_len = 0, sge_off = c_tx->sge_off, sge_idx = c_tx->sge_idx, pbl_idx = c_tx->pbl_idx; - unsigned long kmap_mask = 0L; if (c_tx->state == SIW_SEND_HDR) { if (c_tx->use_sendpage) { @@ -457,10 +350,15 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) c_tx->state = SIW_SEND_DATA; } else { - iov[0].iov_base = - (char *)&c_tx->pkt.ctrl + c_tx->ctrl_sent; - iov[0].iov_len = hdr_len = - c_tx->ctrl_len - c_tx->ctrl_sent; + const void *hdr = &c_tx->pkt.ctrl + c_tx->ctrl_sent; + void *h; + + rv = -ENOMEM; + hdr_len = c_tx->ctrl_len - c_tx->ctrl_sent; + h = page_frag_memdup(NULL, hdr, hdr_len, GFP_NOFS, ULONG_MAX); + if (!h) + goto done; + bvec_set_virt(&bvec[0], h, hdr_len); seg = 1; } } @@ -478,28 +376,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } else { is_kva = 1; } - if (is_kva && !c_tx->use_sendpage) { - /* - * tx from kernel virtual address: either inline data - * or memory region with assigned kernel buffer - */ - iov[seg].iov_base = - (void *)(uintptr_t)(sge->laddr + sge_off); - iov[seg].iov_len = sge_len; - - if (do_crc) - crypto_shash_update(c_tx->mpa_crc_hd, - iov[seg].iov_base, - sge_len); - sge_off += sge_len; - data_len -= sge_len; - seg++; - goto sge_done; - } while (sge_len) { size_t plen = min((int)PAGE_SIZE - fp_off, sge_len); - void *kaddr; if (!is_kva) { struct page *p; @@ -512,33 +391,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) p = siw_get_upage(mem->umem, sge->laddr + sge_off); if (unlikely(!p)) { - siw_unmap_pages(iov, kmap_mask, seg); wqe->processed -= c_tx->bytes_unsent; rv = -EFAULT; goto done_crc; } - page_array[seg] = p; - - if (!c_tx->use_sendpage) { - void *kaddr = kmap_local_page(p); - - /* Remember for later kunmap() */ - kmap_mask |= BIT(seg); - iov[seg].iov_base = kaddr + fp_off; - iov[seg].iov_len = plen; - - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - iov[seg].iov_base, - plen); - } else if (do_crc) { - kaddr = kmap_local_page(p); - crypto_shash_update(c_tx->mpa_crc_hd, - kaddr + fp_off, - plen); - kunmap_local(kaddr); - } + + bvec_set_page(&bvec[seg], p, plen, fp_off); } else { /* * Cast to an uintptr_t to preserve all 64 bits @@ -552,12 +410,15 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) * bits on a 64 bit platform and 32 bits on a * 32 bit platform. */ - page_array[seg] = virt_to_page((void *)(va & PAGE_MASK)); - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - (void *)va, - plen); + bvec_set_virt(&bvec[seg], (void *)va, plen); + } + + if (do_crc) { + void *kaddr = kmap_local_page(bvec[seg].bv_page); + crypto_shash_update(c_tx->mpa_crc_hd, + kaddr + bvec[seg].bv_offset, + bvec[seg].bv_len); + kunmap_local(kaddr); } sge_len -= plen; @@ -567,13 +428,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) if (++seg > (int)MAX_ARRAY) { siw_dbg_qp(tx_qp(c_tx), "to many fragments\n"); - siw_unmap_pages(iov, kmap_mask, seg-1); wqe->processed -= c_tx->bytes_unsent; rv = -EMSGSIZE; goto done_crc; } } -sge_done: + /* Update SGE variables at end of SGE */ if (sge_off == sge->length && (data_len != 0 || wqe->processed < wqe->bytes)) { @@ -582,15 +442,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) sge_off = 0; } } - /* trailer */ - if (likely(c_tx->state != SIW_SEND_TRAILER)) { - iov[seg].iov_base = &c_tx->trailer.pad[4 - c_tx->pad]; - iov[seg].iov_len = trl_len = MAX_TRAILER - (4 - c_tx->pad); - } else { - iov[seg].iov_base = &c_tx->trailer.pad[c_tx->ctrl_sent]; - iov[seg].iov_len = trl_len = MAX_TRAILER - c_tx->ctrl_sent; - } + /* Set the CRC in the trailer */ if (c_tx->pad) { *(u32 *)c_tx->trailer.pad = 0; if (do_crc) @@ -603,23 +456,29 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) else if (do_crc) crypto_shash_final(c_tx->mpa_crc_hd, (u8 *)&c_tx->trailer.crc); - data_len = c_tx->bytes_unsent; - - if (c_tx->use_sendpage) { - rv = siw_0copy_tx(s, page_array, &wqe->sqe.sge[c_tx->sge_idx], - c_tx->sge_off, data_len); - if (rv == data_len) { - rv = kernel_sendmsg(s, &msg, &iov[seg], 1, trl_len); - if (rv > 0) - rv += data_len; - else - rv = data_len; - } + /* Copy the trailer and add it to the output list */ + if (likely(c_tx->state != SIW_SEND_TRAILER)) { + trl = &c_tx->trailer.pad[4 - c_tx->pad]; + trl_len = MAX_TRAILER - (4 - c_tx->pad); } else { - rv = kernel_sendmsg(s, &msg, iov, seg + 1, - hdr_len + data_len + trl_len); - siw_unmap_pages(iov, kmap_mask, seg); + trl = &c_tx->trailer.pad[c_tx->ctrl_sent]; + trl_len = MAX_TRAILER - c_tx->ctrl_sent; } + + rv = -ENOMEM; + t = page_frag_memdup(NULL, trl, trl_len, GFP_NOFS, ULONG_MAX); + if (!t) + goto done_crc; + bvec_set_virt(&bvec[seg], t, trl_len); + + data_len = c_tx->bytes_unsent; + + if (c_tx->use_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, seg + 1, + hdr_len + data_len + trl_len); + rv = sock_sendmsg(s, &msg); + if (rv < (int)hdr_len) { /* Not even complete hdr pushed or negative rv */ wqe->processed -= data_len; @@ -680,6 +539,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } done_crc: c_tx->do_crc = 0; + if (c_tx->state == SIW_SEND_HDR) + folio_put(page_folio(bvec[0].bv_page)); + folio_put(page_folio(bvec[seg].bv_page)); done: return rv; } From patchwork Wed Mar 29 14:13:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37BC3C6FD18 for ; Wed, 29 Mar 2023 14:15:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEBD26B00A2; Wed, 29 Mar 2023 10:15:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9B616B00A3; Wed, 29 Mar 2023 10:15:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B62726B00A4; Wed, 29 Mar 2023 10:15:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A6E4A6B00A2 for ; Wed, 29 Mar 2023 10:15:31 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 51869C0BF6 for ; Wed, 29 Mar 2023 14:15:31 +0000 (UTC) X-FDA: 80622133662.21.DF3E497 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 7E5D3A0021 for ; Wed, 29 Mar 2023 14:15:29 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IfNhNpFJ; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099329; a=rsa-sha256; cv=none; b=mPOd2WIxKG7Dj5HplhkqdtV+ON0Pm4d8+Nb+56c+ZNaJBxN+L8wJ5tvJPHsvyIkhO5DGCz zJFp8gbMJAhO15fEvkTRetzP5L2DFwQcbv2KbYEufG1zjq6Ci7M5lOftHV+zVMQIixFcj2 cSChLDEwdPgcy1Q8KiTegzHYZ60H8Eo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IfNhNpFJ; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mcxOsbWhFQyukCMWrPOKlIpXJGoD9GGTGJfL3NnBqs4=; b=VSlTT2XJU9QElo3Dp+iIpwmTsL8tCHsTbtgih0Qf3MKfWx5ofO4WB4UMRmXwvmMN6UgQR/ FScZe2WldDw8uPrQUS0TySD3ATcH9akI7PXJaq/0n3NVKOfW5TuN+JX2yc9MUUIxQGGjMw eQo228G6YHpGuYX1DgB0N8MD6lI8c64= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099328; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mcxOsbWhFQyukCMWrPOKlIpXJGoD9GGTGJfL3NnBqs4=; b=IfNhNpFJtF2v/kzPgqhARM1T1CqL+SQUKkfgQE/7iladQVZsSi4LJCmdLcr7Sksa8rx1pX M7xmTdpasgthwc6kEgG0l69bS0JwIb3wcT/SMdSLE8o13EoDpyhkjPSTz5rtnZEmqVp2HY JjPhzOKgnCocKO8g/CrmHSIrtuXz9X4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-67-R6b5KJCXPSqhwAA81fk1XQ-1; Wed, 29 Mar 2023 10:15:26 -0400 X-MC-Unique: R6b5KJCXPSqhwAA81fk1XQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1BD933C0ED4D; Wed, 29 Mar 2023 14:15:24 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9FF1D14171BB; Wed, 29 Mar 2023 14:15:21 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , Xiubo Li , ceph-devel@vger.kernel.org Subject: [RFC PATCH v2 31/48] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Wed, 29 Mar 2023 15:13:37 +0100 Message-Id: <20230329141354.516864-32-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Queue-Id: 7E5D3A0021 X-Rspamd-Server: rspam01 X-Stat-Signature: 8or6kisarz8ncbnzwn5wh8oxte4rtz14 X-HE-Tag: 1680099329-849071 X-HE-Meta: U2FsdGVkX1+Drjr8MSPkeQzA9g2GuXCGgAHPdnMrpFFYIYg3KfkXBVl7yhFwVKwbNkF6Qj0LAF6f4JBkxL3roKELLn/5ryWs6P4eQZnbWcOcyYvnPHaPSK9zGtuOt9QfcZ7yh3J2Gu8exd9ZKILm2zQKwpDSGVre+4lBigvdHRvJmLWk80lHLOWIeL0/2s4ueYidzYjMZabQBNE04bvHQG7hAWmoQBtfj9dXWPxI/+YWtdfrwBW7zZf+DDVCJ7OQACdtSlH6t1Q6hZcbmwOWXbwRPkNUQfsxXNJ6wTuO1U2cruPrVIVD7TwpJBBgCjkRU7T5FN2w9wxDGINj8GdPuKIP0hA9a0hZmDe34I6rScwbtDU7oicMIoSfs8uiKugra+pzUjFuChtrEyiJN8IM3S7NQ0z3P+Y+CfqgM3psMZhUpdTvdEsDsrBR4WXSzyLTQYw50hFRkkagsE2qelfxxxPVuyj5DguoIV+tD5sTfdWtt/ycjmkJosKO+MB9luutoRI6Eyzl2JGGjkENSOMzN+JyNmSlu2m33bAHwFPdXIfFhzpMfiNqa42RvJuKAX72UX8xl4lJ7BqtX2LvRqMQO5jWkGpzhshIhd0PdE+w1dR579y8ZWTXlyoeV3Zed9bpp7lq5+cXkZkKPZCHwFH98zwYJwhbS1ZsKeYP1SQG7Xi4rszkLr+kJjlcxnLYD+6OvOx3cr87zufl54rIfNoX2iinuw/Fg/dZrUfOjQWFRyJwKrQ49f098LVdyML4NagmZa2VMSCQCEA2nfEScjKPD5/k0N3YbiwERFZBDsr3J3eIOjK/cvKUQkvkBnngdkgzKFmZDxtyX7tQ31jcI9eQdRyMTiyDhkIz8J/fBCRXYknYpDzR6jiscesBm6UYB+5kJB0vKkpqeN6UTRy/0h9hK2eIl1PkiU6T4Ut47Hac0Q4ffTWgRsX/bKGHuCsmHjwyhCBDy2MzlWaM7OCPTBW ukjHjdS9 yehLp2L0xDi4qDannTYRtHRR+RNoC8xlki3XDLqJslLcVxThd0NLspBvpvB4da3qf/wgpeBIP1K7tKGC5yReNKiIVCFo/XmrxwWMC5JA2evIpx7dd6kDypFUJS8opLjDdbJekizPrz1jbk9Np9xJgxY/79sSQUhhLL3HKOMycW7uiwDaxtgX7ZEgJaPayOxw3x/Vg3M+TyhBbjpCaBHCiuf3wAJY9W9Qutc0Hs2Ow8Wb/e0/xDYNm/Tf0kJx7XWimLopQddYvMdS0296DJxKorcQ0NZXcognWF6HpDF/jJsRKJuc8PrKdhvJc43Yn/6ntcJWU2EvwRu+4cU96xvBJcby89s6KsKEp+fIQRyRxgCNd3SSZqRtIepQc61b/aVJA4DwsM541DOOQzA3vzSgWH3CWy2EX3T7L+iJtzb8fW4qdLsTEHZcO+DdM1avkZ8YMpfzoYm4JHMCo7X41fwTB12hfKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v1.c | 58 ++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 39 deletions(-) diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index d664cb1593a7..b2d801a49122 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -74,37 +74,6 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov, return r; } -/* - * @more: either or both of MSG_MORE and MSG_SENDPAGE_NOTLAST - */ -static int ceph_tcp_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int more) -{ - ssize_t (*sendpage)(struct socket *sock, struct page *page, - int offset, size_t size, int flags); - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - int ret; - - /* - * sendpage cannot properly handle pages with page_count == 0, - * we need to fall back to sendmsg if that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag which - * triggers one of hardened usercopy checks. - */ - if (sendpage_ok(page)) - sendpage = sock->ops->sendpage; - else - sendpage = sock_no_sendpage; - - ret = sendpage(sock, page, offset, size, flags); - if (ret == -EAGAIN) - ret = 0; - - return ret; -} - static void con_out_kvec_reset(struct ceph_connection *con) { BUG_ON(con->v1.out_skip); @@ -464,7 +433,6 @@ static int write_partial_message_data(struct ceph_connection *con) struct ceph_msg *msg = con->out_msg; struct ceph_msg_data_cursor *cursor = &msg->cursor; bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; u32 crc; dout("%s %p msg %p\n", __func__, con, msg); @@ -482,6 +450,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ crc = do_datacrc ? le32_to_cpu(msg->footer.data_crc) : 0; while (cursor->total_resid) { + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST, + }; struct page *page; size_t page_offset; size_t length; @@ -494,9 +466,12 @@ static int write_partial_message_data(struct ceph_connection *con) page = ceph_msg_data_next(cursor, &page_offset, &length); if (length == cursor->total_resid) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, page, page_offset, length, - more); + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, length, page_offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, length); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) { if (do_datacrc) msg->footer.data_crc = cpu_to_le32(crc); @@ -526,7 +501,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ static int write_partial_skip(struct ceph_connection *con) { - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST | MSG_MORE, + }; int ret; dout("%s %p %d left\n", __func__, con, con->v1.out_skip); @@ -534,9 +512,11 @@ static int write_partial_skip(struct ceph_connection *con) size_t size = min(con->v1.out_skip, (int)PAGE_SIZE); if (size == con->v1.out_skip) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, ceph_zero_page, 0, size, - more); + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; + bvec_set_page(&bvec, ZERO_PAGE(0), size, 0); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) goto out; con->v1.out_skip -= ret; From patchwork Wed Mar 29 14:13:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35107C6FD18 for ; Wed, 29 Mar 2023 14:15:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C279F6B00A3; Wed, 29 Mar 2023 10:15:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD7196B00A4; Wed, 29 Mar 2023 10:15:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A78926B00A5; Wed, 29 Mar 2023 10:15:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 981D16B00A3 for ; Wed, 29 Mar 2023 10:15:35 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 760D9140941 for ; Wed, 29 Mar 2023 14:15:35 +0000 (UTC) X-FDA: 80622133830.19.8F4503A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id AE73740024 for ; Wed, 29 Mar 2023 14:15:32 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=B1w1ZF3S; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099332; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gZm+OH3nuU2w5lgBIDJ5zMpRPGFGHnStb+oddo8E9cs=; b=a+KQTzURRCTjBEnYl6Sgyzr37WB64NVKoykkkWZpCTnHT7CLSH82Mcg1bbHOq3Rvxb4nZL iXOblTxY27rrFnKpCspCG4kC+A3rZaOcrT4Vp/KzlRFq0AN7aq9DjZr0AWB+MIr5I5NV/J PowYm+bSc8RiHDGiFSarhuKfKDOv2uc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=B1w1ZF3S; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099332; a=rsa-sha256; cv=none; b=RNYbBLSXGLeh4lA4BYxXtcGjfHsz1nKZbX7jtrN92ag1m1wDfReb6sQudqJkI8OZIEhy/r FdBijsZ9x7TyjsFutEYZEPN9b3xiFHvIWQfT3NE0Gsgz2E2z7PfySNNkYK4Pm3NlDAH8Ru 0oOi5S/53CJSqBeZ8rytJ74lFSlfXOU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099332; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gZm+OH3nuU2w5lgBIDJ5zMpRPGFGHnStb+oddo8E9cs=; b=B1w1ZF3SQs4MDgmV4lwaw3FWVXtS6Y/9MnGgEPsIsOHEBzPYfhMsofVK88/qj8oLlBUw1+ /FDWk2yr8YKB8103wGfVQRcCLRac0YGAo0nENfkWwGWL2ozmIuxowh3g6uQN6y1fiNUf/t UqDB+mUJsR4d3Baj6sDGIm0IpXDzX5s= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-427-ZeMxTUzWPHqK1lFKrZJGyg-1; Wed, 29 Mar 2023 10:15:27 -0400 X-MC-Unique: ZeMxTUzWPHqK1lFKrZJGyg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DB5A61C0758C; Wed, 29 Mar 2023 14:15:26 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B2C022166B33; Wed, 29 Mar 2023 14:15:24 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Martin K. Petersen" , linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Subject: [RFC PATCH v2 32/48] iscsi: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Wed, 29 Mar 2023 15:13:38 +0100 Message-Id: <20230329141354.516864-33-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: AE73740024 X-Stat-Signature: kprs81fg8z93r76nxwsapaxdrrqhkywn X-HE-Tag: 1680099332-63669 X-HE-Meta: U2FsdGVkX196KnaP30444eD2Zi4NdFKWGaGNuyv0VYlPvjFT18UlUdSu2XsNXEtm+uZ61JfDPp8OxsfSNsJFdu7B/6fGCjy/9KS0gvALFQDpfl7wcMiFOF4Mb+zRX15WmmFAP3pWlgGmZUijGNyocxiicu2sfypdOk8wBvyNU0bXO5x9L4Caf4H1qbN/zR8M0K4c19VBYRpP4nU2SzmrOLWGdjqsTv2ujsnVxhEi8rfacgRntfwIItVidysWyzngiyor6+WA1mxAPWwDJvvgF7ih9VIVYOKR4YJKyWYqNpNirEIVa8EXGuZrfnTAE5IEYkek2ZkT6C5MNxNRdYoggnVuZmDEHnylXeipx8PHu5UCBQK9lwyhyx6vtMqldZ9NWnvV/VeZs7O/yZONFfDW14jR/W16KlIAn3O7beneGiS4pYL3QSjYwMtKERaPRSoGeqgVqN5BpDFDtfynU7PNkrgZfvplYeKvQu/DXEuc+j5bGrAiM8Fw5FA4YW4LQFdiUaPVJrl+PiwgI0ZppAyXIs3BUuK0nKc4MzRpkeRKn1QEyoCCt/PzVhHS/3wynFEC1ZBpJRAodTJA+pqlF6PjoFTZXRoevP9TuCiaO0GsaxjjryMaLN3y2Y8a5RqSRC7dPYn2A8q536rPg1qVjCIPFaWI2QNeiFqTHzGhQGifp61tlHKADEtQnbGOnd4TIQh/5XLtfCZ+8vcUF3KrjQVGaR6DLS9gS6/8nipBVuT2r19ZEfIvJ23ru4JxS6n5PosLSV5BiY8QJ8q6+9SXAwy3r47s4kQhoE3+s0ropgFuXbm0XjmctLIDyFWGSuoD+n9z74jgPSVumCMLBqHpBC7pumWO2zmh2/ZNl/bGmogAvDwWkyW0vVEZ90Vg5C/eBTItwa37Gx2lZau3qN0bMMlLryCHFRLJcEx4DzvzZzx6nixhcEKZVAQai1YWgc1kZ0lz+pD53S8Zd2q0oWwAxcF 3eCLAkUG WnC23V9skVZFeoDFtJaQL2Dx5HXO853yoIZZ118ven1yCVV9O6mtu4sR5RpXH7niUgneyR8upqt7Wh14SccMBGF8HNY89tWLyF35zYKaYSw5xFiKVJ8vKig6D+paw+4Pvl3CUtLUX8Mm5wgJlhK6yQ2XVsd0FTnh/D5JHyv+56jfYxaMqwcZMQus/ukFt6P9haXH3U1hNNiO/RcjFJcYpcJQC4o0h9QLG2m/jrmRnuew6zP6iKOemj/aA7BrusHb9IacgkDaPfPPLjCyTQZJVkbRGvr4q4tH6fsFpn+RXsIZdlNkNP4rrt+HrI2ajXtrO0Z114ya5ftMGX2zaUKHyecpsS5AvdzlWu6vzlZ6oKFV4fFS9gyCCCBc6TSaJezJCYaeBFHNHVBlUubJ5edgD+eoPWgw98rMYtqYYOhaSsMdwRfNUKqo82ukGJQ6tyqu19DnPttaUlCb6nekWuikoMs+V/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage. This allows multiple pages and multipage folios to be passed through. TODO: iscsit_fe_sendpage_sg() should perhaps set up a bio_vec array for the entire set of pages it's going to transfer plus two for the header and trailer and page fragments to hold the header and trailer - and then call sendmsg once for the entire message. Signed-off-by: David Howells cc: "Martin K. Petersen" cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-scsi@vger.kernel.org cc: target-devel@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/scsi/iscsi_tcp.c | 31 ++++++++++++------------ drivers/scsi/iscsi_tcp.h | 2 +- drivers/target/iscsi/iscsi_target_util.c | 14 ++++++----- 3 files changed, 24 insertions(+), 23 deletions(-) diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c index c76f82fb8b63..cf3eb55d2a76 100644 --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -301,35 +301,37 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, while (!iscsi_tcp_segment_done(tcp_conn, segment, 0, r)) { struct scatterlist *sg; + struct msghdr msg = {}; + union { + struct kvec kv; + struct bio_vec bv; + } vec; unsigned int offset, copy; - int flags = 0; r = 0; offset = segment->copied; copy = segment->size - offset; if (segment->total_copied + segment->size < segment->total_size) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; if (tcp_sw_conn->queue_recv) - flags |= MSG_DONTWAIT; + msg.msg_flags |= MSG_DONTWAIT; - /* Use sendpage if we can; else fall back to sendmsg */ if (!segment->data) { + if (tcp_conn->iscsi_conn->datadgst_en) + msg.msg_flags |= MSG_SPLICE_PAGES; sg = segment->sg; offset += segment->sg_offset + sg->offset; - r = tcp_sw_conn->sendpage(sk, sg_page(sg), offset, - copy, flags); + bvec_set_page(&vec.bv, sg_page(sg), copy, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &vec.bv, 1, copy); } else { - struct msghdr msg = { .msg_flags = flags }; - struct kvec iov = { - .iov_base = segment->data + offset, - .iov_len = copy - }; - - r = kernel_sendmsg(sk, &msg, &iov, 1, copy); + vec.kv.iov_base = segment->data + offset; + vec.kv.iov_len = copy; + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &vec.kv, 1, copy); } + r = sock_sendmsg(sk, &msg); if (r < 0) { iscsi_tcp_segment_unmap(segment); return r; @@ -746,7 +748,6 @@ iscsi_sw_tcp_conn_bind(struct iscsi_cls_session *cls_session, sock_no_linger(sk); iscsi_sw_tcp_conn_set_callbacks(conn); - tcp_sw_conn->sendpage = tcp_sw_conn->sock->ops->sendpage; /* * set receive state machine into initial state */ @@ -778,8 +779,6 @@ static int iscsi_sw_tcp_conn_set_param(struct iscsi_cls_conn *cls_conn, mutex_unlock(&tcp_sw_conn->sock_lock); return -ENOTCONN; } - tcp_sw_conn->sendpage = conn->datadgst_en ? - sock_no_sendpage : tcp_sw_conn->sock->ops->sendpage; mutex_unlock(&tcp_sw_conn->sock_lock); break; case ISCSI_PARAM_MAX_R2T: diff --git a/drivers/scsi/iscsi_tcp.h b/drivers/scsi/iscsi_tcp.h index 68e14a344904..d6ec08d7eb63 100644 --- a/drivers/scsi/iscsi_tcp.h +++ b/drivers/scsi/iscsi_tcp.h @@ -48,7 +48,7 @@ struct iscsi_sw_tcp_conn { uint32_t sendpage_failures_cnt; uint32_t discontiguous_hdr_cnt; - ssize_t (*sendpage)(struct socket *, struct page *, int, size_t, int); + bool can_splice_to_tcp; }; struct iscsi_sw_tcp_host { diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c index 26dc8ed3045b..c7d58e41ac3b 100644 --- a/drivers/target/iscsi/iscsi_target_util.c +++ b/drivers/target/iscsi/iscsi_target_util.c @@ -1078,6 +1078,8 @@ int iscsit_fe_sendpage_sg( struct iscsit_conn *conn) { struct scatterlist *sg = cmd->first_data_sg; + struct bio_vec bvec; + struct msghdr msghdr = { .msg_flags = MSG_SPLICE_PAGES, }; struct kvec iov; u32 tx_hdr_size, data_len; u32 offset = cmd->first_data_sg_off; @@ -1121,17 +1123,17 @@ int iscsit_fe_sendpage_sg( u32 space = (sg->length - offset); u32 sub_len = min_t(u32, data_len, space); send_pg: - tx_sent = conn->sock->ops->sendpage(conn->sock, - sg_page(sg), sg->offset + offset, sub_len, 0); + bvec_set_page(&bvec, sg_page(sg), sub_len, sg->offset + offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, sub_len); + + tx_sent = conn->sock->ops->sendmsg(conn->sock, &msghdr, sub_len); if (tx_sent != sub_len) { if (tx_sent == -EAGAIN) { - pr_err("tcp_sendpage() returned" - " -EAGAIN\n"); + pr_err("sendmsg/splice returned -EAGAIN\n"); goto send_pg; } - pr_err("tcp_sendpage() failure: %d\n", - tx_sent); + pr_err("sendmsg/splice failure: %d\n", tx_sent); return -1; } From patchwork Wed Mar 29 14:13:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AA30C6FD18 for ; Wed, 29 Mar 2023 14:15:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D88A6B00A4; Wed, 29 Mar 2023 10:15:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2881E6B00A5; Wed, 29 Mar 2023 10:15:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 150706B00A6; Wed, 29 Mar 2023 10:15:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 02DAA6B00A4 for ; Wed, 29 Mar 2023 10:15:41 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CC76C160D5A for ; Wed, 29 Mar 2023 14:15:40 +0000 (UTC) X-FDA: 80622134040.14.A7754DE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id C1AC0180022 for ; Wed, 29 Mar 2023 14:15:38 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c78w6wnc; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0V0VYuNRGUsctVySLnEH8GinGQ4l5nQcc0Sx6Q5aq0A=; b=ftOTAnXzy09RO7Q0SvVW9J8BRaD5hTWqdMQQNQCMc48eMtfq926DfZJRnzvPBUoVC3VZSi rNbkmjL04iSLxGP1QfjYVCx4j7hT1us/bDGhtPnd978n37EUQxFWX39ad1t3QsvyohyGzC 69eN98b5M95z2aW2WOzKH9Y+VTPMbLk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c78w6wnc; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099338; a=rsa-sha256; cv=none; b=q+XlAQ5Dy15A5QVcXtqMRdHiYMK138w9DvHx2GvPZvtuNCQtGsG/9PcpsavRxHaV3RodkH /fh3wbVkuItLQSaOAU+PsZCWT1jDXPB4zuzr+M08LsavTNSvsvwmmsDS+tpOTsUzgjK9g4 KEiYXbs6sS4etWUE3eJUNs+bxnJUGxw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0V0VYuNRGUsctVySLnEH8GinGQ4l5nQcc0Sx6Q5aq0A=; b=c78w6wncXK2A7qKBNp/fqp/QLdog2d7a7+Y5riDnZFpKgCuQJ2A3HhS6otgSGW1eMfIgI1 uVeWcEMz3sZgfa5fdgtpVM1MHiGRO7STJ6WiN2FnnpiqYwwV0G2ocWgMLnXfKEPV8GrZBr oW+sF5ijcJT9WBVx6yfv7g/mFiq7EpM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-167-kCehxZq9PD6SIf6owZvSow-1; Wed, 29 Mar 2023 10:15:33 -0400 X-MC-Unique: kCehxZq9PD6SIf6owZvSow-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C7FF3185A790; Wed, 29 Mar 2023 14:15:29 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9BD4C1121330; Wed, 29 Mar 2023 14:15:27 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Martin K. Petersen" , linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Subject: [RFC PATCH v2 33/48] iscsi: Assume "sendpage" is okay in iscsi_tcp_segment_map() Date: Wed, 29 Mar 2023 15:13:39 +0100 Message-Id: <20230329141354.516864-34-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: yrb83c1fqg1moeq4eosuiqhe1rhd6kji X-Rspamd-Queue-Id: C1AC0180022 X-HE-Tag: 1680099338-278962 X-HE-Meta: U2FsdGVkX1/5d1rYGF5aGCSjVPdq8In/9kL4VcBIgD1kd4Sh0CU1fmFHq93K0acAPd8HWncX1dAvmouWE5lpOp1w6/V6cCoP0ugH2neKnmHZ2e4/rsEYJFVrEGj65DHVByQO6inidrLYMQqMXsEnHe/GRhTz3wqEc9ivCVvmNdip1WljcI1HzHn4ndzYwqMWT2wpcl2IJUKEjXLWqKL53W5/JkJOe+2qQQJ7MJ2/P14WmWQ9x4GQySOJ+JEnBd5Xc/GjFHgCtcAb++ydqGw2sRH8/FqeLUg5FLaOZbj5V2gmgN1pEvXT2aSd4fCbs4F7HpV2XRHltcF5ItkmkXikbXo/4KqWvA8A549GAYAT3X367InY9IBznP3tNCm9sMffXomVtz6ZjlrR8aFHX6Mrl5r44z1vMRxz/WsFaNuLXxYUcoeesTa+93XwGkwEeMc/G+ii7e4jeiyBZ/13m5xF5exCH7X7itSqDfwgR4ydPDxlmdAefGC9lBVoOdkK9EETRQepHmXDJiKF55T27tnb4xhJC65y8wGrxfSkNupeHBDIzeQOpWfSCdwj5JCwC1viAa78lJrPIjYHzsJr/fwESn9AlssEMV/ySDQ6WDKCWpDyhYzTpRP7gL0Im/WZOVwRh0tk8x554AthmyWP5X2q40NPPCXnDrXgX2ZivPIBNFmAVGZMLUNb37LqO5sMooUbW0v9h9Djd8arN/S8w2YJXwmXHy93YN8ZDgXgHLWyO0Xwt7wI9mOryGdEKak2fs1ctENjVE20ZO8JkARzlVL4Gx0w4H2ejXOfhwwgco9DEH9NQcg330W0F927rWFjAV0u58We4F2/ezgQJ9EftuX1QuKLm0vr1IkuEhEoalUC1/rHw78upArSWW6T9g9LtRsA2PJu1rmbJVoSV6+6sTmRPMjbZNTYAWkRBJEHR+UQKGzNPH5Y50Gq43aMo9KKMIbB7j7m+bLRJYhWyZ8l1Xv FVTRJy6V uwLKDCIaWGp4m7fb6KzksTrxfQeyPfU5GS6dmx70x2zsJ/myfk3Wb5tqJFMDpibWxXBdYSqRh1mpaMhMuBS8sD5m4ZBmPtVPP0D4VpeC+ITW15Zt3VMHO16P+KrKBE10XR+9EXH197CkcLUwUrnQAWzwL/prZZpgM3wedrHQT2QewCxcKx6drVEOPbO89M0+tR1ai6XGDE3Rp2pyIcys9ubnXjgGCoTv8rmCpErVeC/WDkJXpmdPrwRA65+Q0IWnxCULxsr329R6SScRMGzWgWSOXe87uljCOOBhHGes4gIGyyP/WkUAiZRu4a885GcPsknMIfiZVqRtcZKS/EV+FQ9vGaw38EebMEzOw1/ll6meP8FLr4oI3nFGwq2OAdWaSJMUIQyLy6pexK0wDMlzwToXEm1U39WbT9dNMu8B2ANZvfyi4dd58WPaVaXbc7CAHmlhhO+MKLtAuyxeAL8OJGgN83A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As iscsi is now using sendmsg() with MSG_SPLICE_PAGES rather than sendpage, assume that sendpage_ok() will return true in iscsi_tcp_segment_map() and leave it to TCP to copy the data if not. Signed-off-by: David Howells cc: "Martin K. Petersen" cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-scsi@vger.kernel.org cc: target-devel@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/scsi/libiscsi_tcp.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/drivers/scsi/libiscsi_tcp.c b/drivers/scsi/libiscsi_tcp.c index c182aa83f2c9..07ba0d864820 100644 --- a/drivers/scsi/libiscsi_tcp.c +++ b/drivers/scsi/libiscsi_tcp.c @@ -128,18 +128,11 @@ static void iscsi_tcp_segment_map(struct iscsi_segment *segment, int recv) * coalescing neighboring slab objects into a single frag which * triggers one of hardened usercopy checks. */ - if (!recv && sendpage_ok(sg_page(sg))) + if (!recv) return; - if (recv) { - segment->atomic_mapped = true; - segment->sg_mapped = kmap_atomic(sg_page(sg)); - } else { - segment->atomic_mapped = false; - /* the xmit path can sleep with the page mapped so use kmap */ - segment->sg_mapped = kmap(sg_page(sg)); - } - + segment->atomic_mapped = true; + segment->sg_mapped = kmap_atomic(sg_page(sg)); segment->data = segment->sg_mapped + sg->offset + segment->sg_offset; } From patchwork Wed Mar 29 14:13:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E09CAC74A5B for ; Wed, 29 Mar 2023 14:15:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EE02900003; Wed, 29 Mar 2023 10:15:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19EC3900002; Wed, 29 Mar 2023 10:15:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 018C4900003; Wed, 29 Mar 2023 10:15:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E2C97900002 for ; Wed, 29 Mar 2023 10:15:48 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6A0AF1A0653 for ; Wed, 29 Mar 2023 14:15:48 +0000 (UTC) X-FDA: 80622134376.23.7009E74 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 9B94B1C0039 for ; Wed, 29 Mar 2023 14:15:46 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="B7/Li8vP"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099346; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cLmy3QT3PxPcXOXP8Tmrcj7ottmHeL6VPbhO66kxXIU=; b=7tAhM1mobQ15tvz4Bnv4FXWoQ2QeadBsKCrZAoQ4skmtePtFqYgo/usk6s3pDGxv9KbZSl qeR7PS6jECpuD+/1QycAZCcC2UL9hdnr7z768xt49WzccdJtcXTzYnaQmsXhVUz2OW6wU4 QCjtfkomiGQ9K0MZ8yKI31FNtXONR5M= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="B7/Li8vP"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099346; a=rsa-sha256; cv=none; b=Kk5uYPgJuvHU0r7s6TCL4inJVmrlCm9rDNEuI28N8FRg9s065nuMqd/7TY+Nl6A91gTQOr va+cuG7TjWV8MUHWgBt6Xp2/S+lcbnhal0hFRH69EBJP2U5/MGCyGc7qzcVQCI4HE+eSdJ bwzX5IzW6ojrQMokGcDlMdqWl2DAI8g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099345; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cLmy3QT3PxPcXOXP8Tmrcj7ottmHeL6VPbhO66kxXIU=; b=B7/Li8vP54U6ZNw+x0ktzje7bcG2jarXaqAb+uhKJRybtsBxbErhfv/erhaKLeOy0qyjlj 5pumXGQs5nGsAyX8buKYRap7HXJXWoIge+MXEnLnMTyASM1SJOZ/oePZHtphkvvOrNTeb8 Kv1+M3lvlULCkABM0sTqdfNc43tYOxs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-avHEhrC2M-SiXN4UZyk2qA-1; Wed, 29 Mar 2023 10:15:42 -0400 X-MC-Unique: avHEhrC2M-SiXN4UZyk2qA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9129B85530C; Wed, 29 Mar 2023 14:15:32 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 64BD62027040; Wed, 29 Mar 2023 14:15:30 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [RFC PATCH v2 34/48] tcp_bpf: Make tcp_bpf_sendpage() go through tcp_bpf_sendmsg(MSG_SPLICE_PAGES) Date: Wed, 29 Mar 2023 15:13:40 +0100 Message-Id: <20230329141354.516864-35-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9B94B1C0039 X-Stat-Signature: hekjn6chh39qda8bqwc75mdsjqmjoifi X-HE-Tag: 1680099346-789323 X-HE-Meta: U2FsdGVkX1+FxWFp3Zn4l+PiY2Iw9ZYyRCTTqpxd2X5DB/kwoVgRIvfzvdgox+1rFqanlF9DVZbaJc6mYMQFYjzDYLe7oXqO/fKzsnR4kaJ6orp4HORR+QLkIYMXKB787f44zjmABImnmvifP2MsfDCXOFha/V2sazZbt4t0xy7cUDYUxz5BTW5D2aXneXJqNBCuG6wVooiDFiM3dDkdqMB1e9UiE/0Jytqfmp743rH3/S1nSf50ikmM+6XSMMTBJAd+tASmfBj7CXEXDJvpuhuEmuBdai12CwiWotMdzzA/R+IZYxuUQi6MqSinb0BiBJMl16fIBljFNQsHhKBzzcGzJp+Wjs8YFS3YvLycyfO2SgVDZTU/xFUAZlLcpEC2mQ1YrsvJG6iUnpOhU2mZ+fZR1mngvcqAjSW0X/ljB6vHbm4DDg+JMxhrsi4MdEB1i+S8LkhGfP8OmSvx6EyBw3gOFH/8Qam7KQq8uEWg8JMBJtC8ege63jgoHpqSsTRDYbttLi9k2YtTw576FhFou1FiaNoMh33O+7PpJUUFdlYRbT61feeWpywOH3H+LcWw/ncSibgw7+hCt5K1lOkOpCcw7HgnyRMsRLEPOMrXwU/uEmHdNdgJ8BO9oPYhBU6IRPY2cQ+BJtKyufS40TsCJEiJ2+YF+gv0D/pO79BZ2DfHy/rw7afACzwGBJVrOP4vPCvNQDnh9t53VXHDsUbE87t5YE5Ki6PdNLL/u+nTQg3zZHU+NdN+5d47Fi14ts4K4AGZS5mzPksrjtK/0dmpk6mdGAdh6Sd9/dTKEkA7Nt8vBm+hrkYc4tRo8GxsFird8fYcq42LHD64FbH59oeaCBPa0bqqxwEB3Jz3XjV3y9cS6qG4RVJnhx8cko7vfK5RmWd+t3knNwLMmmGoDn6ZFCsg1gE/t9c6KcNR6PV5pJo6dkjUt2ZS/pFVd3kCWMmXKMSDOpbgI8IA1Xe5Z+P pqkXzdOQ Z6BxV2FzUPCQek4QDWjEBZRd9Wk6OAGPfHM2f3ZAOLwVcdHGrwWOgKlSA9/jemOr8IZZIjX6WvfB6LKv5gsvQKHoE6dR3HRuLiSXFfrh1Njd7TPfX2IbPrJYvhewnPnArkqlCI/moflH+SlUeIuCJRGdBvPR/4+TKwS7MgjyU/4vno+PBKOy8To0JBB9M+zttD/0jI82C0pGr21tOdZFdGtnL0z5wSiZ8gbK03dZeodoj/y5Ckyfl8v0gdBy51nzU+n/HqJGYHULvnERJiUiQn5Ler1zDWGeFRzibTD8FjNBRFZmJj98+Hs0Dok2avZJi06uZ3WzTPjUFF72r+C1qM14mg+USnkew/IWmb8p/DUJeLocPDo1Z9QwZd05Yv1QubQng80WZ+Pao5E9Vh77dElPhYqApTZAYc/LFAwFr4skwp1Q4IA54b9M8og330HZNvn56rZYSqp1QDodA+IYSxRw9K8RM1aOV+2T49k72YZTtq8Bh+6lCoGngggRG9SkNGGitZ45GH4koeT6FBH0F4snYWX+9agukCq9zVbOuUPsjhlIJnPozsh0apQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Translate tcp_bpf_sendpage() calls to tcp_bpf_sendmsg(MSG_SPLICE_PAGES). Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: bpf@vger.kernel.org cc: netdev@vger.kernel.org --- net/ipv4/tcp_bpf.c | 49 +++++++++------------------------------------- 1 file changed, 9 insertions(+), 40 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 7f17134637eb..de37a4372437 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -485,49 +485,18 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) static int tcp_bpf_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct sk_msg tmp, *msg = NULL; - int err = 0, copied = 0; - struct sk_psock *psock; - bool enospc = false; - - psock = sk_psock_get(sk); - if (unlikely(!psock)) - return tcp_sendpage(sk, page, offset, size, flags); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - lock_sock(sk); - if (psock->cork) { - msg = psock->cork; - } else { - msg = &tmp; - sk_msg_init(msg); - } + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - /* Catch case where ring is full and sendpage is stalled. */ - if (unlikely(sk_msg_full(msg))) - goto out_err; - - sk_msg_page_add(msg, page, size, offset); - sk_mem_charge(sk, size); - copied = size; - if (sk_msg_full(msg)) - enospc = true; - if (psock->cork_bytes) { - if (size > psock->cork_bytes) - psock->cork_bytes = 0; - else - psock->cork_bytes -= size; - if (psock->cork_bytes && !enospc) - goto out_err; - /* All cork bytes are accounted, rerun the prog. */ - psock->eval = __SK_NONE; - psock->cork_bytes = 0; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = tcp_bpf_send_verdict(sk, psock, msg, &copied, flags); -out_err: - release_sock(sk); - sk_psock_put(sk, psock); - return copied ? copied : err; + return tcp_bpf_sendmsg(sk, &msg, size); } enum { From patchwork Wed Mar 29 14:13:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85F98C74A5B for ; Wed, 29 Mar 2023 14:15:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F5736B00A5; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 87E566B00A8; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CDA9900002; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5BB7F6B00A5 for ; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 37BF4140D06 for ; Wed, 29 Mar 2023 14:15:45 +0000 (UTC) X-FDA: 80622134250.07.5EEEA7A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 561D140005 for ; Wed, 29 Mar 2023 14:15:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QNIFJ478; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lLisHPaWAEED9gZclxyhbGIkHvgJDoUiXuyeeqOKByg=; b=5PlqZANnyDlBN57f9d7FAeTgWsM20/tgtnB/BZGYqFtb4v+zcJWz4TnXXOoQR6tOfw5R/1 FOMnYMUv9OXe1nxfuOTSnQtIsR+xm0H1eF80WSAWWB/422joabYM5czC/DUyppD1XthbOq jliFeZVLJWMW5uvkgbZSL99xZq0eEXo= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QNIFJ478; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099343; a=rsa-sha256; cv=none; b=NbpHbS+s9aWEo41N8+p5grGK51F1WQR1ERSr5U917ynP50gK+nWxIdWCe5w1VmRL6rhrYh EuTJaYx0pkLxHniNOesLlI8T/TkjeRED04ZsUnnKIZ9vRqh/Os/0HKdWPaWkj9lhmtkA19 +GPywKpfBF2T/Jn4jdJmMhpEf4YyspY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lLisHPaWAEED9gZclxyhbGIkHvgJDoUiXuyeeqOKByg=; b=QNIFJ478XtBqGOh2XSM3dQh2Vj4WYzVzTLm1WAqx6Usn+tmaU/UP/lBr5x64VHbGogdSby qjiJ4htFwy2hxWtcGfGBxlf3Xu335E9gq9QTmpeUsZuSfX01rzScnBvxYGJF3HnF3WFJZA QAJosoRTGzY8WVX0m5J6XXkbVnvx2VY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-663-69Q30cO8PE2HT8MQERo7Mg-1; Wed, 29 Mar 2023 10:15:36 -0400 X-MC-Unique: 69Q30cO8PE2HT8MQERo7Mg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 33DA1101A552; Wed, 29 Mar 2023 14:15:35 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4C5DF202701E; Wed, 29 Mar 2023 14:15:33 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 35/48] net: Use sendmsg(MSG_SPLICE_PAGES) not sendpage in skb_send_sock() Date: Wed, 29 Mar 2023 15:13:41 +0100 Message-Id: <20230329141354.516864-36-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 561D140005 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: j9u65jh86m8jexuo9zc8ii4syykkkp8b X-HE-Tag: 1680099343-965657 X-HE-Meta: U2FsdGVkX19+BAeDWq5dHzsPgQ9Wn569Vo7PPIpegCxalgPmU0imknkCBk9PKokLtn+pi5Ioyd3k226pPbbIXX19lDbJUVwxjy9aIleTrea/C1czqetJq7hqV8kCUyiT43Kjz8wLoUTFXM9te1iW6rGqi0/KeAMsJkN8b65dM0JTnUjD8RQm9pMtyl7kSKSH8H+wKr/xCJq89Xm0qmtQXeZDGI0K/GH7w0efMBZnG5gK0eQOR91Jgz+0m/ZrosGMesyVzgjMcE4OZqdBiIPFxGGPW3ygJVKoi130wbEsnZaX1F4O6Fi9hJRjI5FrlVIAXiczZFxWCzxz54VA0UbiwguvD7oxayjBMbB8+vOat1lJK+XudmO8mp2/Ajllwab2K/pdazm/FWGaQooGVJHx1hAKbd80zieCOvG5SPbHnb+UJL+C1QWsdsmUrWpBuR7SVrlW6xiBaaBAmK9iFxS3S81mdciGnLLu3pNahDqSa+qaMZBKZZbs7ACgMXup+beD8bXxCNTIMtfj98TFRcMm26kEe+8hSoobivlMAIJ+nwDDHiS7tQyXi9BztS92FegDpj3epcE+qlWVyRDgcL9DN6VQTdgX5y5AICwt2GeX8NpUzdHG3bCj88ks+lYuuF0GLZ01h1oXY1H14EyJCWfc1ixJxgwb7bGMq7wo6mJU5hDqq9EYj58YPmXdokc3nilxtApdQbRtZGwj6PMmr/1c7a0bjgQTLR1QA6tsLrnPFOx4b5vHpDdAQqCh05uLyIR/b0I/j+qTuJq8tcW/aJ52r+c4lk/4a0PhTAYci+ZqiomUiONMEQkf9eohbn9/Qu03BwAQ3V523kIfp2mBVLRLZ4tBBxYHTEx5HsghFfRhq1nJnExKBImWQ198NpPdgWJE/2C0Ldg7Z7naREXbE5eWM/49+kK5WVAgXC3e9rbd/hqzBJhdKHFn1J0VnIDRZPZxrC5Xk04u0S/K44ikG8E jwDREv7k N2vWP7HaPiBvSHPP+iNySjwrt/eagVAre2HC2xLuV8itYTSDwWRXgcOV+MQVPaxhH2Z0Z/3TP6NZWxxB1nl6WrFuvba1T0ri74W0wuGSe7+rQilm2j2mNBDK3XTBjjeVJitiw1NHCx/9LWXK89NuzRAE5VicfvpsxFz7PX3fy/B6Ruqeo9v60c3ik0cHoxiJibq2OdwqZ5XJM572N/8jNU9uzJ2GS2J+Zf1psTYnhQBJff05OIKSm8J+Qj4NZb0LI2tFKDtOlpNGz+dVqmNjACYski82awU8inmM+2zRyXryY6cCf1kUrbBT9oqy9zysjIAcuE4f5KhtAPVeFRqMmB4VJ+avmlDE2REi5sA9S4d9Z7aivR+2tjl9P7qOeeN3YH/U6eTrAVt+zNkOVIYCtXrPmzvYKM5pxH1rQZvWC59nsPRfes0cQaHftV9yQ80fW4nMs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage in skb_send_sock(). This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Note that this could perhaps be improved to fill out a bvec array with all the frags and then make a single sendmsg call, possibly sticking the header on the front also. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/core/skbuff.c | 49 ++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 0506e4cf1ed9..3693b3526d33 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -2919,32 +2919,32 @@ int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset, } EXPORT_SYMBOL_GPL(skb_splice_bits); -static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size) +static int sendmsg_locked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; + size_t size = msg_data_left(msg); if (!sock) return -EINVAL; - return kernel_sendmsg(sock, msg, vec, num, size); + + if (!sock->ops->sendmsg_locked) + return sock_no_sendmsg_locked(sk, msg, size); + + return sock->ops->sendmsg_locked(sk, msg, size); } -static int sendpage_unlocked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; if (!sock) return -EINVAL; - return kernel_sendpage(sock, page, offset, size, flags); + return sock_sendmsg(sock, msg); } -typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size); -typedef int (*sendpage_func)(struct sock *sk, struct page *page, int offset, - size_t size, int flags); +typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg); static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, - int len, sendmsg_func sendmsg, sendpage_func sendpage) + int len, sendmsg_func sendmsg) { unsigned int orig_len = len; struct sk_buff *head = skb; @@ -2964,8 +2964,9 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, memset(&msg, 0, sizeof(msg)); msg.msg_flags = MSG_DONTWAIT; - ret = INDIRECT_CALL_2(sendmsg, kernel_sendmsg_locked, - sendmsg_unlocked, sk, &msg, &kv, 1, slen); + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &kv, 1, slen); + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -2996,11 +2997,17 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, slen = min_t(size_t, len, skb_frag_size(frag) - offset); while (slen) { - ret = INDIRECT_CALL_2(sendpage, kernel_sendpage_locked, - sendpage_unlocked, sk, - skb_frag_page(frag), - skb_frag_off(frag) + offset, - slen, MSG_DONTWAIT); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT, + }; + + bvec_set_page(&bvec, skb_frag_page(frag), slen, + skb_frag_off(frag) + offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, slen); + + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -3037,16 +3044,14 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int skb_send_sock_locked(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, kernel_sendmsg_locked, - kernel_sendpage_locked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_locked); } EXPORT_SYMBOL_GPL(skb_send_sock_locked); /* Send skb data on a socket. Socket must be unlocked. */ int skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked, - sendpage_unlocked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked); } /** From patchwork Wed Mar 29 14:13:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0A6BC6FD18 for ; Wed, 29 Mar 2023 14:15:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 793F56B00A6; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 744936B00A5; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52ABC6B00A7; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3FAAA6B00A5 for ; Wed, 29 Mar 2023 10:15:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0EB75140CBA for ; Wed, 29 Mar 2023 14:15:45 +0000 (UTC) X-FDA: 80622134250.12.06792CE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id F302120008 for ; Wed, 29 Mar 2023 14:15:42 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gwk1YHJu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=taClZ7tV+6NGiC2MYzyqiyjdYhpNZJPUqY1OQ7XAIPM=; b=HqpbRns63XnwRkYA4+FNIdIYLijiNlYz+HMQGWCZQz8I+UFaeQgkhWm8GB6BzA8Q/aSg92 0pOMs4KAEyNBu+upuyb34nfeuVnrbO112XKPl6tQpU8KczKIt541+R643LVgNrMcEgd3xa JLyw0LOBnG8dxFJ11B5EfjfsDSZgiXY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gwk1YHJu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099343; a=rsa-sha256; cv=none; b=npsAE9Bt6ahZXQePp0FV1B7lKb2Kg6ZGLbJQEa1ayW1OXZVPR2iRYB6ZMalUeOQtDdvyI/ OHX+vqQIJjfnOUx9eNjyDXkXKx+SlJbbPsW7Wwjpv9ygsFMpmfCMU+V2DLwjoLMUQOCGsB S8LFl8xGKEnvNDRNf1ouwKEqBi8yDeM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=taClZ7tV+6NGiC2MYzyqiyjdYhpNZJPUqY1OQ7XAIPM=; b=gwk1YHJutJ+3yKXMP8HpHYdp1FdHl0aMkbtzXS4Gg4RWAYACA0hWRj9sa2OG/+65YvNLiZ qCCs/mgQuIUM0kLMTQY8s8vOARy2V4q4FShkFOLd8tWA5QkoFlGggwVIpWMJew5Qg9X96S Xkos6sXrKySvemRtjUiJ0BjBL6I3p94= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-572-EfC1k2O2OHKPPpyIGoOAVw-1; Wed, 29 Mar 2023 10:15:39 -0400 X-MC-Unique: EfC1k2O2OHKPPpyIGoOAVw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DCF561C0758D; Wed, 29 Mar 2023 14:15:37 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id C9F6B4020C83; Wed, 29 Mar 2023 14:15:35 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH v2 36/48] algif: Remove hash_sendpage*() Date: Wed, 29 Mar 2023 15:13:42 +0100 Message-Id: <20230329141354.516864-37-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: F302120008 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: q9m6df9pamph6eb9oyirfeo77a8j8h9q X-HE-Tag: 1680099342-288999 X-HE-Meta: U2FsdGVkX1+GocOOc8jpbFggHwB41evZfpvMdFmGxvF01eRt+3OuYxK+il99eiO4iQ95D5oJfkkf2FzrBl6MwZ5lyU2AZCNTMpBWqI08+2eYNFGVP2XYb/bYJZk6eCDMQc3PB7msBRBsglEbg8dXAIufPBvcN+f6LFV+tit0YH7IOA3SJvi23Smrtu8ZTrO8jXsmoR5IS62gYeA5oadJup+4UU9qwC74msqa4XxGh7Slg9Lre3N7tqW9+0y0pM2DKAKSk9mq4LxjKwC8qvoj1D64fe28aQTBKsaO//WCrLYdGTh9HTy94ljR1OhZZ/VpnW0Vpx9wLJBsAsSxqFemaNc3HTu1t7hGq3mCPrRxIp4Clt6N6EqN5dlPhzWZXPlnA4dqEgXaUBBBaA5yPloBnzWNuhphshdfm2YmsllWJUCGOmVucouDjTZfT13ap08PoDEdd9fPhQLFaEdSFmGf39yL+xAf1GH5g14xMwSvGam5pWe8/TyTmeRSYH2zpXFkk2AT7nN5E/sWx9eKz57f1DvwAO8lL+t+scAZ2+3FP1FivEG3c118zLQikq3kHfMbfwAQoVPiMbhxfs+jyxvBGuazUJJXDt/NExYocR16aQs8gTCx1MWA1RY5vRwzj8HZWvdEEp6MFlT7LwsfU28xqQy0DddQYgrNDu3yO8v7aMUNFTiM9zFMWl5FKD/gNqgB9NRO/1rrC2st7v8U5FF0t3TgEmMdGYRaneQpIBzIeu/tkJFIWucMGfYZFT6xfPYBi3EJzx8m4jn/A1z3NUX5a7dXpR2W1OpWEUqOqR7gmoRZuZx1dxXy9kovSVkWcCyppEjN2U7uRMthQYXKRAzZkRibnAFn2MzeYNeJ22wgh8f5W2PWwOayzMifHuwoHlz+2/dgmlDUK0BSLR9dqdUHaPotfd9Q+urnSUDgWOVNG93dizA/Z0KERyQp5LL/e3bVxY5J56GqOBzz2q5FAaR an7zPzZS lNtdUzHyrFZoRdBrRnLWfNrUT8S6/XZ/aNF0Skr6nOOZqosOJYMe3tLNp1yWeIwXbQplUEEhjkbr36//cFoxCqPZfVh+XnlwrfW0osRIaUwQa0l6OJUs8ePsf40nHCL/ltEkWlMOl2/teen1ASLbEZgelmgjWorCEqYi7wICiht/6NzLb257C/76rr8VPxvKrKpz9WuaoWflvEPibMlsZehd82zcRpMlEr2AoPmg2nrIeRHu17r/wzPUozUYIL5gMYViGw6r9sDdWSL0wgxXmXtu0EqAyx7JdrCCKy+U4RzAlzrcc1WQVCgdSYb9e0BQTSyokvcPDmVUPTQwS8GfJUyC6dvmnMD/ZKUyLnk/1vHYsCfX/EMIrE0s23sK8uit9ikMXTATeykFv7b4mbVaC8qyjpbwIMQ2NMTtHU+78xX4+in02sXawWHMPnX6tSksCw9dyBu9FuA258WIWfGstnuBMYKIl4ozDoovW+gtsvbr3vsbf55EpjH3jxw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove hash_sendpage*().. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/algif_hash.c | 66 --------------------------------------------- 1 file changed, 66 deletions(-) diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index b89c2c50cecc..dc6c45637b2d 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -162,58 +162,6 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, goto unlock; } -static ssize_t hash_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - struct sock *sk = sock->sk; - struct alg_sock *ask = alg_sk(sk); - struct hash_ctx *ctx = ask->private; - int err; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - lock_sock(sk); - sg_init_table(ctx->sgl.sgl, 1); - sg_set_page(ctx->sgl.sgl, page, size, offset); - - if (!(flags & MSG_MORE)) { - err = hash_alloc_result(sk, ctx); - if (err) - goto unlock; - } else if (!ctx->more) - hash_free_result(sk, ctx); - - ahash_request_set_crypt(&ctx->req, ctx->sgl.sgl, ctx->result, size); - - if (!(flags & MSG_MORE)) { - if (ctx->more) - err = crypto_ahash_finup(&ctx->req); - else - err = crypto_ahash_digest(&ctx->req); - } else { - if (!ctx->more) { - err = crypto_ahash_init(&ctx->req); - err = crypto_wait_req(err, &ctx->wait); - if (err) - goto unlock; - } - - err = crypto_ahash_update(&ctx->req); - } - - err = crypto_wait_req(err, &ctx->wait); - if (err) - goto unlock; - - ctx->more = flags & MSG_MORE; - -unlock: - release_sock(sk); - - return err ?: size; -} - static int hash_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, int flags) { @@ -318,7 +266,6 @@ static struct proto_ops algif_hash_ops = { .release = af_alg_release, .sendmsg = hash_sendmsg, - .sendpage = hash_sendpage, .recvmsg = hash_recvmsg, .accept = hash_accept, }; @@ -370,18 +317,6 @@ static int hash_sendmsg_nokey(struct socket *sock, struct msghdr *msg, return hash_sendmsg(sock, msg, size); } -static ssize_t hash_sendpage_nokey(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - int err; - - err = hash_check_key(sock); - if (err) - return err; - - return hash_sendpage(sock, page, offset, size, flags); -} - static int hash_recvmsg_nokey(struct socket *sock, struct msghdr *msg, size_t ignored, int flags) { @@ -420,7 +355,6 @@ static struct proto_ops algif_hash_ops_nokey = { .release = af_alg_release, .sendmsg = hash_sendmsg_nokey, - .sendpage = hash_sendpage_nokey, .recvmsg = hash_recvmsg_nokey, .accept = hash_accept_nokey, }; From patchwork Wed Mar 29 14:13:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5AC2C761AF for ; Wed, 29 Mar 2023 14:15:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D64F36B00A7; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9B26900002; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B145D6B00AA; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 99EBE6B00A8 for ; Wed, 29 Mar 2023 10:15:47 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6A965AC301 for ; Wed, 29 Mar 2023 14:15:47 +0000 (UTC) X-FDA: 80622134334.05.E1C2B28 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 5B7FA18001B for ; Wed, 29 Mar 2023 14:15:45 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C2BnMd5z; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=14pAi6JHwU3C53h/ZQrPmmNB396J3kzB8Cmz3sg/vjY=; b=FO5pw6EqJkNexuSwzcQB9IPAg9GHwkfFf8+3s2r98FFP75Bz7mYJz3+IX/AfhznC6uKjQ8 bOD6gmto55AT5tQiIc4axx07xn9yuywJGESRL6psoZ8s0gCO7jenmmBjfeHWXnlIwyueTE XvoZSnyMwIqHbpVmrtmRYf1AoQOpQD8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C2BnMd5z; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099345; a=rsa-sha256; cv=none; b=7B1oltq9hho04ejCJD3ryz7HXSOhtRmOd7cE+ge3uphalxJf0fprRjRBBbj1PzyuJMYmhc jmEiQdGU0c6pgdqWmwjptUOHQN+yJs9aScHoIqbkmLjYiL2F89UWjBrzYfEYeCdwUAFgll ys8Rtx0PR0oSevEFeEze2OPtiWZo0M4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=14pAi6JHwU3C53h/ZQrPmmNB396J3kzB8Cmz3sg/vjY=; b=C2BnMd5zALhoFoNijhylO+9/hIJjvLRUVddgyVnuGKITVSSg/DO7BDuCNqwkpCgOHQZjIS vdjKvZt1R3kkyR4H8zqrbLbORBluF92XPAeXRNM0vg8ND5OJLSkkIWuMfHgdCopw+gegPK EYLd5WIcaTarQ1ppRR/vKDU2CFNLhbc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-618-VHJgX1wyNguGyTVDaKNkkw-1; Wed, 29 Mar 2023 10:15:42 -0400 X-MC-Unique: VHJgX1wyNguGyTVDaKNkkw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C9AAA802D1A; Wed, 29 Mar 2023 14:15:40 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9E5D714171BB; Wed, 29 Mar 2023 14:15:38 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , Xiubo Li , ceph-devel@vger.kernel.org Subject: [RFC PATCH v2 37/48] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Wed, 29 Mar 2023 15:13:43 +0100 Message-Id: <20230329141354.516864-38-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5B7FA18001B X-Stat-Signature: r5k44e8kikm156ftyttq8ocgrum3bncq X-HE-Tag: 1680099345-537242 X-HE-Meta: U2FsdGVkX189lIPhUq6YsA47gLNpVrNogj02GPyGzitfPB4rdjUK5C2CYBRHeDZfvzgNH5N4HxwxjlVOIiyh7AfsgI58ptkF02BFgYeNy1OVmlbFgPzMTLDcrn0N2i5jGgJStbiBVDWueI+iqTR/+mP4xTv/FQK5t/DoPzF0TwS1dTfyjWIPpD3f8JeJk6Xl37vIoQvvHEM9cUXWwyOkRVL3qhkf+dsHqTXnonDMHi/cxd8OjSWLcIS7tGhsZivVYtvMzYCwD2m8r5uLUGo8Zoys0wVlVMgn6ypM9gM8HrdpZ35r/i36DliOaxKip/Z0GOmNsIXsQ7YTXbufKoGyxnsjjQR/dp+WMLHbCAusg41SxRi5hJ4ylyhqK8NU9xJnwb9K3gs7qUK4p+PXCoKemSRayPasuUyMVyT7EkL0yYnVRPBtGApS1pNV8+9uai9kmixkVsFbKDd2QIzov12Vx3Wt01rVm7loYUT1gX0oKDykf4HT/XQ42q+The/OfneiEK/ad1PcOVA6IiQg4+nPK26uDaz98PUJ2uS9WejscBkFSfBmBApK/cNIXerBHXu6RSQbOk0/zjC6AbmScYqGJqg4pUJiBAZqccPm8AIDZNSjXdjJlHe30QemCM58tl4F9NN3YJ8vlQ6/Z1wjSLTO8bPpGWklGoccgPwFD/FGRFmhYlHDuIVEheA8QYQbcN+kYc1rP39ZzU15/49+cD2Ip8Ab+suZije5LRmpXjx4t/nYglrDQETLMq+bGknrLQ8xPhlEmv/nAixu9YLKv5skQQPPjmV3SkJ7rI2auaOS4KnKCEvtmuKaz6STwTr/pCs+XpjnWrOiXOVpsbQL5XGfCEdMLFXVS1jt5mdgZH3ii30ejAP7suD6mx+1awo5cgP479OHlia0LEtzfkG5wF/PyYlYMzZMWPcr7EHigd6LsWdqOFaNDIbHqaJ7ZpnQlzdpUToE6IK3tVnpr8jjM+t 1pCFb8t3 dmGW675ea6nKDQOzln+CjDws7HJP2SAYDbRNVjYz8p2aWFp3QgfP1CeAf3W4FAQlXl4+rQiAdDYOkn+YW99+pqssRx8W9GgAkXUzraT7m8Ih5RzSMWYiG1/4HgEtQ7l2bf48NMTW2Z85MNWuPbFhheletmtvm6xxMHw3jpaY6IENuJAIUYG96MF97EdAF5p1MA7Gb7VCkLLDkOX4aWSFg//1r8hQ+6fBCvSYFDIdhE7u3qRtVJSuS/DaUWPKDLaLcfiYvMRp3SeyNTsuIgZtuVrG8Gfu2JyF97ALOOLhCMJudQtOZJAGNrfDqadFU/TH+w47apEnN/Ni3Jnl6JPoDeoYd6SSBc/4jccim0pMaC5dWfq97OONIj/vUB/wxQZnsKD5azuO+guwcpNPQXp0cQMi4Z/eGfydcMzIa06AFHtAYsOgP0dG3dgH34WI87u3+mqHm4wlpqTQLHHM+d52S7wZRmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v2.c | 89 +++++++++-------------------------------- 1 file changed, 18 insertions(+), 71 deletions(-) diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c index 301a991dc6a6..1637a0c21126 100644 --- a/net/ceph/messenger_v2.c +++ b/net/ceph/messenger_v2.c @@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con) return ret; } -static int do_sendmsg(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - int ret; - - msg.msg_iter = *it; - while (iov_iter_count(it)) { - ret = sock_sendmsg(sock, &msg); - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - WARN_ON(msg_data_left(&msg)); - return 1; -} - -static int do_try_sendpage(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - struct bio_vec bv; - int ret; - - if (WARN_ON(!iov_iter_is_bvec(it))) - return -EINVAL; - - while (iov_iter_count(it)) { - /* iov_iter_iovec() for ITER_BVEC */ - bvec_set_page(&bv, it->bvec->bv_page, - min(iov_iter_count(it), - it->bvec->bv_len - it->iov_offset), - it->bvec->bv_offset + it->iov_offset); - - /* - * sendpage cannot properly handle pages with - * page_count == 0, we need to fall back to sendmsg if - * that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag - * which triggers one of hardened usercopy checks. - */ - if (sendpage_ok(bv.bv_page)) { - ret = sock->ops->sendpage(sock, bv.bv_page, - bv.bv_offset, bv.bv_len, - CEPH_MSG_FLAGS); - } else { - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, bv.bv_len); - ret = sock_sendmsg(sock, &msg); - } - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - return 1; -} - /* * Write as much as possible. The socket is expected to be corked, * so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. * * Return: - * 1 - done, nothing (else) to write + * >0 - done, nothing (else) to write * 0 - socket is full, need to wait * <0 - error */ static int ceph_tcp_send(struct ceph_connection *con) { + struct msghdr msg = { + .msg_iter = con->v2.out_iter, + .msg_flags = CEPH_MSG_FLAGS, + }; int ret; + if (WARN_ON(!iov_iter_is_bvec(&con->v2.out_iter))) + return -EINVAL; + + if (con->v2.out_iter_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + dout("%s con %p have %zu try_sendpage %d\n", __func__, con, iov_iter_count(&con->v2.out_iter), con->v2.out_iter_sendpage); - if (con->v2.out_iter_sendpage) - ret = do_try_sendpage(con->sock, &con->v2.out_iter); - else - ret = do_sendmsg(con->sock, &con->v2.out_iter); + + ret = sock_sendmsg(con->sock, &msg); + if (ret > 0) + iov_iter_advance(&con->v2.out_iter, ret); + else if (ret == -EAGAIN) + ret = 0; + dout("%s con %p ret %d left %zu\n", __func__, con, ret, iov_iter_count(&con->v2.out_iter)); return ret; From patchwork Wed Mar 29 14:13:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFC3CC6FD18 for ; Wed, 29 Mar 2023 14:15:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EFFD900004; Wed, 29 Mar 2023 10:15:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A0CB900002; Wed, 29 Mar 2023 10:15:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE4ED900004; Wed, 29 Mar 2023 10:15:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DBDFE900002 for ; Wed, 29 Mar 2023 10:15:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9D0751C60C3 for ; Wed, 29 Mar 2023 14:15:52 +0000 (UTC) X-FDA: 80622134544.22.A0262AB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 7A319C002A for ; Wed, 29 Mar 2023 14:15:49 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V86LIBwS; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099349; a=rsa-sha256; cv=none; b=qHgnccjhZ5LDwob6nvQnlxrSp0HKl46irOB/GRgwkTsLX79ayORcy2TAjGQYMEmWHKjO94 xNKkQjejHuA2xdJpY12JkuNDYMJMlpoLRLdnskqXWoYGbgiY38gK+n4lhOjSEJPd0R9eQ+ LKYA7gGwPYrLU5IdGAbIaZy6WmQgYBI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V86LIBwS; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=U+eygar7rqS4PdLVXXM5nMIIcTG+n39OJc0zt46Qdt9uDbTl3cK81X4Il9+Bp34aV3vBEa rlEmLfhr5MQZGrR/aRVHVKvaVqEqtnQ/oAWeQRtI1K+RIkhwOPiHrOrH0Ft5dfmxJCzvsn uj7FUsIbxVl16/brWhT4oUopDr624zM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=V86LIBwSGpkib+VJnc4A7sLtmU4N1fe3eRmG35IlpueGTtNSkHJosMJc/Pm3cCiaE6gf2S L2BFbpKsaWUHY2LrgQLaMfcHec0LHbQZgpM09+Ew77tZsxynpTXMPetnFjmGARiFeh7cU5 8zirau/ZGS/QfRE7EctJaMw78GGxw3Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-494-exPjG2AiPg65VJXx__yQjQ-1; Wed, 29 Mar 2023 10:15:45 -0400 X-MC-Unique: exPjG2AiPg65VJXx__yQjQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 99A6F185A7A4; Wed, 29 Mar 2023 14:15:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6BD31202701E; Wed, 29 Mar 2023 14:15:41 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [RFC PATCH v2 38/48] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Wed, 29 Mar 2023 15:13:44 +0100 Message-Id: <20230329141354.516864-39-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Queue-Id: 7A319C002A X-Rspamd-Server: rspam01 X-Stat-Signature: i9dzoj839rkp7s9tpmmp97o7j84d3t1u X-HE-Tag: 1680099349-590121 X-HE-Meta: U2FsdGVkX18tk5BChFRTw9Rdb0UfVC1qIqyQmV7385eXff8oMwRUtMUv8Q/ncv/I6UEBAI/L5+T0B5Sy6E6eb3zDUfy+8aPFdZma1S5xyL/uulukDa17qMx4eA742mIlzKFC4THQsM5jWv+DjdFqr5mWfykfaDp7wOss3XyBSXjKrmkhKpOBRzjnNwRv4afT8zNsaafGbYaEAJH8f1Ylzy9tjz/mOXE+DDwQ9Nj/44SBa4/9Hn8Bum9fCreHqw5SHEySDhYeGkfLu66yWZRnuGJZTPrHAM89A8UXOtzOnrsteGCIDAmw8mmVQMYdaytfFMOmnTXHE5BrdvuFmHYzIenp2/EBGOvCl6lq1jOzAl5LTj8Kq4F2DDPgt6b04R35ouK+OvS9JlgF+KMVKn/yjyyenpgroTu9Izj2ttcVzM+ICEfXFX+YFXuTpdrzdz8iSX/tdd+ISyomuU90boJIGFdUN3wE4C99Gk2ED7Vf+qjJPirZ08EtGNoo/Ugl3D4Liub5U6FyePfCKfSWa9R99piLYaTB6oMNDWZLMG1ZjL0qaN8wRIcjxu3oAlKM4V3RCaOKic91GEPBQGth7jSji38hzEG4OhR78pzOE4ysp+NTZZpg6U140SwoRAC3Z6GMidATY9kB/yzSG8ONNzFBwR27lE3oJNyyxDcHvbsZ6/9gFHoCS2TsIjS62KY0xxt8BZwvuVZPeK3A9wLp3efNG4p+6c7LoAgme70VfpWK69t6ymV1N47GjTK6azc/KTHYynQOXRwk5bEen8uO+0aWBmd4chxt8ANelEGsfGyL/tWnoj5rT4wOusv3sy15Ug7h7Li97nxZH3Ixm+Ln54RdpjFOQpi0f8DdfRE/WFAZXiStizZ5qZ5Mcg/gkuSROnFLzFSLC5oEk695D3zHCTJgdQvI9zyXzMfAw25h+J3MtyUV965deN4NSTDol6CzVP0o3fQwJX7FQ4HR71vY7Nf EHjPQrfC 0g1L69m4+FFyGGWs9SkiOQSErrgVsZho9hp7M8JaDw30U5v7lThCOQFO5HT1ipdNAT6NlsAu1RSdn9/Za2EdflBlojCE1IvwUDoloGHWjFIitWGBZthEDeRfj8fqm3oTztH5G5WO/T2nVpdxE7o9mRGH9sJLYBlsdiAAhEdLhNzllLIvo0TvY1nLslHaLOs4VJtcg2I5h9MpcBrT+wwhmKnnhpvbil6QvSjBC2HUoUb9TopUnOavykY7Kw+3EGo/Mey9dYfDMZEVi+qlky+49e8ydw6DbWS9H6DFe86PnAFKacqei7a+ELWedxxKgZ72oOnKzYyzUYVT7FbvU4cxxk4GeIKftId5PHD7Hgg59utfBTB70ptNXoMtPlBgfrOvZkrOBbZVPXbRGIYXWeUAMojJhk6Y84x1ouq+geM0YswH3oWeMMij2XKDQusaHEJPAmwqn6hHTqh4CJ5eeAxs/lkC+cg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 86 +++++++++++++++++++++------------------------- 1 file changed, 40 insertions(+), 46 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..660d9f203d99 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -52,29 +52,24 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; + bool free_hdr = false; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -99,43 +94,37 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, if (hdr_off < sizeof(struct rds_header)) { /* see rds_tcp_write_space() */ + void *p; + set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); - if (ret < 0) - goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) + ret = -ENOMEM; + p = page_frag_memdup(NULL, + (void *)&rm->m_inc.i_hdr + hdr_off, + sizeof(rm->m_inc.i_hdr) - hdr_off, + GFP_KERNEL, ULONG_MAX); + if (!p) goto out; + bvec_set_virt(&bvec[ix], p, sizeof(rm->m_inc.i_hdr) - hdr_off); + free_hdr = true; + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +147,11 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + if (bvec) { + if (free_hdr) + put_page(bvec[0].bv_page); + kfree(bvec); + } return done; } From patchwork Wed Mar 29 14:13:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79902C761AF for ; Wed, 29 Mar 2023 14:16:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 968BF280003; Wed, 29 Mar 2023 10:16:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 919AC280001; Wed, 29 Mar 2023 10:16:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76B20280003; Wed, 29 Mar 2023 10:16:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5738C280001 for ; Wed, 29 Mar 2023 10:16:04 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F35A4140D15 for ; Wed, 29 Mar 2023 14:16:03 +0000 (UTC) X-FDA: 80622135048.16.2EC0AD4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 1E3FE40028 for ; Wed, 29 Mar 2023 14:16:01 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EsEQu02c; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6d/Oxl1Cdu8ybDL5tOgPD1gv3cW2fyMj4HBClKMadWo=; b=APGYS6jH/zNJeoeIRi+QIBxAGZhvHa3ZEWy9fmVJjhOUge+2mNViz3W671PklBxg9xCCB8 ht1QVSF3gNAXG06RtZl9oQKPGNoVjYxFhqPK+VNBv5YUKt8KONxU9ejvFo886aNL6BUGa6 VapoEFuUMIHgB+WUILleRn9vLjJp2Vk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EsEQu02c; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099362; a=rsa-sha256; cv=none; b=sOMllwSHbO89KUbqQiv3i4z4eSSttGDdGpdqzmREZ3WI5xw/xGUGOAvVup2Y2PHPnW4nl4 QpzQKO/6PnVPrX6IEreY3AFxe+bxGIOPWZJxYxQC2ZhMzzALsmHUZeiAjo0xxgaLx70ait VFITwI0J2V9FoaKIuPwfF3HBTSnsKIg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6d/Oxl1Cdu8ybDL5tOgPD1gv3cW2fyMj4HBClKMadWo=; b=EsEQu02cerYn/z1IvneKcuc61mHflIEjirnRZ1OnNPxy2hCNdiEx/tz6hfHE0de6PIe03z sHIEjuXgb25jXrOT+6a+4nZNvJYw57Osq34xoxUgqG/fZ+QcFdJDhlCkb3JmcYIU/5U+Pq w1TgJExeE1SpykWWmuEikvlocO0eVEI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-619-i1SRZaD-MumdNlQEtFGyZA-1; Wed, 29 Mar 2023 10:15:49 -0400 X-MC-Unique: i1SRZaD-MumdNlQEtFGyZA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D152F299E75D; Wed, 29 Mar 2023 14:15:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6BAE12166B33; Wed, 29 Mar 2023 14:15:44 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christine Caulfield , David Teigland , cluster-devel@redhat.com Subject: [RFC PATCH v2 39/48] dlm: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Wed, 29 Mar 2023 15:13:45 +0100 Message-Id: <20230329141354.516864-40-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Stat-Signature: ex5p87n561kmazay3ku8493d1x4k8g4a X-Rspam-User: X-Rspamd-Queue-Id: 1E3FE40028 X-Rspamd-Server: rspam06 X-HE-Tag: 1680099361-819219 X-HE-Meta: U2FsdGVkX18PoCBHm8jyIdFaH4GORrNe363C4ed676VdbJs2rbGTPeR38vDta0U8fs9UfgcndlQ1hkduOHtbGjAL/Fa+6YfUv2iRcUNsQGgvNhadIULbOnClU89T/5TuiXgaEO4JhweyvsNuPVPR+e813LLBGPeOxivX6XBwbpunbOdnqMxesVN8stp4TcraiTtzdfzpd18cIt+pciXz/a4YQoR36RmhdSaCr1TBizSryEbCeen+WBYb3e3sVmPJAr4uvyWJwIcuXMRPGI3u8ZnOUJXkEK9/l+ZFbg/m6KmsW15eC72ziPvLXqvMne0NXMIOZePjM601B0DKdAKMrAPN0JZ9LDetGIxmkY8X94d5zhwx6YPXdhBzPjM8ltbnny152WGm/qYWiHVgqdTU2QqGU2L3SdU0yHiy9LeHA5wI/X9++yvPuYpr/TmJRN82fhpTpmno6mfG3LdOSRQttRLqQJKZsrlbJF/lelGcBcKZpXcxMWyW6xXhwvwaVez0X+jv+k06S9ktVM4ScInQNnJAtqWtID36uCV8Nf8wSxPhgA6Sr6oIdK9DaaPhO4pM6vkvLOlav24dcOYhvkdKPwGHY8uRLkfqKt/UajVXHMcwaqqPqsqjDZsLMI2kGfVWUNcnvfY/HCxHJOKzQ8AsQ4Zu9A3ylT5/mftyA8BSTtxjqP1nzJTZmddbZ6qmaoRdFR2yLZYbmdL80DAC7cnUc0Yj+7QsAab0zW8FgrXUP80QYgxJuPIlFwkU0v5xw5sU4J5D79sL1qaZ0I2XCfKBRdM484UDWzwgseQ9JKQcbb1zkwvCymUGvgVEFF5dnCnsJLBCdQp1RkNkKn8tUCrg2E4TcfArqebsvu9vpKL9NCUqoiHQkl4+QFftDnK0iwIYaN4duHP+HSASbtrfZgAXkQCXbZt1uehX+O3joRi9MutEodS+IfabrdnFQ41O5MoY6VowvnhyAxLrqePc2rP hwall4/7 TXhUjO9utfZ4wEzDIAbNRTszjkvwRSpGefIdEGZFibeoGCozfDHbj6iKVqzyWou+t9RSxQ1MHfi5Gh8/Hj+vjFUuumuqaAXLOcNsUyy0OGf5btwxYxOVZ5txv0NEM88u8SXLyigbPdcdUbxEQNUcVbsmIV6iMtt0QYBwIq2dU8OLvY7M/UWiibql+l2a9ZEPJZERrjczerR2pN7aA3jUXVwHMfpXsIJPiz8QvjtHzdal1KpTXi+c68jgyg0LI/6GiDLlXCgHAwkFNUYqxdWqo/VTlsRpv/hEICD6vVmlTyli4RPFBY6s6rFtVFnRNOZW+zgRJymhKQ5vvIc11jQ2l75ncA/9JffvlHyAuHSVR36LBgf5qkErm2eDhd4uv13DzXjGyxVcvyTW/YGnv09gzscHRFTvhtib2j116UqdmpgR4/kUMC+lhYIf9tZ4yF5QaF/tG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down a layer using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather using sendpage. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Christine Caulfield cc: David Teigland cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: cluster-devel@redhat.com cc: netdev@vger.kernel.org --- fs/dlm/lowcomms.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index a9b14f81d655..9c0c691b6106 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -1394,8 +1394,11 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg) /* Send a message */ static int send_to_sock(struct connection *con) { - const int msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL; struct writequeue_entry *e; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; int len, offset, ret; spin_lock_bh(&con->writequeue_lock); @@ -1411,8 +1414,9 @@ static int send_to_sock(struct connection *con) WARN_ON_ONCE(len == 0 && e->users == 0); spin_unlock_bh(&con->writequeue_lock); - ret = kernel_sendpage(con->sock, e->page, offset, len, - msg_flags); + bvec_set_page(&bvec, e->page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(con->sock, &msg); trace_dlm_send(con->nodeid, ret); if (ret == -EAGAIN || ret == 0) { lock_sock(con->sock->sk); From patchwork Wed Mar 29 14:13:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ECCAC74A5B for ; Wed, 29 Mar 2023 14:16:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEE3E280002; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9EDA280001; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F193280002; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7D800280001 for ; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4263980D64 for ; Wed, 29 Mar 2023 14:16:03 +0000 (UTC) X-FDA: 80622135006.03.95FCFC8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 11049140025 for ; Wed, 29 Mar 2023 14:16:00 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LzZhK2yW; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=soF5gavmzneSvIIvHwhPEmiQq18BggR/9B/uQZFrN8w=; b=uSrCVy1/y86QAhYXMKO9aQYdMHIFKncdrKiqV0TOBG+dekWTpSfCQsPIE+AzMLo/+UHyPw SuA6lopRJQWLfpK7RMyisdLeE4cjJhapgTF2dfqxKTJeUIIpcMebraz/TuGdUu261fjN5H +Ei9w/d6Owyd0fRQdgsAaZceqKnIiyA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LzZhK2yW; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099361; a=rsa-sha256; cv=none; b=Yy3lWFMTaQu3wSyDype7E7dDmBYAsnYnAOt46Vq7oSM9e1FXGaEIoF24o3wQm/mu+guaQu zdj1TNP6IQM96pY5ig7YW2wZX1R+pRsXxL+SkJy4MxBauuljTEo3NNMPb47OrmexoyiGCz Be3YBmGjGIGC0PYG/YGbBMqX7ssagyc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=soF5gavmzneSvIIvHwhPEmiQq18BggR/9B/uQZFrN8w=; b=LzZhK2yWfs3lJh32dj1p8CrZrcpoQnBtF9CzxGfvS2BUFOOYWaKraZcbqH7FA9BurO1ED1 tggVJyEJhAPhKCcM5rhn5H37Xl8LR4X/4sobQMsvEOpwwzMSoikd9H7EHnXj2622vV26jY OlzF+VRpDgIgxPvYWP/v2uluR9xZhBA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-56-ruOkGzEjPIqn8iT3b-ztNw-1; Wed, 29 Mar 2023 10:15:54 -0400 X-MC-Unique: ruOkGzEjPIqn8iT3b-ztNw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 99B85855300; Wed, 29 Mar 2023 14:15:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6CE9C2027040; Wed, 29 Mar 2023 14:15:47 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org Subject: [RFC PATCH v2 40/48] sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Wed, 29 Mar 2023 15:13:46 +0100 Message-Id: <20230329141354.516864-41-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 11049140025 X-Rspam-User: X-Stat-Signature: a4cf1su3afyfgd5m3ndd36jqwgxiqj1a X-HE-Tag: 1680099360-120205 X-HE-Meta: U2FsdGVkX181oTBmekla6Spf+6gJ2wPkElrCEBuwZLwIOfNcTQuP1QbvlUbVcunCkFkDbe0uw579KhLxHYddcJse8lRULeBJNC5CC00VOMojcJ09JdLBmA7bbSpRntgC0QLHgYDSu/582Kuvs29a+xFBRn73w0yIVlyYqBQO7X2WdpkUj4WcAUyDm0XIr7y8G2OwIGr40XKQ1uI+yWWMx5S5X7270pbR0MX+rbNwkW/vRTDJfs8cXMJkr1L9Z5S5SOcD/xtESXlsR+am90PrQgYuqLbiBWxfYTVc9jtMBQbp3IpKdH7LrJcnT64UwEZGg4bzm4lCf37uNQ4OOkah5C54OkL6X7Nvy1lEAi1ze/vqeZrh1WfG5ITmqMKgEg8tcK7rDjVIbLIbizhCu3OR1LLcpnIWsT7qFNgR3eEoENIqPqsDtzrbkcPJluAuJB/NaoMcE9jbNY2UaA45CWxnrd8EkKLXMgIxxt2btdi7R/OUmNGkyrPzPPM6NVOQzxlAlT6XjYzNTH+EZacnWwEKnbieRtLYs5Cg8F/DdcjY4GmTGdT+BOZWWw/4oT+lKJ4Q9cBkV7z6883wzISfaI+INJAlm5FLij76s6tznmSQDPJfMmKqj8DPh51sPpz+qZytq7Jhg5//GUBCgl6WTN8mu03sq/7Me3V2f456wJPk8/cWx09eAnlxvgsNZsmZdcA1cd2NzJusTw9tiUQx52E68itb0S3PxLZn0FrccU/F2bfjCrpdmh6ZMWHCU6AnC5Grt4/n2rRxVBLgAt9e6kPUc5S5EZm8ZiVRbsC2uyubvoPd5+nsvQfy/6KfuR3R9jJaII8xRq3eYuZ3WtvP+F3syKwi/9snvNH6zz0vMU2HnKECPWKR1mEacw0y4mpT8Cu2vCTREzaMDup9/3ffLGhs+7mWSq3geKh5lZKvv92FIfxrp0u4nf+w/YcER61KSKeUoyrnadbTCfeCkgmZalC Aq+PziCr Znzg3ZEgOVt6prsoK4VOw/ojQxTbURuNMJFah7d6e5bPY/ekyhJhNkzscZEW6/JVNd2Ygpm3ddZpqdcu+CIqLDUjF2FhFIT/gsGTKCOi1tuOVCpZR6Uyspm6oCn2WDc8N0Y+SEiO/vkIpG29k9Y7UJg7S3NYEuA90Vs/T8dHNykzNMtZDKRQB5AVpa3aeuvHfbqdWkTZdnEgHU573OJ5CtBtHhk2AvR4KLSy4UHfVoNHWC9OFuLVLKyqXVIIP6J9zMhMwh9rse93FSgrL2OPpuFDXOT0gpJfe6FMqQQWO36SzAQ3N5UAdJ1DfwrcsS6m6hid4IB+OzEjQcHDusBlTDRnzCNhS3YURSwzZIKkaszx0qw17sX0IS/zz4K1lXj8GfVL5bAkTV8+ZP7zDRV+4dlaK3lMB7+r9oGd6sp8zOUjSWxGKbA4X3h5X3aNOuN85OUMHtSTBzgAQpanIM2vjLGOujoPwTxPpqq2GClnJCM9r+KMqMM24Z4rTpCmeHwG9PX24p+LJc6KgsISzYnfhOTPhTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer are copied into page fragments so that they can be freed with put_page and attached to iterators of their own. An iterator-of-iterators is then created to bridge all three iterators (headers, data, trailer) and that is passed to sendmsg to pass the entire message in a single call. Signed-off-by: David Howells cc: Trond Myklebust cc: Anna Schumaker cc: Chuck Lever cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nfs@vger.kernel.org cc: netdev@vger.kernel.org Signed-off-by: David Howells Signed-off-by: David Howells Signed-off-by: David Howells Acked-by: Chuck Lever Tested-by: Daire Byrne --- include/linux/sunrpc/svc.h | 11 +++-- net/sunrpc/svcsock.c | 89 +++++++++++++++----------------------- 2 files changed, 40 insertions(+), 60 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 877891536c2f..456ae554aa11 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -161,16 +161,15 @@ static inline bool svc_put_not_last(struct svc_serv *serv) extern u32 svc_max_payload(const struct svc_rqst *rqstp); /* - * RPC Requsts and replies are stored in one or more pages. + * RPC Requests and replies are stored in one or more pages. * We maintain an array of pages for each server thread. * Requests are copied into these pages as they arrive. Remaining * pages are available to write the reply into. * - * Pages are sent using ->sendpage so each server thread needs to - * allocate more to replace those used in sending. To help keep track - * of these pages we have a receive list where all pages initialy live, - * and a send list where pages are moved to when there are to be part - * of a reply. + * Pages are sent using ->sendmsg with MSG_SPLICE_PAGES so each server thread + * needs to allocate more to replace those used in sending. To help keep track + * of these pages we have a receive list where all pages initialy live, and a + * send list where pages are moved to when there are to be part of a reply. * * We use xdr_buf for holding responses as it fits well with NFS * read responses (that have a header, and some data pages, and possibly diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 03a4f5615086..f1cc53aad6e0 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -1060,16 +1060,8 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) return 0; /* record not complete */ } -static int svc_tcp_send_kvec(struct socket *sock, const struct kvec *vec, - int flags) -{ - return kernel_sendpage(sock, virt_to_page(vec->iov_base), - offset_in_page(vec->iov_base), - vec->iov_len, flags); -} - /* - * kernel_sendpage() is used exclusively to reduce the number of + * MSG_SPLICE_PAGES is used exclusively to reduce the number of * copy operations in this path. Therefore the caller must ensure * that the pages backing @xdr are unchanging. * @@ -1081,65 +1073,54 @@ static int svc_tcp_sendmsg(struct socket *sock, struct xdr_buf *xdr, { const struct kvec *head = xdr->head; const struct kvec *tail = xdr->tail; - struct kvec rm = { - .iov_base = &marker, - .iov_len = sizeof(marker), - }; + struct iov_iter iters[3]; + struct bio_vec head_bv, tail_bv; struct msghdr msg = { - .msg_flags = 0, + .msg_flags = MSG_SPLICE_PAGES, }; - int ret; + void *m, *t; + int ret, n = 2, size; *sentp = 0; ret = xdr_alloc_bvec(xdr, GFP_KERNEL); if (ret < 0) return ret; - ret = kernel_sendmsg(sock, &msg, &rm, 1, rm.iov_len); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != rm.iov_len) - return -EAGAIN; + m = page_frag_alloc(NULL, sizeof(marker) + head->iov_len + tail->iov_len, + GFP_KERNEL); + if (!m) + return -ENOMEM; - ret = svc_tcp_send_kvec(sock, head, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != head->iov_len) - goto out; + memcpy(m, &marker, sizeof(marker)); + if (head->iov_len) + memcpy(m + sizeof(marker), head->iov_base, head->iov_len); + bvec_set_virt(&head_bv, m, sizeof(marker) + head->iov_len); + iov_iter_bvec(&iters[0], ITER_SOURCE, &head_bv, 1, + sizeof(marker) + head->iov_len); - if (xdr->page_len) { - unsigned int offset, len, remaining; - struct bio_vec *bvec; - - bvec = xdr->bvec + (xdr->page_base >> PAGE_SHIFT); - offset = offset_in_page(xdr->page_base); - remaining = xdr->page_len; - while (remaining > 0) { - len = min(remaining, bvec->bv_len - offset); - ret = kernel_sendpage(sock, bvec->bv_page, - bvec->bv_offset + offset, - len, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != len) - goto out; - remaining -= len; - offset = 0; - bvec++; - } - } + iov_iter_bvec(&iters[1], ITER_SOURCE, xdr->bvec, + xdr_buf_pagecount(xdr), xdr->page_len); if (tail->iov_len) { - ret = svc_tcp_send_kvec(sock, tail, 0); - if (ret < 0) - return ret; - *sentp += ret; + t = page_frag_alloc(NULL, tail->iov_len, GFP_KERNEL); + if (!t) + return -ENOMEM; + memcpy(t, tail->iov_base, tail->iov_len); + bvec_set_virt(&tail_bv, t, tail->iov_len); + iov_iter_bvec(&iters[2], ITER_SOURCE, &tail_bv, 1, tail->iov_len); + n++; } -out: + size = sizeof(marker) + head->iov_len + xdr->page_len + tail->iov_len; + iov_iter_iterlist(&msg.msg_iter, ITER_SOURCE, iters, n, size); + + ret = sock_sendmsg(sock, &msg); + if (ret < 0) + return ret; + if (ret > 0) + *sentp = ret; + if (ret != size) + return -EAGAIN; return 0; } From patchwork Wed Mar 29 14:13:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192548 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11E10C74A5B for ; Wed, 29 Mar 2023 14:16:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A53DC900005; Wed, 29 Mar 2023 10:15:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A030A900002; Wed, 29 Mar 2023 10:15:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CA5A900005; Wed, 29 Mar 2023 10:15:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7DD67900002 for ; Wed, 29 Mar 2023 10:15:59 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 499B7C0C08 for ; Wed, 29 Mar 2023 14:15:59 +0000 (UTC) X-FDA: 80622134838.09.D7472E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 4C88340017 for ; Wed, 29 Mar 2023 14:15:57 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h28jJEc5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099357; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wvXdMcltJv+r5QoZnXaavWZcwnuZQd4mm223J864Zdo=; b=AQiewUmGnvFuKBJchTzgKVsDG1gxYme8GQNaintiHDTIgatul/k+s77+2ayODMvAQWqaF4 IcWQYiqXC6bnc0Gg4uCM7WvLbiFOaJfq6Phx3S7BipYuqMYXx8ZFa70+19iFs6w4O9jKK7 VhLHaCQCSWil26+rgWg2pNqyu1uhCAE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h28jJEc5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099357; a=rsa-sha256; cv=none; b=d6vup/G7NP8nnPJKnLfWNLUAeZq/zmgi3TcQph0L9K9MSWY4fCkbta4Az590O63liWjuJm QghFFCuXXP0xzE/zoESSxCBo4NoEnidcLIZeMy2jtbekMSOKUCx1ysd9r4zki2zldglvT6 yjzpMO99j4mGMQ+/+r4T/DA68XT7gXg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wvXdMcltJv+r5QoZnXaavWZcwnuZQd4mm223J864Zdo=; b=h28jJEc5iqv5DTA5FHBJ+Rd3KG/BK0O59T4O3AH0VrYSOD8ZtY2JWnhyWaIWpsV1NtJc2m zkXFpXM0MwQsrXrwLtW1YTEAFW49aoR3QvbX8fwSYjp+hpL3nDOQPPNdceD0fflbRnB4w+ 9GmGl2fNoej2dJbPQBMWA2d+0rXTChQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-482-Ff5bQfAPN6KgS6uIgJRP1w-1; Wed, 29 Mar 2023 10:15:53 -0400 X-MC-Unique: Ff5bQfAPN6KgS6uIgJRP1w-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 82A463814947; Wed, 29 Mar 2023 14:15:52 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5750F492B02; Wed, 29 Mar 2023 14:15:50 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org Subject: [RFC PATCH v2 41/48] sunrpc: Rely on TCP sendmsg + MSG_SPLICE_PAGES to copy unspliceable data Date: Wed, 29 Mar 2023 15:13:47 +0100 Message-Id: <20230329141354.516864-42-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: 4C88340017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 91yhthw1gupcjoke3j3bqhbusq4ambux X-HE-Tag: 1680099357-537643 X-HE-Meta: U2FsdGVkX1+EVLcLbos96CeZysV4N+eS8MmynRk5S083+WVvThNPHDZ4N1H0vCHl4mdhkae5+WVljxmIMjZPtxpZYQXhPpmpvH+sGZsDY9K73QWbILm/t60bl9KiOPUZPCre3STyeD6A5tlpEun/Fipr+kkLx/SWsb8U8FgM9SfDRhkzIJEoavsMsomJUzvdP6WuEvg8mJaw8aHw0bKsa/CobM7F7rHimXVunfPGOhN89K6zv6Le/7QTep893lKnlFrRKTlbdYvPkQBYiMT9f4E/AByMAYuN1rlxJsiX8Rb+2c4v6NZATFdTzAUl+Hw7AlakRTQlVKuC6b6COBqSwJgIRARqZ9rfGZpj2FUgMKaBE7sSuxzLgw0225BJi8+zfESRwaXQrALo/3oF1VJSZYnMHyv1GFEZjlzGXHU3HikdkhrbOCb6y04hTk236l1iUT7x3fQei9Kms7LTZTjklL30WDqz8A9XAyaz/BPo15xFldSzQb6sb9IvY1qbyPVzz8buAJre/hr6tJTfzQdfGeTjnMgKxR1uY9MGDmgLOXCmTyQM4jxd3nlWio3P5oVz5vfves+3FCLxzh7aYZyOsZWr06JyPKub3apPL4G2+oLPAwPI1Vre27j6MxqOr28TBhsL5hnJiNmtDPo82nko+ZZoF0dPUkuD9SpmKD0yb+U8enrcn+mbDGBehVZdVSeqF71nWHYiGmd6NNPiJJItWyGpccM8aefrvXXTxiROcfnYrN0lg4dn/KpwOaKOqhVjn4ekqvY0AmjazIXhGpSzPIY8+yeLamo4v9kJeahz6+dyjVqtP/4blNDMWurbNeAfQhsB+BXSnklF4uWfcEkTPpcqPzRirM+rZReLBgDZqY31b/0Z57Fr/WbhNgPtBAUQCFp0FMriFAZ93QBwQeJm8zwKeZTLqVT9OWmmrWF3W9QpFBABJ+V+ii4O8L4ahDdNwCk6ZsowH+ZSJQx+nzW S5Zg9UD5 TUch3ljuwzLZGp7pq6R1b1NYq7+CdHjaeyjaukfyT4Ro6uQ/sr29LvO/hjC7fn76TqiLnJ+fsmwnzPyMNbdeGtkZ3Inqcgyd7Maiuqi2PJWcEVZRwCf93csAaV54+OND8CeD8NBlzPeS4wXSwgJf2z2gRGFTlU54j2gZ8y8WPstotGRSuvAPXbTsu38isqJ171pO5Ya1OsEX4Wt5Rpjlq7fw0LGbswrh06cYB+0qW+FvG23jHPSlSVkHOIoJ10v+P726ygtgRV4yBEhc+F0JJ25mklBlZvxoj3kVyW9tvBCQwH5BQPV48SR5SgycdF0SsGNaOR3u7/KTIlTvfKySaEZ9AocfbDvr+VHGybCGCQ+62Jk+wbzDVQLKqpbfsDOmhL+9NILA6m29bFeRPqwPM7SsNJXgyag6eMACMyQQ7PQnEiy8S58g6owcIlDMX65zX0Hb8Vp7rzY/jMj15H9Fcdx9NnRIx5yV3MBkBG4A5/v2t3XpRfJ27Q8KRNr4GB5M8/DWAfTy3XdG+5aI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than copying data in svc_tcp_sendmsg() into page fragments, just hand in ITER_KVEC iterators as part of the ITER_ITERLIST and rely on TCP to copy them if the pages they're residing on are belong to the slab or have a zero refcount. Signed-off-by: David Howells cc: Trond Myklebust cc: Anna Schumaker cc: Chuck Lever cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nfs@vger.kernel.org cc: netdev@vger.kernel.org --- net/sunrpc/svcsock.c | 44 ++++++++++++-------------------------------- 1 file changed, 12 insertions(+), 32 deletions(-) diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index f1cc53aad6e0..c1421f6fe57a 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -1071,47 +1071,27 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) static int svc_tcp_sendmsg(struct socket *sock, struct xdr_buf *xdr, rpc_fraghdr marker, unsigned int *sentp) { - const struct kvec *head = xdr->head; - const struct kvec *tail = xdr->tail; - struct iov_iter iters[3]; - struct bio_vec head_bv, tail_bv; - struct msghdr msg = { - .msg_flags = MSG_SPLICE_PAGES, - }; - void *m, *t; - int ret, n = 2, size; + struct iov_iter iters[4]; + struct kvec marker_kv; + struct msghdr msg = { .msg_flags = MSG_SPLICE_PAGES, }; + int ret, n = 0, size; *sentp = 0; ret = xdr_alloc_bvec(xdr, GFP_KERNEL); if (ret < 0) return ret; - m = page_frag_alloc(NULL, sizeof(marker) + head->iov_len + tail->iov_len, - GFP_KERNEL); - if (!m) - return -ENOMEM; - - memcpy(m, &marker, sizeof(marker)); - if (head->iov_len) - memcpy(m + sizeof(marker), head->iov_base, head->iov_len); - bvec_set_virt(&head_bv, m, sizeof(marker) + head->iov_len); - iov_iter_bvec(&iters[0], ITER_SOURCE, &head_bv, 1, - sizeof(marker) + head->iov_len); - - iov_iter_bvec(&iters[1], ITER_SOURCE, xdr->bvec, + marker_kv.iov_base = ▮ + marker_kv.iov_len = sizeof(marker); + iov_iter_kvec(&iters[n++], ITER_SOURCE, &marker_kv, 1, sizeof(marker)); + iov_iter_kvec(&iters[n++], ITER_SOURCE, xdr->head, 1, xdr->head->iov_len); + iov_iter_bvec(&iters[n++], ITER_SOURCE, xdr->bvec, xdr_buf_pagecount(xdr), xdr->page_len); - if (tail->iov_len) { - t = page_frag_alloc(NULL, tail->iov_len, GFP_KERNEL); - if (!t) - return -ENOMEM; - memcpy(t, tail->iov_base, tail->iov_len); - bvec_set_virt(&tail_bv, t, tail->iov_len); - iov_iter_bvec(&iters[2], ITER_SOURCE, &tail_bv, 1, tail->iov_len); - n++; - } + if (xdr->tail->iov_len) + iov_iter_kvec(&iters[n++], ITER_SOURCE, xdr->tail, 1, xdr->tail->iov_len); - size = sizeof(marker) + head->iov_len + xdr->page_len + tail->iov_len; + size = sizeof(marker) + xdr->head->iov_len + xdr->page_len + xdr->tail->iov_len; iov_iter_iterlist(&msg.msg_iter, ITER_SOURCE, iters, n, size); ret = sock_sendmsg(sock, &msg); From patchwork Wed Mar 29 14:13:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AA72C74A5B for ; Wed, 29 Mar 2023 14:16:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74494280005; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F5EE280001; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56ECD280005; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 49855280001 for ; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 267C880D3C for ; Wed, 29 Mar 2023 14:16:09 +0000 (UTC) X-FDA: 80622135258.26.6AB6E89 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 37EC98000A for ; Wed, 29 Mar 2023 14:16:07 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gMnrT1VZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XvqldHWl7xh/L1V+KGWEm9YizW/K8SDxYbPNnLnjdW8=; b=YbfjlCwPpRTHw1vty9rVYRVmFIhylXsR7iKgrx2UkX4O9RY+Wm2Lyl/qwwcugtW6TSUZFG oU9kVATgvcjyZX9r8p06BHNAHnWq5vdA4nAegmblCaXIH/8cro0ye+dzJn8UQNspPBwqeD RtFtK7OGoT0mfPhM36/Bv/+tTAeFUKc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gMnrT1VZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099367; a=rsa-sha256; cv=none; b=NmDXPEcOOXAT+i/FDy9AR3LCpGxknzt1p0hUhNC3j8wQeJzASdfYtvcgGZ7iGODcbxMeAp kmJma/5qimJv7V/dSclppRnVhnmlWyLs/3/zrP0QuHCcRdpxLVxsAhboWgr5aE5bc0DFiZ 19UTia/P5FbGKe6s/Jcky7Yq1lPKB28= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099366; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XvqldHWl7xh/L1V+KGWEm9YizW/K8SDxYbPNnLnjdW8=; b=gMnrT1VZReB3VCGPW4HfPeDVWSJNt+WUHWs41zPUZNaE9xUAaZOeexhMW6U1hLj4O3o1Gg 6T6U5bh9xMUZr36vuXqCR3VM/N560Ucexw4sP50/Xfo93ZFnGaaVRGJf5nn/YwMrdrnzbB PP3FYhYmxS6MHi7y+ASYuvmp2gxCEcs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-588-COzuqKOVOGS_OcuoCYtLdA-1; Wed, 29 Mar 2023 10:15:59 -0400 X-MC-Unique: COzuqKOVOGS_OcuoCYtLdA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9E5AC811E7C; Wed, 29 Mar 2023 14:15:55 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 263DD1121330; Wed, 29 Mar 2023 14:15:53 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org Subject: [RFC PATCH v2 42/48] nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Wed, 29 Mar 2023 15:13:48 +0100 Message-Id: <20230329141354.516864-43-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: 37EC98000A X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: w1ne8wi1ke1gkzafaq1r5mmefh715hxj X-HE-Tag: 1680099367-963315 X-HE-Meta: U2FsdGVkX189tWTLrr49QuoGSHdh2pl+ZqAUCErhnW+3jDUeVtYTIzHEY7FIK1MSTDB/K+N9SsVafxKNGaYKX2t0EqWwHBUPXgK1+22oTC/hLuSRS2jZcC6gs1VfOLFUTY6looNkQHhuE6pQVJyxOsvelVRGgVTUAmlSVdYhEBWEV4cAO9Fy5grzrQRLAJaDR8NfAqzxMRaqVAJGwh4eCJ/RM04F6ktFJJRpFNWEa44dHbUXfo36/kews6psCCpsp33HMSkI+Wnkx7oRE6vXpaBQIWecJ6pc2DcUyOKKtWr3FAvxNwcTU/2yP5q8GbNKyg+SJAaixaSdrXqBr3hly2ptWUzuf3/KNGUTzbHW+CQthyuMlV/6N0K51omhJEAR+G5uE0EiSIvmAM+dINejGc4fdjUyS3Sj59vyns0nQWn8OI+1zD7j1wQEL8vkVp90aQJv5BXhk4k3X1ZKbIG3w3x2K60NsUReKGU6tnTlcEkYKbxeTr13LMEQ9nhcXOIAeTEajzH3pcb/r1FcblwYcEfwOmAFuqIAeJoPvoP3X9PIp8cvJPoxflDaFCUaTBfpcA/ZySP0zBV64tHyoYd74G7xy2RQ/weN9k07xxxpBO3siB4jyxvYr11eAkTFeS0nPCEE46lQ5TeoeQaWXcW0p8m/hziluY+hMzlwgHUl31FoRklSn2plCIHj3Gi/wmYkRTYZObSnb0BMCg0Vxh5WCnfYK5NbR/PLSspipMFw4GgPxaqoW07Q/WDvMGbEihOo00aZtWJbZL90fHFI1X54uZli1FdtvZpDKPWdgnqbCz/U2/QDd8EhK1jAgaOkZMUc5VWsZMaQFo3Re+87u8DsprDkI+Ur4Ti0k03LoOZK4/KHZ6hl+JG3tU/8I1pfunyheDlrk929qN3GMvY2oRn4d0jZQdfbDAP3u1ivsD6Zx48N7PX2/gkpYEf+4op5nf4flgKmO/pjYiLgzfo6AKf nPSkFzOR Qmq1HELyW/+WXR8xNnYnOafyLqe49A9+IbkD9ofVioUj/+aynUepa9KmHTs1JrwYi1mfyhRBGZJ9QBrQmB7IUqF9GJwLukdwaUU9jFjNqeMj2qJyWZoy4o9M2AtGlkjoVO/rX4MYODsqi2CxnqUxf9fS4fefryNxh1riMWLQb5DB2K1mau/a89gprmU6ouNsQrifTUSStKLtS91FtacXyoB5BwF72EAyzOP7KiBK4/SnXaQRoqJJrHoCJUycVJcJtdVC7If67Lq8h0BHkD+fCOggo/IpLhOQppQNuvw637uEu+9XakiDVUHbPMSOxhr0MVrpuL0xTOmFFErWPXX0hEJe+iFWPBbMEx+gYwIfb/1sMuqjy2PNXad2ialhit5ct2rXBUEUiyNsQoq4CWoAVpztcFJ//fDyaoBXZ7TBbqT5Mwe5+CkdKdm4eN61HbbwfiUK0Ui9Dkztdp4HHEaWGHWvaef8agYPalVRxVsSLi4cPSJ1oir4v9pMDTLwtxeIyqF1N30qsXCx0lgs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. Signed-off-by: David Howells cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nvme@lists.infradead.org cc: netdev@vger.kernel.org --- drivers/nvme/host/tcp.c | 44 ++++++++++++++++++------------------ drivers/nvme/target/tcp.c | 47 +++++++++++++++++++++++++-------------- 2 files changed, 52 insertions(+), 39 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index fa32969b532f..cc617692702d 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -979,25 +979,23 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) u32 h2cdata_left = req->h2cdata_left; while (true) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; struct page *page = nvme_tcp_req_cur_page(req); size_t offset = nvme_tcp_req_cur_offset(req); size_t len = nvme_tcp_req_cur_length(req); bool last = nvme_tcp_pdu_last_send(req, len); int req_data_sent = req->data_sent; - int ret, flags = MSG_DONTWAIT; + int ret; if (last && !queue->data_digest && !nvme_tcp_queue_more(queue)) - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; else - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; - if (sendpage_ok(page)) { - ret = kernel_sendpage(queue->sock, page, offset, len, - flags); - } else { - ret = sock_no_sendpage(queue->sock, page, offset, len, - flags); - } + bvec_set_page(&bvec, page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (ret <= 0) return ret; @@ -1036,22 +1034,24 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_cmd_pdu *pdu = req->pdu; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; bool inline_data = nvme_tcp_has_inline_data(req); u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) + hdgst - req->offset; - int flags = MSG_DONTWAIT; int ret; if (inline_data || nvme_tcp_queue_more(queue)) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; if (queue->hdr_digest && !req->offset) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, flags); + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (unlikely(ret <= 0)) return ret; @@ -1075,6 +1075,8 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) { struct nvme_tcp_queue *queue = req->queue; struct nvme_tcp_data_pdu *pdu = req->pdu; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_MORE, }; u8 hdgst = nvme_tcp_hdgst_len(queue); int len = sizeof(*pdu) - req->offset + hdgst; int ret; @@ -1083,13 +1085,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); if (!req->h2cdata_left) - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); - else - ret = sock_no_sendpage(queue->sock, virt_to_page(pdu), - offset_in_page(pdu) + req->offset, len, - MSG_DONTWAIT | MSG_MORE); + msg.msg_flags |= MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST; + + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(queue->sock, &msg); if (unlikely(ret <= 0)) return ret; diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index d6cc557cc539..00b491abf50f 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -548,13 +548,18 @@ static void nvmet_tcp_execute_request(struct nvmet_tcp_cmd *cmd) static int nvmet_try_send_data_pdu(struct nvmet_tcp_cmd *cmd) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->data_pdu) - cmd->offset + hdgst; int ret; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->data_pdu), - offset_in_page(cmd->data_pdu) + cmd->offset, - left, MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); + bvec_set_virt(&bvec, (void *)cmd->data_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; @@ -575,17 +580,21 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd, bool last_in_batch) int ret; while (cmd->cur_sg) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, + }; struct page *page = sg_page(cmd->cur_sg); u32 left = cmd->cur_sg->length - cmd->offset; - int flags = MSG_DONTWAIT; if ((!last_in_batch && cmd->queue->send_list_len) || cmd->wbytes_done + left < cmd->req.transfer_len || queue->data_digest || !queue->nvme_sq.sqhd_disabled) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; - ret = kernel_sendpage(cmd->queue->sock, page, cmd->offset, - left, flags); + bvec_set_page(&bvec, page, left, cmd->offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; @@ -621,18 +630,20 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd, bool last_in_batch) static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd, bool last_in_batch) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->rsp_pdu) - cmd->offset + hdgst; - int flags = MSG_DONTWAIT; int ret; if (!last_in_batch && cmd->queue->send_list_len) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu), - offset_in_page(cmd->rsp_pdu) + cmd->offset, left, flags); + bvec_set_virt(&bvec, (void *)cmd->rsp_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; cmd->offset += ret; @@ -649,18 +660,20 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd, static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; u8 hdgst = nvmet_tcp_hdgst_len(cmd->queue); int left = sizeof(*cmd->r2t_pdu) - cmd->offset + hdgst; - int flags = MSG_DONTWAIT; int ret; if (!last_in_batch && cmd->queue->send_list_len) - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; else - flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; - ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu), - offset_in_page(cmd->r2t_pdu) + cmd->offset, left, flags); + bvec_set_virt(&bvec, (void *)cmd->r2t_pdu + cmd->offset, left); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, left); + ret = sock_sendmsg(cmd->queue->sock, &msg); if (ret <= 0) return ret; cmd->offset += ret; From patchwork Wed Mar 29 14:13:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2681CC6FD18 for ; Wed, 29 Mar 2023 14:16:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0001280004; Wed, 29 Mar 2023 10:16:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B897E280001; Wed, 29 Mar 2023 10:16:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93F5F280004; Wed, 29 Mar 2023 10:16:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 80FFE280001 for ; Wed, 29 Mar 2023 10:16:07 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 486131401E5 for ; Wed, 29 Mar 2023 14:16:07 +0000 (UTC) X-FDA: 80622135174.30.C268A0F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 81317120027 for ; Wed, 29 Mar 2023 14:16:05 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VZTDfih0; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099365; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T9xMfNdsrDCo/ep+SBwQk+++P3r8+NfDYHGjOqRd99M=; b=aokjbonNtnfBn9l0sTPCREF0ocvgKPod21Gioh1pRuy358JHQYbCmYFfZA0HWVlEJo+TtM +94+4aDo3tEksGbo3YTJLzxaTSrR1qJNLSm9SWXUIuWwey9mkAnjSCoM5x4ayFhGHZMyPk 7x5x5w0qDO/BNN1U8/mIUaHrOWvtfQ0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VZTDfih0; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099365; a=rsa-sha256; cv=none; b=BkgPP34DafeylAve1Y5eaMFWGQJJe3tF38uEGdY7wc2q5NdF1koz8FaqYxiWrYje43FvKP 5/LgvFIVHPr6EM0xgoR/RY624Ny5cIYoyc03o6WZzXPug4eWDxywerP+KBCl5RNDBFn5Ji /ZFxXC4mJzJLJacmYW64ioYfUrTINjc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T9xMfNdsrDCo/ep+SBwQk+++P3r8+NfDYHGjOqRd99M=; b=VZTDfih0xKpPSpaze8DW9Or8Vwt6+rEYFJd1h06L8xU7nrLDORCD7pZtOxYgF3ZsuMqrob UZrRfJ6T+6MaWAlkI8Lov2ZFQ2FA8h6Rqw/Di91D1+eOwHc2NceL9FVciY63SVdhp4KL2C R1YeFplyxtmMMN+/s6oU/4NWKhsfddc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-610-Ojh5V2k9OAOrBsNjIDnCsA-1; Wed, 29 Mar 2023 10:16:01 -0400 X-MC-Unique: Ojh5V2k9OAOrBsNjIDnCsA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7C34E299E77B; Wed, 29 Mar 2023 14:15:58 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 615572166B33; Wed, 29 Mar 2023 14:15:56 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Tom Herbert , Tom Herbert Subject: [RFC PATCH v2 43/48] kcm: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Wed, 29 Mar 2023 15:13:49 +0100 Message-Id: <20230329141354.516864-44-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: tnagcudetsm8ggj5yigq1c8mcqpft9x3 X-Rspamd-Queue-Id: 81317120027 X-HE-Tag: 1680099365-689601 X-HE-Meta: U2FsdGVkX1830V6t+00IfsU3rgeMaFfpM2X4OSLl4Aj+RIP+OCxUaDDNb2wuy9rXU3jfw9cB5nd1jMsla1dLhwxv5PydhyhcT8vf5qU9Nx6dc5Bpgvu37t+UXtnm3WmMe6kDHBoT2WT/AkvMH5X/eFC01kXISP7HGnSJ7RbgPVjr6oMw3vcnlJyxFMD4k/uuR2uCfzyYJFdXi41aOpngWJvrc/NiQKXHQwUTvo0OJGZ16DtgcY6NhjgVFXgq/Ky7PL02cgVnG5gsh5QlzbysMWEBkedL3GpbgrKoTovlAdRnQ3g3Iryd3UJ+6GhK0yfej3VdQFzxZ1OX0J2LjkI6752iCgkKH64IjogL/oIxJ9KeT9ftjQGs4pvcjcKnAwy7VJvYheQt2ggq4/nmF0YPt1U8LlVTqJB+KwFnsZwLiwYTxC3wJujKk0LVe4NyrGNi+FsyAxPy4uu4z4moJslAsjcjW7EtXBHSmMg720pkKoP44VGQbJte6/SAmwF87WAADG3RY4GbQncARHfTPrLJtUsSH0lQF7cxNRjx+jg+1WFPMF03MQaC3MOwshgAABQJJo/6hS8S1AITx+e7ku/36n5ptbUK6HTv8pHf9tZVhFbqzRSYaKWJtmOkz8iPB4mIB2b+2iPLTxE8KvOvIR9IJGbIFzEbl2tCwQ+a6bwlA43zlymWu4LwZqSetoQcMVRlQKFsGt4ZLBVpyhZ0bI9sb5fsl1TIx6zcIMBdbop1kJksoubBrW9D91Y1WJZLM3asiU3cc7+IyDWPNi4pEGfDoKcPR53DcdJAY2D6eXihVI8lKYIPHvdBUMVtq0QA1DkaQuH75a2nRO9fOsP9bQC2tImngknB5EFkQa/GVlDSnOXTh/l1oNc93CwA2PlNYN8JBmERgV69gX0oV7OsTSZjd46GgUXJRcluoqmI0qZKk8l3cwS5au6zkoCzLjrMuTt7l0c3j0IHHs3bRUTtXXL vXDsM52W 6soFVvgtNlQ0VC8CwCgIrK9KMkHS1ZF2N+J/trPRA4OgEXwLh/+gANU1OYRBlmVVqiZMTt/GdZOZ45tMeQFw8i64NEDT1Us1hA078+8dYGUQf8E1dlGIdMJ+SgAd6umqnMAMQ9IDPjzmDW7Y2yUN5PBh/60p6uxYzPZY+rOb50tXX8hvGHMKVvTarDXTTqIxJ5M33FxixaQEV430pRISeyNZe7KqzfSojlPgLSeSz6WR6e//A9nLabL+D9hxwGBedpcpp/rdKPBzqcvqXoHxyxRG5QZU+NdeVmOlCTxwHYDiFy+AGegcqjRmh8/yJSVCFcujaYzpfnQxPAqbfW4wCsomi0ZCfm+oi/QKhtannQ2C8jvAoRANQczLuX0D9LUDIMO+ycEmDw7OCI5kcXg/BUSL+ix/hUR2GezTilD8p61vKDPcGYqjJABj64taLA5OfShkCHVYjNPlDbDqq3cOru/yTtBXbifb50lGRJQS+zPP1qwUvGJWa09T/x1DYInnfe/0B64748G3vNH4UD1KJB1uuPy1hX/LI4cHOBGqoVOfLRWQuPdTVgFyRcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. Signed-off-by: David Howells cc: Tom Herbert cc: Tom Herbert cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/kcm/kcmsock.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index cfe828bd7fc6..819259149952 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -641,6 +641,10 @@ static int kcm_write_msgs(struct kcm_sock *kcm) for (fragidx = 0; fragidx < skb_shinfo(skb)->nr_frags; fragidx++) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, + }; skb_frag_t *frag; frag_offset = 0; @@ -651,11 +655,12 @@ static int kcm_write_msgs(struct kcm_sock *kcm) goto out; } - ret = kernel_sendpage(psock->sk->sk_socket, - skb_frag_page(frag), - skb_frag_off(frag) + frag_offset, - skb_frag_size(frag) - frag_offset, - MSG_DONTWAIT); + bvec_set_page(&bvec, + skb_frag_page(frag), + skb_frag_size(frag) - frag_offset, + skb_frag_off(frag) + frag_offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, bvec.bv_len); + ret = sock_sendmsg(psock->sk->sk_socket, &msg); if (ret <= 0) { if (ret == -EAGAIN) { /* Save state to try again when there's From patchwork Wed Mar 29 14:13:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89876C6FD18 for ; Wed, 29 Mar 2023 14:16:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3708280006; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E6F80280001; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0F1D280006; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C01B6280001 for ; Wed, 29 Mar 2023 10:16:09 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A682FC087B for ; Wed, 29 Mar 2023 14:16:09 +0000 (UTC) X-FDA: 80622135258.11.6028E76 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id DFA3840025 for ; Wed, 29 Mar 2023 14:16:07 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KBOPmya5; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7fCBjYGLxHqikc/tNao5Lfq+HewWC5ZgTjyFTAyeW0Q=; b=TEHAuHzibRgmAljNU09jynAyrvfgvt3QXfKPf9ldCNv/Ndzafilf8hNnFUEUuEbkjdap7S b23zt2gQRDHn3p1b5A46fnkYmOw0Xtam8YLXe3krKr2rAtquAd7FFvF0LzWSd7qpIj6Dpp 5/b5oH7leOtE219+S+Anmj6GNpjOR/I= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KBOPmya5; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099367; a=rsa-sha256; cv=none; b=bsjqt3CnvZ+IF1fF52H7yv0HGzyTuoXtqU+cmyPM/WcHA/C/WJoXCM81QFzeefj6iES7RK yQ4xah+VjUuK4EM56+kspn7zC7Vgi5/3wxMZYaF7AmsQ5KRcwh+6LS7EK4fpAsMXL6mA8F XRkjPQZHwZPYPnAnI/ZBoep5BHbwqzI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7fCBjYGLxHqikc/tNao5Lfq+HewWC5ZgTjyFTAyeW0Q=; b=KBOPmya57uLtfFCnt6RqWs1HdcNCaRpAyGONfZh+qQoOoz/ce+iLhvkprxUZwcLi08j9Qo 6ilriPPUbRFmw/l7VP4o/LWAd6yEqvqifHV7oMGpKG+B9jSfzcyueSek70M/zoNIFwVi4R QaAYN4sV6DrsF2BJIJriiGusxH4Il+g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-114-46eAsXOAN6OrZ73_2qvGIA-1; Wed, 29 Mar 2023 10:16:03 -0400 X-MC-Unique: 46eAsXOAN6OrZ73_2qvGIA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5E15C8030D4; Wed, 29 Mar 2023 14:16:01 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1928E202701F; Wed, 29 Mar 2023 14:15:59 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Karsten Graul , Wenjia Zhang , Jan Karcher , linux-s390@vger.kernel.org Subject: [RFC PATCH v2 44/48] smc: Drop smc_sendpage() in favour of smc_sendmsg() + MSG_SPLICE_PAGES Date: Wed, 29 Mar 2023 15:13:50 +0100 Message-Id: <20230329141354.516864-45-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 8ad3pf9i643or1jm646wgxi83efafz9s X-Rspamd-Queue-Id: DFA3840025 X-HE-Tag: 1680099367-346507 X-HE-Meta: U2FsdGVkX19BTS5qkq7+e1Q9ifoIZOLR32hwXnkNIAp/wbMfGvPpjEczLzWtfXOq808EOxXYeK0LAatqxJZU1GMAaw2fccW0iWS9EFRDpI3oI6OM3czJZZ63lJudSpL7BKA7Mv78o418sUd4o4TISCJjWNdFd8rD3s7x411OxiRj/GsQNPAe4FYN8KpKOd4cnVOUugPrWZnSXXhm5+Rssp6lABemsjL9g/T7lMIK+GAC6AYSS9kKndS1aBRbtkVDCa0M39lxpW/yZro+PNGYiXm/IeJu55r7UbnHeX1gLT/tfl6kNgwvHHt3KXjYW6NbanydEjhggzYcYvj9SNJFJtfAMKomDiRZZcQiTMOh9oPybVoSRx0TxcmDta7SLFAAcc7EKcKv/OtwOEolzksJFLo06oEpGI1l5+oJ/EsvFoXkmJ5GQy0Kt8FMnDyazKdgL4vZsskXBpiG8i22JnBOz+slmac8mI7af5xFgt0FEadCjqxUS37RuFFffd60zRtBY+N/4ppHn+T+DkYKRYTGHTspM3sSVxRJdiWX/uR8xNzdKBzAFUJ1+sBuHBkiy3JpGTB8Bng5Oii3gG6E3wFenlHWKqw6FYpy8ljpLuSQsL5KYGCFJ5B5qoQOL6wn1SgLElCWU1Ua4TUZjyvjaQuyA6F1E59gddJ2libOEjTg2UOUMVln3N4vLGtwP4LOOAVbgbzm+mddQUwipuYMDGgyq/9psNzW2Sc8bH1ohRhcYHMeP+5kpg5GPaJ4AiQzgTa1OL0K9hsQFCOY+vLfqehxcUitBnLo/9awIc5p5l43QL298pqCTZ00ne0mgfXcHf8GsfV2/BWmHxRV0WBkvmmO+k1Ubrrbju463nW3sNtSyjcVR3qfGiISR87DwOqdRQGBD/p25UZRTZ6dq6EJXrtTeo/uz9qBgz8U+/XoGad9rOISgegWbmI5pdUlixxKwRJ0wBDsoE2vOQrrHTxDa90 zRL3kN2u TkGYp7IqRlg71AZBWTGy1XNKJlGhg4b78GzW0KIfngCJEIN1qt1CTMfNN4mV584xD9jb01BSRmT7Ei70fKKJwvOxOrXpA5EtGZLwunjamuKDNErg7wrEp8xl3WmEsex6h6S8Hcxw9upImC6UBpz1cqP0K7WH5DMoXyZYbWktoEElSLK3QfjKuMfUZ1/SOBk27vjPj8b5oiNv0t8/AwrQhvo3bzZ9jEEEYKMl2Snry9pfTO6FelTz1nE+bUFuqRwwB1oUMQQgyZwCJWWa9F/X9OjBv4p9fnM+UdaFwIM/FD3bvImjI1lBxoD4MXeiafLHG0thklX6f4ntUfXGYIg6CfmZ5OouaKsL4PhAcLKdEspg+40Nm71yoBYrVNsAFRE/GlMHTFjCd/Gf+pXSAapxh6Gp8kHVegdyE3pvWodrQDW/37CL4SgtXLBc6Go0qt4NMZRt9i8miNXlTgnYLes/B5vVVaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop the smc_sendpage() code as smc_sendmsg() just passes the call down to the underlying TCP socket and smc_tx_sendpage() is just a wrapper around its sendmsg implementation. Signed-off-by: David Howells cc: Karsten Graul cc: Wenjia Zhang cc: Jan Karcher cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-s390@vger.kernel.org cc: netdev@vger.kernel.org --- net/smc/af_smc.c | 29 ----------------------------- net/smc/smc_stats.c | 2 +- net/smc/smc_stats.h | 1 - net/smc/smc_tx.c | 16 ---------------- net/smc/smc_tx.h | 2 -- 5 files changed, 1 insertion(+), 49 deletions(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index a4cccdfdc00a..d4113c8a7cda 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -3125,34 +3125,6 @@ static int smc_ioctl(struct socket *sock, unsigned int cmd, return put_user(answ, (int __user *)arg); } -static ssize_t smc_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - struct sock *sk = sock->sk; - struct smc_sock *smc; - int rc = -EPIPE; - - smc = smc_sk(sk); - lock_sock(sk); - if (sk->sk_state != SMC_ACTIVE) { - release_sock(sk); - goto out; - } - release_sock(sk); - if (smc->use_fallback) { - rc = kernel_sendpage(smc->clcsock, page, offset, - size, flags); - } else { - lock_sock(sk); - rc = smc_tx_sendpage(smc, page, offset, size, flags); - release_sock(sk); - SMC_STAT_INC(smc, sendpage_cnt); - } - -out: - return rc; -} - /* Map the affected portions of the rmbe into an spd, note the number of bytes * to splice in conn->splice_pending, and press 'go'. Delays consumer cursor * updates till whenever a respective page has been fully processed. @@ -3224,7 +3196,6 @@ static const struct proto_ops smc_sock_ops = { .sendmsg = smc_sendmsg, .recvmsg = smc_recvmsg, .mmap = sock_no_mmap, - .sendpage = smc_sendpage, .splice_read = smc_splice_read, }; diff --git a/net/smc/smc_stats.c b/net/smc/smc_stats.c index e80e34f7ac15..ca14c0f3a07d 100644 --- a/net/smc/smc_stats.c +++ b/net/smc/smc_stats.c @@ -227,7 +227,7 @@ static int smc_nl_fill_stats_tech_data(struct sk_buff *skb, SMC_NLA_STATS_PAD)) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_SENDPAGE_CNT, - smc_tech->sendpage_cnt, + 0, SMC_NLA_STATS_PAD)) goto errattr; if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_CORK_CNT, diff --git a/net/smc/smc_stats.h b/net/smc/smc_stats.h index 84b7ecd8c05c..b60fe1eb37ab 100644 --- a/net/smc/smc_stats.h +++ b/net/smc/smc_stats.h @@ -71,7 +71,6 @@ struct smc_stats_tech { u64 clnt_v2_succ_cnt; u64 srv_v1_succ_cnt; u64 srv_v2_succ_cnt; - u64 sendpage_cnt; u64 urg_data_cnt; u64 splice_cnt; u64 cork_cnt; diff --git a/net/smc/smc_tx.c b/net/smc/smc_tx.c index f4b6a71ac488..d31ce8209fa2 100644 --- a/net/smc/smc_tx.c +++ b/net/smc/smc_tx.c @@ -298,22 +298,6 @@ int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len) return rc; } -int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, - size_t size, int flags) -{ - struct msghdr msg = {.msg_flags = flags}; - char *kaddr = kmap(page); - struct kvec iov; - int rc; - - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &iov, 1, size); - rc = smc_tx_sendmsg(smc, &msg, size); - kunmap(page); - return rc; -} - /***************************** sndbuf consumer *******************************/ /* sndbuf consumer: actual data transfer of one target chunk with ISM write */ diff --git a/net/smc/smc_tx.h b/net/smc/smc_tx.h index 34b578498b1f..a59f370b8b43 100644 --- a/net/smc/smc_tx.h +++ b/net/smc/smc_tx.h @@ -31,8 +31,6 @@ void smc_tx_pending(struct smc_connection *conn); void smc_tx_work(struct work_struct *work); void smc_tx_init(struct smc_sock *smc); int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len); -int smc_tx_sendpage(struct smc_sock *smc, struct page *page, int offset, - size_t size, int flags); int smc_tx_sndbuf_nonempty(struct smc_connection *conn); void smc_tx_sndbuf_nonfull(struct smc_sock *smc); void smc_tx_consumer_update(struct smc_connection *conn, bool force); From patchwork Wed Mar 29 14:13:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7391DC77B61 for ; Wed, 29 Mar 2023 14:16:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E968280001; Wed, 29 Mar 2023 10:16:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 596D6280008; Wed, 29 Mar 2023 10:16:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40F70280001; Wed, 29 Mar 2023 10:16:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 033BE280008 for ; Wed, 29 Mar 2023 10:16:27 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BA3581401E5 for ; Wed, 29 Mar 2023 14:16:26 +0000 (UTC) X-FDA: 80622135972.07.CD15B83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id EF92940004 for ; Wed, 29 Mar 2023 14:16:23 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Oh7rLRp8; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099384; a=rsa-sha256; cv=none; b=YPI+WEITGHp2QaiyMwNfIdbxnA5pctygSuAHbsjcocbMXLHUfqsnIdPesSPH1afSoja4oi rn/bEShh5TqNLCO6Ie3VSnigne4BTKBBMM7XLIvFIvcBdKXSsGTvUSFtCTVX3TnKX/+bwQ c7mVdSXnkmu9b3GGfRFC2BxlVwF4Y4M= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Oh7rLRp8; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UieBw+fd7YOFzWwOOxTgfiaOtpF+6tMbkd08W9f/ez8=; b=twkbHcwiXIPwhVc7Wo33t86hMR2Cdk5crk+akl2IMJzyDLu1CdHxnUCC7uU2qMZLJNu/a1 43ZWDShwNjLLXLxtb2yE7G6xCd94UDACqsn9YRVOu3BBmFy7lHYTIT1LdgUENKQmRM55aC mvMt6RMcwWZvLrxjpMHSPDhcuGuX+O4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099383; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UieBw+fd7YOFzWwOOxTgfiaOtpF+6tMbkd08W9f/ez8=; b=Oh7rLRp8xJIBeVqnPVbKvpi1s0JXtKt84sG3AKbBNDeNdLYlcri4a5fiofLXj+1aLFXnSE 2xBt7Xx0RSVVXwXH3grOF+F44ugWyU8hxEycTJIAM+ViUmPYeef/iQAfGabienF5uRkQ0b jnKePLHFUptMBbCprQSYz8PSwqB1L8A= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-446-OM0xdWQoMNypMjdol7M5aQ-1; Wed, 29 Mar 2023 10:16:17 -0400 X-MC-Unique: OM0xdWQoMNypMjdol7M5aQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E9A9299E75F; Wed, 29 Mar 2023 14:16:05 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1FF3340B3ED9; Wed, 29 Mar 2023 14:16:02 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mark Fasheh , Joel Becker , Joseph Qi , ocfs2-devel@oss.oracle.com Subject: [RFC PATCH v2 45/48] ocfs2: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Wed, 29 Mar 2023 15:13:51 +0100 Message-Id: <20230329141354.516864-46-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Queue-Id: EF92940004 X-Rspamd-Server: rspam01 X-Stat-Signature: 7fgpjn8yxm715nzbnydkoc97p8r145ij X-HE-Tag: 1680099383-753161 X-HE-Meta: U2FsdGVkX19zmV+wU7iiS3rPoFA3ok+mmoJJkWKBOegruAZ+aH5DeYuB93LaI6uvPZwWrEx+wRxSfMEG04U4Va6sexk7FNMA2l2adXDeK6lr2l6GYe1CFe2SDxn13yMjR5VyuPl9N8xXtuicuJXt38MuTrIJBrOfUXpQiG+VYiSzDvMtOwyYRvOsvaWjxkVExybcA8+uQO85M03oNPaho2N7bpShMUPzl3vsd0ToKtFuYnXMQYxGJ8KzcwuHg/5Egfpsp2LksWioUMhZdPtatU/jGxqOYxZ9Ly/7GRBVJQ2z/2GfJUy0s2cKIUGhGAuCP1JuzyKhGbimlmEFNLb7awIGP4lfNlRxL2a4EXZwbar8fgtbX/5WtPlH1J20tY4bysbHY3+xjmgxWVLAdS3FCTYIlYcttqtMwO4Y4hAKvYXTBpusDttO+8uLLw1vYhW13YnrhRrj/XdxtMrN6n+uNG5dctumilR9X0Cw2+SY627eRf2VEaN6elFwAzqbC238K3sK2xJyJIePHXIDtLX2+vYvISyzl8U0pDmb0MsW9wl1pbLuF4O4zLtge66VZqYWGmRYO8qVHaMJE6uk72psk2sMAuHBK/1kfpObTa/E1p/R7N/IqwsHv+R1iVVRmuU54jZabseAffpydc1dLe174LMf1ijImNJRX8EwYQv6zqZWZ1WZZ6Y2gHLa2wXDtB9f1chnAtSuXI6vb+YuWTqivYHUTMoA4bjdxfNvbNDKNP6gKjRo4Q++mYcIsTwrN0ZHBHZXQkakqJB+rrL2DwqRUD+varg7YN5u5JEqe8gVU0p0G2a38UiLcQgmNHaQH6+TC2o8CZIsuv5xAn69VYwXVwyHC/Kfb4GzzlVD2cIDO8BGPXKV2oETvi8apBgLt03LxWffyM4/0HX7PozlTPbU5YZamvgw00G0KOlOuI/t6qq7jPF2/e0vW/myaTLZ/8rq5vI2U4IbU0wNlSWugx/ e2CKGDQ2 YXeAzs4fJ1j5vM8W8kvN3RSsITHO+ym/9DTp3/tCUivYqSg2T6RFUG220TRNs3qu+JAw5wBVToblIu8UDN1kIK/KgF64FHqIBd2gJ8bTnUELFM9o1XEP/RcT/C/BDEykes8zX8UttQuB4pprVdhI41Fg+WgTNkg1DoUs6iiF6zbUjxsl4wnzKe3DfYakCCPA2qS1EAqTbjj3CiWFbusihrnZI/NsninhTtSdA7gcgqZ0+IVbJUtlGNoxk7yQFf4Yey8Nijo4wdVoj18Lman2fylAGmCdsFvKrW0fD5GFmuZAt+Baa2XSpnDgPehAsb4vbslM6/tEKH6kv5Xwg197wCFv6c71UTvRJAMD7W4pOxmv1c7YuR/V2g5hab3t5UxSOsinlk6iuhJ89YFfGKWT/F8aLNpMhhKusdsv9PvaAqtlbJSUM2To82B/51JFL2y5PJDVNWmq4NDnAMTBIS1boNFeSy71wFHNQUoUqcw/ZJL+FgiXlRgD8xWA4WhWUX8rlzTLo4xfXLa2AcoZZM8AxPBhCNcDNCiWtGIP5rc7sdeUXeAo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix ocfs2 to use the page fragment allocator rather than kzalloc in order to allocate the buffers for the handshake message and keepalive request and reply messages. Slab pages should not be given to sendpage, but fragments can be. Switch from using sendpage() to using sendmsg() + MSG_SPLICE_PAGES so that sendpage can be phased out. Signed-off-by: David Howells cc: Mark Fasheh cc: Joel Becker cc: Joseph Qi cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: ocfs2-devel@oss.oracle.com cc: netdev@vger.kernel.org --- fs/ocfs2/cluster/tcp.c | 107 ++++++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 49 deletions(-) diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c index aecbd712a00c..e568ad2f34bf 100644 --- a/fs/ocfs2/cluster/tcp.c +++ b/fs/ocfs2/cluster/tcp.c @@ -110,9 +110,6 @@ static struct work_struct o2net_listen_work; static struct o2hb_callback_func o2net_hb_up, o2net_hb_down; #define O2NET_HB_PRI 0x1 -static struct o2net_handshake *o2net_hand; -static struct o2net_msg *o2net_keep_req, *o2net_keep_resp; - static int o2net_sys_err_translations[O2NET_ERR_MAX] = {[O2NET_ERR_NONE] = 0, [O2NET_ERR_NO_HNDLR] = -ENOPROTOOPT, @@ -930,19 +927,22 @@ static int o2net_send_tcp_msg(struct socket *sock, struct kvec *vec, } static void o2net_sendpage(struct o2net_sock_container *sc, - void *kmalloced_virt, - size_t size) + void *virt, size_t size) { struct o2net_node *nn = o2net_nn_from_num(sc->sc_node->nd_num); + struct msghdr msg = {}; + struct bio_vec bv; ssize_t ret; + bvec_set_virt(&bv, virt, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, size); + while (1) { + msg.msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES; mutex_lock(&sc->sc_send_lock); - ret = sc->sc_sock->ops->sendpage(sc->sc_sock, - virt_to_page(kmalloced_virt), - offset_in_page(kmalloced_virt), - size, MSG_DONTWAIT); + ret = sock_sendmsg(sc->sc_sock, &msg); mutex_unlock(&sc->sc_send_lock); + if (ret == size) break; if (ret == (ssize_t)-EAGAIN) { @@ -1168,6 +1168,7 @@ static int o2net_process_message(struct o2net_sock_container *sc, struct o2net_msg *hdr) { struct o2net_node *nn = o2net_nn_from_num(sc->sc_node->nd_num); + struct o2net_msg *keep_resp; int ret = 0, handler_status; enum o2net_system_error syserr; struct o2net_msg_handler *nmh = NULL; @@ -1186,8 +1187,16 @@ static int o2net_process_message(struct o2net_sock_container *sc, be32_to_cpu(hdr->status)); goto out; case O2NET_MSG_KEEP_REQ_MAGIC: - o2net_sendpage(sc, o2net_keep_resp, - sizeof(*o2net_keep_resp)); + keep_resp = page_frag_alloc(NULL, sizeof(*keep_resp), + GFP_KERNEL); + if (!keep_resp) { + ret = -ENOMEM; + goto out; + } + memset(keep_resp, 0, sizeof(*keep_resp)); + keep_resp->magic = cpu_to_be16(O2NET_MSG_KEEP_RESP_MAGIC); + o2net_sendpage(sc, keep_resp, sizeof(*keep_resp)); + folio_put(virt_to_folio(keep_resp)); goto out; case O2NET_MSG_KEEP_RESP_MAGIC: goto out; @@ -1439,15 +1448,22 @@ static void o2net_rx_until_empty(struct work_struct *work) sc_put(sc); } -static void o2net_initialize_handshake(void) +static struct o2net_handshake *o2net_initialize_handshake(void) { - o2net_hand->o2hb_heartbeat_timeout_ms = cpu_to_be32( - O2HB_MAX_WRITE_TIMEOUT_MS); - o2net_hand->o2net_idle_timeout_ms = cpu_to_be32(o2net_idle_timeout()); - o2net_hand->o2net_keepalive_delay_ms = cpu_to_be32( - o2net_keepalive_delay()); - o2net_hand->o2net_reconnect_delay_ms = cpu_to_be32( - o2net_reconnect_delay()); + struct o2net_handshake *hand; + + hand = page_frag_alloc(NULL, sizeof(*hand), GFP_KERNEL); + if (!hand) + return NULL; + + memset(hand, 0, sizeof(*hand)); + hand->protocol_version = cpu_to_be64(O2NET_PROTOCOL_VERSION); + hand->connector_id = cpu_to_be64(1); + hand->o2hb_heartbeat_timeout_ms = cpu_to_be32(O2HB_MAX_WRITE_TIMEOUT_MS); + hand->o2net_idle_timeout_ms = cpu_to_be32(o2net_idle_timeout()); + hand->o2net_keepalive_delay_ms = cpu_to_be32(o2net_keepalive_delay()); + hand->o2net_reconnect_delay_ms = cpu_to_be32(o2net_reconnect_delay()); + return hand; } /* ------------------------------------------------------------ */ @@ -1456,16 +1472,22 @@ static void o2net_initialize_handshake(void) * rx path will see the response and mark the sc valid */ static void o2net_sc_connect_completed(struct work_struct *work) { + struct o2net_handshake *hand; struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_connect_work); + hand = o2net_initialize_handshake(); + if (!hand) + goto out; + mlog(ML_MSG, "sc sending handshake with ver %llu id %llx\n", (unsigned long long)O2NET_PROTOCOL_VERSION, - (unsigned long long)be64_to_cpu(o2net_hand->connector_id)); + (unsigned long long)be64_to_cpu(hand->connector_id)); - o2net_initialize_handshake(); - o2net_sendpage(sc, o2net_hand, sizeof(*o2net_hand)); + o2net_sendpage(sc, hand, sizeof(*hand)); + folio_put(virt_to_folio(hand)); +out: sc_put(sc); } @@ -1475,8 +1497,15 @@ static void o2net_sc_send_keep_req(struct work_struct *work) struct o2net_sock_container *sc = container_of(work, struct o2net_sock_container, sc_keepalive_work.work); + struct o2net_msg *keep_req; - o2net_sendpage(sc, o2net_keep_req, sizeof(*o2net_keep_req)); + keep_req = page_frag_alloc(NULL, sizeof(*keep_req), GFP_KERNEL); + if (keep_req) { + memset(keep_req, 0, sizeof(*keep_req)); + keep_req->magic = cpu_to_be16(O2NET_MSG_KEEP_REQ_MAGIC); + o2net_sendpage(sc, keep_req, sizeof(*keep_req)); + folio_put(virt_to_folio(keep_req)); + } sc_put(sc); } @@ -1780,6 +1809,7 @@ static int o2net_accept_one(struct socket *sock, int *more) struct socket *new_sock = NULL; struct o2nm_node *node = NULL; struct o2nm_node *local_node = NULL; + struct o2net_handshake *hand; struct o2net_sock_container *sc = NULL; struct o2net_node *nn; unsigned int nofs_flag; @@ -1882,8 +1912,11 @@ static int o2net_accept_one(struct socket *sock, int *more) o2net_register_callbacks(sc->sc_sock->sk, sc); o2net_sc_queue_work(sc, &sc->sc_rx_work); - o2net_initialize_handshake(); - o2net_sendpage(sc, o2net_hand, sizeof(*o2net_hand)); + hand = o2net_initialize_handshake(); + if (hand) { + o2net_sendpage(sc, hand, sizeof(*hand)); + folio_put(virt_to_folio(hand)); + } out: if (new_sock) @@ -2090,21 +2123,8 @@ int o2net_init(void) unsigned long i; o2quo_init(); - o2net_debugfs_init(); - o2net_hand = kzalloc(sizeof(struct o2net_handshake), GFP_KERNEL); - o2net_keep_req = kzalloc(sizeof(struct o2net_msg), GFP_KERNEL); - o2net_keep_resp = kzalloc(sizeof(struct o2net_msg), GFP_KERNEL); - if (!o2net_hand || !o2net_keep_req || !o2net_keep_resp) - goto out; - - o2net_hand->protocol_version = cpu_to_be64(O2NET_PROTOCOL_VERSION); - o2net_hand->connector_id = cpu_to_be64(1); - - o2net_keep_req->magic = cpu_to_be16(O2NET_MSG_KEEP_REQ_MAGIC); - o2net_keep_resp->magic = cpu_to_be16(O2NET_MSG_KEEP_RESP_MAGIC); - for (i = 0; i < ARRAY_SIZE(o2net_nodes); i++) { struct o2net_node *nn = o2net_nn_from_num(i); @@ -2122,21 +2142,10 @@ int o2net_init(void) } return 0; - -out: - kfree(o2net_hand); - kfree(o2net_keep_req); - kfree(o2net_keep_resp); - o2net_debugfs_exit(); - o2quo_exit(); - return -ENOMEM; } void o2net_exit(void) { o2quo_exit(); - kfree(o2net_hand); - kfree(o2net_keep_req); - kfree(o2net_keep_resp); o2net_debugfs_exit(); } From patchwork Wed Mar 29 14:13:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 551C6C761AF for ; Wed, 29 Mar 2023 14:16:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E70E7280009; Wed, 29 Mar 2023 10:16:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E20E6280001; Wed, 29 Mar 2023 10:16:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9D7B280009; Wed, 29 Mar 2023 10:16:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D6DAB280001 for ; Wed, 29 Mar 2023 10:16:26 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9FB7812065D for ; Wed, 29 Mar 2023 14:16:26 +0000 (UTC) X-FDA: 80622135972.07.7AB97F8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id CB75218001E for ; Wed, 29 Mar 2023 14:16:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CdvziWM6; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZJvfcCj7beM2l3aL2q8ZqMjrzC72DifYH9bqCAjpsUU=; b=FskvQoimGjykPhjHVLBVFrwOuEQfGB026NhkbQ5ifL2ZoUUB/Pb7zX2txyfUAHuqNTu2jJ ttq8BHuJ3nPuEiHf2C6cQbjf/xzRHEUi6QI5qyMhbCROQDzXGC1r6JbSczXtGq6RA0zXsj A/W4XAPC/OMHOyTb3gOYJy9jO779rrI= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CdvziWM6; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099384; a=rsa-sha256; cv=none; b=4GFn1b3JDjhNN9q67rElzvnnGxVE/r1Q0L2otVYO+mOtPPuOzsGzuPiosfJH5JOEdsWXyN k3FpnnlbP787z+/xBNp5Iwb1pSfqb0gYU/0SfL+M6n/7Vbw5fhkG1v/fZNsLi6j6r6Tdw3 iOTzks+kT06zjV+u/94QKhpE+56Dfk4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099384; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZJvfcCj7beM2l3aL2q8ZqMjrzC72DifYH9bqCAjpsUU=; b=CdvziWM6Ujwnt9+3lB9VivjD2RRkPkxVmQGAdooBZKEy4FJ9mzL9hN3noMnMC6vBCIW2cO b89fUk7YbcB/Fnb31xR6bXF6vPPF/sLaQbO5ePw0K275Fi/x6CCnRtassRBoYh/dDZOJmO LNDaRRJF6dQVUIOSzI92Fw55ie5ZCKE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-478-i-sNnR3eM--eeVz7bh3LKA-1; Wed, 29 Mar 2023 10:16:17 -0400 X-MC-Unique: i-sNnR3eM--eeVz7bh3LKA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 241CE185A790; Wed, 29 Mar 2023 14:16:08 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A585114171BC; Wed, 29 Mar 2023 14:16:05 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Philipp Reisner , Lars Ellenberg , =?utf-8?q?Christoph_B=C3=B6hmwa?= =?utf-8?q?lder?= , drbd-dev@lists.linbit.com, linux-block@vger.kernel.org Subject: [RFC PATCH v2 46/48] drbd: Use sendmsg(MSG_SPLICE_PAGES) rather than sendmsg() Date: Wed, 29 Mar 2023 15:13:52 +0100 Message-Id: <20230329141354.516864-47-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CB75218001E X-Stat-Signature: g5s38h3w16hufprbx141t6t5buztnw8e X-HE-Tag: 1680099384-16345 X-HE-Meta: U2FsdGVkX1+cSUuJFNVQyuyZRhTDx3J3X6jHmbi4s7ZrwwxxpFNb+ZxtK6zn/NteYhzjccCiWjvmnvYU8HoaxpFiiNhf85C4G1IrpNZdUlmrJIF2AqHO4/7X4nE5PGr33sWaEbLUgUNHtX4QyQFDZh8EI+9wOihgjjSjljo7iJh3fSlvSfq/uvS8v2bcGe3/YrW8EKoo76+vrMK9mzjZDdUP93gUnuedybGc7Q9Hg+1EYQLGLoO5yLz41k1YajLng4NNkft3H1zXy6/ikNN2XmjzbQ4OK4yzQqjF1poXqQa3gecQ2b/89R9wRQEi7YM5Nzx1/XXuu5gHaEh3sByXkGW0e67lE+Zt01BULvHu4THOTcj4knN8EuBLVd6spzjjXDGcrQ24J39fdfNLk/NIqHetWMbZS2JNzmlLgibrYMWGDFEefzW7ju+rdPZAJIm9IjwG4pE0smrPqrEiHh29UYIlS2+5NfQLl6NBGykISXy1fweQXrDEeIE6CBDehxfx4NkHDiHFIxTcjX5tm3LKh7/46QKf8Jl+3Ml2tjPTaU+dh7nmYMaILmUpbXnKSOo81v+tqLk2rdMakcqIEPkbTqaDq0Ma7Ram6aoUbADpXgJ+U+1JTWOe+orxutjnJM2f5cu/EQu2NcqXFTzYJABqJqN06Np1fXSMGTTfp0Gc2DDR4WvT1sr97RcqyvOLimj2Mq62OB2SIBDmZ0w6gGVX7gn/TDthqRzMdF3JdfqX7MuwSRmD4wk8CI0graDxF6Hn9hLmCk7yDBAg5Nc1Zc8TVOCiWvCh4TC8CoUUBMcRIrsEorHwOagQORzCyWjhy5LrsOsPYPV8bJyMwQme4DaVS475sYl61N3iCEP4PDuwGQTUNsor1qIgtq7lI5B4oDwouyPpKqhWk/ymhzX7hLOC0ZBw2Qg4khT0VNF5d0EFRw1R9cJHoQr8MjpGOSup8seVjzyyuFOhd6Czh6s5xv6 OQWM3DM2 CuBtc96y06MhgiHvB4ydBcvUiEAUUH8njL/c691be2TW2ci6AjgI1VYn8bmT6WQlI/y9E90LKIz+ga7aFelapEy58RcuBQIIOSScNE3fQgKmBjFEFPMok3nYZiFVfWRgkoEwZoER+H7L9xfhVIKAN6SWqCFU2irYXu7V+vfgn/ty7wSZkl7jTV6DSFFQz8Z9zWHwp3qUMf1AW7T+kRk91gdB2zIc8AqhwHX84EnixlsxUjDqHvKlGyadfx3eDrwmaLWV9BXJLQCT9VMHAux47uACXx/OAx+IDTG1qegBHf9dQ4SA+qVTc2cXGFGof2W8kUTghVgalz52SRuVwYjyQb1DZhLCqR6yq7HEHSvSuBjpPj6sCrYzPyWOXj6rOwDuxXXCSfCpTvJJUTwteCPZSrdp34EfPMEDsuvlmmzQWydNlr5X8r23vqKNaXeVRDNwD3Oml X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() conditionally with MSG_SPLICE_PAGES in _drbd_send_page() rather than calling sendpage() or _drbd_no_send_page(). Signed-off-by: David Howells cc: Philipp Reisner cc: Lars Ellenberg cc: "Christoph Böhmwalder" cc: Jens Axboe cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: drbd-dev@lists.linbit.com cc: linux-block@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/block/drbd/drbd_main.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 2c764f7ee4a7..e5f90abd29b6 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1532,7 +1532,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa int offset, size_t size, unsigned msg_flags) { struct socket *socket = peer_device->connection->data.socket; - int len = size; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = msg_flags, }; int err = -EIO; /* e.g. XFS meta- & log-data is in slab pages, which have a @@ -1541,33 +1542,33 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa * put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ - if (drbd_disable_sendpage || !sendpage_ok(page)) - return _drbd_no_send_page(peer_device, page, offset, size, msg_flags); + if (!drbd_disable_sendpage && sendpage_ok(page)) + msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES; + + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - msg_flags |= MSG_NOSIGNAL; drbd_update_congested(peer_device->connection); do { int sent; - sent = socket->ops->sendpage(socket, page, offset, len, msg_flags); + sent = sock_sendmsg(socket, &msg); if (sent <= 0) { if (sent == -EAGAIN) { if (we_should_drop_the_connection(peer_device->connection, socket)) break; continue; } - drbd_warn(peer_device->device, "%s: size=%d len=%d sent=%d\n", - __func__, (int)size, len, sent); + drbd_warn(peer_device->device, "%s: size=%d len=%zu sent=%d\n", + __func__, (int)size, msg_data_left(&msg), sent); if (sent < 0) err = sent; break; } - len -= sent; - offset += sent; - } while (len > 0 /* THINK && device->cstate >= C_CONNECTED*/); + } while (msg_data_left(&msg) /* THINK && device->cstate >= C_CONNECTED*/); clear_bit(NET_CONGESTED, &peer_device->connection->flags); - if (len == 0) { + if (!msg_data_left(&msg)) { err = 0; peer_device->device->send_cnt += size >> 9; } From patchwork Wed Mar 29 14:13:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E86EFC74A5B for ; Wed, 29 Mar 2023 14:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85605280007; Wed, 29 Mar 2023 10:16:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8062F280001; Wed, 29 Mar 2023 10:16:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D00F280007; Wed, 29 Mar 2023 10:16:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C2D7C280001 for ; Wed, 29 Mar 2023 10:16:23 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 878A5140CA8 for ; Wed, 29 Mar 2023 14:16:23 +0000 (UTC) X-FDA: 80622135846.24.8D94196 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id A479118001F for ; Wed, 29 Mar 2023 14:16:20 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H0kuHtw7; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/kAjj+4zLWTXkqKqP2B/MWL9umh9MFmha+7vx/dBng=; b=Q03Cp+IDThNJEFjUirlDCnHOTcmnijlZsbI7rt7kScMKurRNmpOOAUKqotWz4cbsVPRsxd kNIf4DEcJqqc/X9CkcyzHDAma5quis3t0Je4pYdNVLKHDQ3qSUCNaF3qJ+sCT2y1E6Q80P fj9uekeXQ4xPfheseBsSvaFlzEwfbR8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H0kuHtw7; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099380; a=rsa-sha256; cv=none; b=lS1JpPkraLL7Z2cQAEBKa6eMxXduSLP8mhWt82r8rMuElT/DON4000r0beogDg9/HpOO/k gtFM2ZylyqddERH25wfdyuLwHn7NhyeUBwPYS+0sOZkTqxdP/pUb8I3w66Ine4I7tIOyPY BJxSJm5kZewwx6L4fvAT6VquUVP59j4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q/kAjj+4zLWTXkqKqP2B/MWL9umh9MFmha+7vx/dBng=; b=H0kuHtw7751v7FEL+HauI7ZmSrTkJ6XNfGy4yZFG5PF2ggonHgegD6kyTV4/WhXz6Lg4FB vo/osgxTqB3Du6RTVSLdzf5YfRWcBE84zBM5JP+kQgnYXnzvTLqnkzQPTsmmGqd+6f2sQI zX/VTyK0QwJU2GuPBbD6GWshlkKqL6s= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-270-JSybz5aLMb2BtboAW1enxw-1; Wed, 29 Mar 2023 10:16:18 -0400 X-MC-Unique: JSybz5aLMb2BtboAW1enxw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4406F3C0ED5A; Wed, 29 Mar 2023 14:16:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DC138202701E; Wed, 29 Mar 2023 14:16:08 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Philipp Reisner , Lars Ellenberg , =?utf-8?q?Christoph_B=C3=B6hmwa?= =?utf-8?q?lder?= , drbd-dev@lists.linbit.com, linux-block@vger.kernel.org Subject: [RFC PATCH v2 47/48] drdb: Send an entire bio in a single sendmsg Date: Wed, 29 Mar 2023 15:13:53 +0100 Message-Id: <20230329141354.516864-48-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A479118001F X-Stat-Signature: f95qg93nig85r3nuu8zm819r3ct9me65 X-HE-Tag: 1680099380-953238 X-HE-Meta: U2FsdGVkX1/uN1AH9VC6/KPL4ZcWsXvY9Bljy7AIR41UMwUSju99lIBQXSBUA8RHHMol7Benh418yF/VcQ0JjEndLEY4Fz8phciE4ueUibQPJFFFYcvFPxS0EcjOIlVJSMi7P6GPxnHVlUcoFNmLbUTG1byJFhH+e6Ap2Y+mdewTSj1kjVmlFPkzUqj4NzmK//ZiRZISJCV6xHM7R7XiHAsUcW7L07qe3SEx7fkhKhcaRowQmIihm3EWutPWVxnHJ4RrigyeE3MS/h+nGNzLbzp9APlPnuv9oZnkv9FB8y3qwpkDnYoHo814baNzL3wtTWHCpyZdH2Ie3iJNKOG/Nk1gZYMy+/d/kFmmNEUaQKS0xtvHdgEBCFPF2QaBSCh7dYki2l9lbWHddjyZULIFmYhTw/a6xg2mg4rnOrTEO352L33atAxMjFWPtVUJVrn53TTQOlcs5zL3grVGblVXnNv5tjZcIpyyZzjGT33EsVjIVcoGHK4V1yayAFQKrhVXAF4mI3KIZ+hjnAsJZcV/rL73sXS3QmvHPDqBvwV53Gr5bZTX6QBoJr3Z0tOsXzsfskTk4+py9qK4Fhj/YarcxmO0ci5sjeVp039O8UYE75oguSp0ezd6CQo7x/T3hSszECOz/AYxsEIVBm1dQ1t1kyPUvim9UYDm0PC+yJ/G7+mV9Q95n3SjzB0hI+J8FYtvbzAskTSmxVDu9xUN9a0EoMfkwyUkdEQWiRd5K3MsKmb9iJ6R0rekFnlOtz1gnQaXJpTlKXCLDrG57FZVwMTmmkJcaHQ/GYLfZ0NHvWR+UzfIpx17pG1NXYPYlfJZBdXF27xIORgAkfgRaAB1ww7jb34Ecn4C48AybpOJ9fQFoXNvzF5lFGbTzukgucTjuRgSppX2Z+E4q5teRKcBb1MNq2FVn11OfVhWgB3my32nI0ufFIPXN/b4VQsN/kb8p2moZUFHhC23iRs/Gi2K6n2 uhufmLnW humQCDldbU521/sUdk/VbgF/vkhaZEaEpRvRdaNgs2h6GkatcBjc27dvkXAYTsZ+oXVS3Bd25PTfZueoXU9ZcBYZwTGcEPLU7UhbVVsJldNMOpM2AVd4rzS24+XeZJTb02RMBAJIGagA57egkwDEQeVRLUIQpORwPsuLOyz6Sm+QR5kEgBJTwv/8v3ctMjv3LwUifvUO8OZWgZjhS1X3yFngoFHKZzbnH+ZJwq5XRVDIksJhwj8GFvrd/DIcfVQHNyR1B02j5BhC+l5IYhp30sY3xTvY70iOZQM6/Z1W4N60ec5wIM3VE4Ct/0wc8/OFkNVbD2Ubbv6Bp5+mPPaKYjF+DqVXO5FOlJfIoK9RPb6o8fm8Tb5bmCIjW8HPOeaNcx7CQ5LZKhSsF/CHSsLZNMluZgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since _drdb_sendpage() is now using sendmsg to send the pages rather sendpage, pass the entire bio in one go using a bvec iterator instead of doing it piecemeal. Signed-off-by: David Howells cc: Philipp Reisner cc: Lars Ellenberg cc: "Christoph Böhmwalder" cc: Jens Axboe cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: drbd-dev@lists.linbit.com cc: linux-block@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/block/drbd/drbd_main.c | 77 +++++++++++----------------------- 1 file changed, 25 insertions(+), 52 deletions(-) diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index e5f90abd29b6..ab63d6138407 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1512,28 +1512,15 @@ static void drbd_update_congested(struct drbd_connection *connection) * As a workaround, we disable sendpage on pages * with page_count == 0 or PageSlab. */ -static int _drbd_no_send_page(struct drbd_peer_device *peer_device, struct page *page, - int offset, size_t size, unsigned msg_flags) -{ - struct socket *socket; - void *addr; - int err; - - socket = peer_device->connection->data.socket; - addr = kmap(page) + offset; - err = drbd_send_all(peer_device->connection, socket, addr, size, msg_flags); - kunmap(page); - if (!err) - peer_device->device->send_cnt += size >> 9; - return err; -} - -static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *page, - int offset, size_t size, unsigned msg_flags) +static int _drbd_send_pages(struct drbd_peer_device *peer_device, + struct iov_iter *iter, unsigned msg_flags) { struct socket *socket = peer_device->connection->data.socket; - struct bio_vec bvec; - struct msghdr msg = { .msg_flags = msg_flags, }; + struct msghdr msg = { + .msg_flags = msg_flags | MSG_NOSIGNAL, + .msg_iter = *iter, + }; + size_t size = iov_iter_count(iter); int err = -EIO; /* e.g. XFS meta- & log-data is in slab pages, which have a @@ -1542,11 +1529,8 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa * put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ - if (!drbd_disable_sendpage && sendpage_ok(page)) - msg.msg_flags |= MSG_NOSIGNAL | MSG_SPLICE_PAGES; - - bvec_set_page(&bvec, page, offset, size); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + if (drbd_disable_sendpage) + msg.msg_flags &= ~(MSG_NOSIGNAL | MSG_SPLICE_PAGES); drbd_update_congested(peer_device->connection); do { @@ -1577,39 +1561,22 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa static int _drbd_send_bio(struct drbd_peer_device *peer_device, struct bio *bio) { - struct bio_vec bvec; - struct bvec_iter iter; + struct iov_iter iter; - /* hint all but last page with MSG_MORE */ - bio_for_each_segment(bvec, bio, iter) { - int err; + iov_iter_bvec(&iter, ITER_SOURCE, bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err = _drbd_no_send_page(peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, - bio_iter_last(bvec, iter) - ? 0 : MSG_MORE); - if (err) - return err; - } - return 0; + return _drbd_send_pages(peer_device, &iter, 0); } static int _drbd_send_zc_bio(struct drbd_peer_device *peer_device, struct bio *bio) { - struct bio_vec bvec; - struct bvec_iter iter; + struct iov_iter iter; - /* hint all but last page with MSG_MORE */ - bio_for_each_segment(bvec, bio, iter) { - int err; + iov_iter_bvec(&iter, ITER_SOURCE, bio->bi_io_vec, bio->bi_vcnt, + bio->bi_iter.bi_size); - err = _drbd_send_page(peer_device, bvec.bv_page, - bvec.bv_offset, bvec.bv_len, - bio_iter_last(bvec, iter) ? 0 : MSG_MORE); - if (err) - return err; - } - return 0; + return _drbd_send_pages(peer_device, &iter, MSG_SPLICE_PAGES); } static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, @@ -1621,10 +1588,16 @@ static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, /* hint all but last page with MSG_MORE */ page_chain_for_each(page) { + struct iov_iter iter; + struct bio_vec bvec; unsigned l = min_t(unsigned, len, PAGE_SIZE); - err = _drbd_send_page(peer_device, page, 0, l, - page_chain_next(page) ? MSG_MORE : 0); + bvec_set_page(&bvec, page, 0, l); + iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, l); + + err = _drbd_send_pages(peer_device, &iter, + MSG_SPLICE_PAGES | + (page_chain_next(page) ? MSG_MORE : 0)); if (err) return err; len -= l;