From patchwork Thu May 18 13:06:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A4C8C77B7D for ; Thu, 18 May 2023 13:08:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11D06900006; Thu, 18 May 2023 09:08:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CDD5900003; Thu, 18 May 2023 09:08:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F002E900006; Thu, 18 May 2023 09:08:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E1A40900003 for ; Thu, 18 May 2023 09:08:21 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9600C16080B for ; Thu, 18 May 2023 13:08:21 +0000 (UTC) X-FDA: 80803404402.23.DE626C7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 1FB57A019A for ; Thu, 18 May 2023 13:07:35 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LFKXaUa+; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=xPOxBX78lLkM9Dynm9Z1ICw7XYb8CuojTs4rWS+QSpq9N7TH2tlu6YTEDUOj2q/BJWkQMd BweB0il8EDuHAfx29sQ1YWNFLcsrcObviy1+lzCUkRsnNW498s+Sjs/NOHb3GMK4V64UfL bIkFANrQr+uRG986Z9fpMeGX1Ecj+4s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415256; a=rsa-sha256; cv=none; b=2F9wyKc7mbWy84NFbQ19B+iX7hTg9zEvJPsJNY1F83eZfkMRnJaKqJFwpMvq3bW5wwvnkj vVMiOXXB6ZGXol5+ILlKfvVhBb/aWL/bdCmiwNur5e0cvplbjKBmLNHO3Luv8VqGHBHoS3 Zek+P+ho8xbQYFBGQcRH2DRHlh6KBjU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LFKXaUa+; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=LFKXaUa+GzmCnU/GqjOdkyLcyVFnFKxbLAJ1aFMGywtVajljDfURyoVd5Unf3zDORuDJ85 s3M95VBkcOMHVv0JTeMQnkdk7YOrA/8amQGyDJniUnQipG7dJvTmq0G3Y4CO37j4nNigsq IhKzOJ0UI8HUpTM54jEQRBH3FqHJC48= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-209-DwdtkeJQPymicpdEc0OY1w-1; Thu, 18 May 2023 09:07:24 -0400 X-MC-Unique: DwdtkeJQPymicpdEc0OY1w-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 673338026E8; Thu, 18 May 2023 13:07:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 73135C15BA0; Thu, 18 May 2023 13:07:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn , io-uring@vger.kernel.org Subject: [PATCH net-next v9 01/16] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Thu, 18 May 2023 14:06:58 +0100 Message-Id: <20230518130713.1515729-2-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Stat-Signature: tubtx1guxz4ikwsqeoqzf4j9i7s3m3eb X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 1FB57A019A X-HE-Tag: 1684415255-704053 X-HE-Meta: U2FsdGVkX19228Pb5BQrhYaYdvxIHfCBm5lMc+wUQhGJB11JOuRqYRRo1GvU1iqrhhprGiE+IpjYy54mR+dJEckMi/6+kerlwC6aSdjvbmrHWeew2G/tyXFuhS5U4yCSSV5iPtZAk+4Ij25eu35LdCvSaA8zWvoSFN2U5mGi1kk8lifCx5QtUjhqXIF5IRhk8nNu22Uo98ZNEUXBEBNMt88Rk0YLi3N5WBrwjYAc+/pQ0VEsgodKMBsvDehHMTfz0wJpXhuLH2S5yqkS4mjNTsOJ0uJUi0TmrCgAK5OqAEJMuJk+tR2WoJT0PufdAp2GCaFMHl4pxr6cqmDchnXnT8DJ8SfivcbjfdfKQDpFs2HeXIYxD5hxQm+YxeACNWqxxwIkJrECTw+lE7dbRH5LOzGhZf8F/UriyE9Cb1jbkXeEr025fln//AEb1E+ccPoxWSZnzb1njpLzw0r9mn8gyf8eH9hWHtOs1o8bQzhc06zx/Ou/nXHI4/mcf987OyjPKMQfsBL+rKvtln0Od2+lYPwQHW9snJCkkJHN3Rt7vAIHw5dS6r8NbAi/54AE7MktFC1a1oJk/mSksuNES4mtXiBKHv9+x1IrmpT84AuKm5i/h0zljo7WhOl3r8Xw+Qkp/AFm2TF0nAdT24/uRjkpEyq0+lgsPyyhUFteHnaIvofZT861QyB0/XK7KoriMKRJTDtfAZkP85bKHE2VwwDGgEcU7B5zkJPB/oi/lBbS6I0ehtv0Aj2FfcEsFa5rXBHeZLt2WtcLdI8MrC8ggm6+NeF2i4haky9DWr4nAkcNdWeQm3AqDgg/wG+bCsspwljY4i1eIMTMyTm6FyZGWGLfbdiIDJ+fOlL7mDB0x/Ut69S1ucQK1ZP+qgdP2QF/XqRmDHgJw8rt4/aQGUrusI+zmmmIF3a/uPr7awYkCo2QyQ+DMyB2tELhGnRl4k8Za5RDZqBzExPZbuqbfHLUKVb kYq00hzJ hQSZtvUkfWhag34xKw6z2KnafIZHid+E4vO62i8th4O+V5yoeVh6xBdW+haGf4JBNIRhEyR0bd5WDHAXs9PLGfSykuq2EvztdvN+hzFyzkzt6LSCt1EQFmrurwTgLN6dik8h1GqD9V6TtHrARIBPHrHYyAHP5imcV8yV5/DPZd55e/m5kqsmNyX8sSucKYtavenBOVtixIryOW+N3rZq3e7xtPstV+lNbFB4xWlRL28V75Rbe46z9FiGu0i2D7i/fnEHPA10qcW1rdk1+nA/SvrMocaD98BuDD84Z3oPa4ktCEVXm7cXGtxRMHW1hEH5P9yjPIppiGsKW1s7FPc76ZO2yVvfhu8d+6iqBX2q7AziumIzU5UTvkWg0LkV5aIWevo4yKt9RXThhIbKXacw05NCgiaK0TcIsXdHitwLe9oTMJkuMdVKV0MOWmMuJLoHGmlqfH6IqIXpVpK11BSf5Fl7pyWyC+Hvze8IsbL/n6zoX6YeuFXZN7DE+Og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: io-uring@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #7) - In ____sys_sendmsg(), clear internal flags before setting msg_flags. - Clear internal flags in uring io_send{,_zc}(). include/linux/socket.h | 3 +++ io_uring/net.c | 2 ++ net/socket.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/io_uring/net.c b/io_uring/net.c index 89e839013837..f7cbb3c7a575 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -389,6 +389,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { @@ -1136,6 +1137,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) msg_flags |= MSG_DONTWAIT; if (msg_flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + msg_flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = msg_flags; msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; diff --git a/net/socket.c b/net/socket.c index b7e01d0fe082..3df96e9ba4e2 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2138,6 +2138,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, msg_sys->msg_control = ctl_buf; msg_sys->msg_control_is_user = false; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg_sys->msg_flags = flags; if (sock->file->f_flags & O_NONBLOCK) From patchwork Thu May 18 13:06:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B05CC7EE22 for ; Thu, 18 May 2023 13:08:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 675DA900005; Thu, 18 May 2023 09:08:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 624CF900003; Thu, 18 May 2023 09:08:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4ED25900005; Thu, 18 May 2023 09:08:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3B4E4900003 for ; Thu, 18 May 2023 09:08:05 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B816940854 for ; Thu, 18 May 2023 13:08:04 +0000 (UTC) X-FDA: 80803403688.18.C858155 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 1E07C1C00B1 for ; Thu, 18 May 2023 13:07:36 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Hx88Nxda; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dMBJPtbzh4IjFjJ1HeY2h7yLKLc/0VkfcRg1OdFJq1c=; b=JMGUgrBHHr2QLtHsSrbZ7HRTmtrdq4le75g2XYAZGim+TJCOUFLwWyCWbwCECFZbbKb8uJ SvUQb8gOFZ97KmBpx613blnX6KimIwOgcFAGXmcGAbDyQCPv4w/Eek4gd+xDzJaQvj9dJ9 3gx9uWmMzpNGJ20PywoGt2Ms08c+Xzo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415257; a=rsa-sha256; cv=none; b=ddiBFewtDkXD/NQz6rhavDldi/DkD9NOCw/8p6gaIuP/QQ9rm7fOIYwtGDJ7HXB5uJl2xD tds1ZCZWaUhQj/rshbGslc1kfP46THYsLf3VlZ9DBsqtq8WMquooRMc+QvvvSSXgZGcTd7 t02x1YyJ2ci56HR01aOSs1Bcz1nq400= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Hx88Nxda; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415256; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dMBJPtbzh4IjFjJ1HeY2h7yLKLc/0VkfcRg1OdFJq1c=; b=Hx88NxdaD04TwGu/y130Lm+6O8HY8kfNc02NPnQvbEP/KCcMuzj77mUTqQlQ49oiW9Zkci 2uIgijDFsegjAi0jSTEUB+84/K+92hpIA3jtgLcMMpJyITh82CulHW3efgxavJbo21RakD s/AwHEdvGqYXuTiaAz8AD9fpgc7p7L4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-659-VyatmphiPrqaOCvVPK9LmQ-1; Thu, 18 May 2023 09:07:29 -0400 X-MC-Unique: VyatmphiPrqaOCvVPK9LmQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D8A64185A7A4; Thu, 18 May 2023 13:07:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3AAD02026DFD; Thu, 18 May 2023 13:07:25 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 02/16] net: Pass max frags into skb_append_pagefrags() Date: Thu, 18 May 2023 14:06:59 +0100 Message-Id: <20230518130713.1515729-3-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 1E07C1C00B1 X-Stat-Signature: zu8f1f9q7xqeyyhdi5grmz6nofuz511e X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1684415256-154934 X-HE-Meta: U2FsdGVkX18sr+gNf66FDRtVzYMtgtzvQBhtV3SR1X8sZuVtisMPlSKiGVkXsD3/SYXdVp/ZoZG8WzrgAfwDoLVfU9Y8S8d/vjpiXABy36+CPQa4QgbE2wsNYilt7ll7U/r0sUltALdj0V7MPtH7JSBtuuC5iaY6Vc1WNlcT2bh2UuA2P/MOTFbBqoVodRAWYyJU5AQUmPCm1Q9Ds2Q0VUAjYRLmUPXBIlN4ThqW7SIpu+riKm868fcbu9ogfhPFC1Q50j925JL/JJMtplERBVgPK+SnS+Zhm2lJGcJ97X9IzrS+KzuvdFl7t6CdgZVUZiU0XXomc0HUcPW2Vv+lv869e54SYhWGxbD9aGFGkuF33JzyYNQ+MQhZah1bRMaeAeZaLyT5EUkqgpmKP2bGW/UNX5HIwjwQpmFpAhHxXf+nYyq0mk+7WhpnxfzDTqj6Marn7pED5nhaKKq57KVd0CZdtiKzgMQBLCMpzqxonTf6jeEOxtreKAC65o/vh4j5fs342k7Rx/0kIiGxndQhAnQfRpnDlzpMERiqDtl+m+L7uFRsvqRGiKJzETjJjtYVkwrK4GLQuZgidi7Y0iheHRlwz8QppMP3M+vFNLmZ4DpRSosMscVVxO5MK3772bXWI8lrGDrABin3YCEH5O9CKzHXTAt5hYQqoOVvd2c85YQN/C4WRVU8ENpdSjV6h/y0A2ejKkkR7byIApv4l7E0NgfUr51n+qfI410+u/Hr/Bli3PIHzPF5bFOZwbyPXWqm7DcbbTrrLLqlLCpkIiGlje6oXqCyH9HYmCs6APx/FWS1aOIkyE15cvSuxhFWPiNt2uy/RBMVAIfucJVaAIvgv1N2bBJifVTIZDkxhtL1jomaG2uAzidqsiIpwg+8BlrvnbisE/4SCgQ6GuAiyv8Pn3i6VI8Rk63y3SSNQ62g0DplFSVgClAKrbZmHNgf+NArobP8V9mo3YyioXcTH4K mZD9XgsQ /RNKCsJtwxow5ABufrAIr+0xTlyv9AY4WvAyPYMhFJONrCehFRQCtwJIk5/xve40zDGzvj/oTbodpoo6N7aiIvzWDi4F+QYjktU69fmyugitZW9V9jjng1GcRe0+w9LKuW4P9WaLsEg4Y4Ke0gkSYIN9gCX7FJgC+EuxFcogdllKlYWD7yNb7gCrdJJ+mJw/YdRt6IYZFesxD86PgDQr+X2s8AeLEeGi9OjL1up2veaf2kXvdkABYi7XRUiXT6Wm/WVO7idClVNvNlGhDE1JjRo4KVNvVaGMAxIABcBJZvi64r5Vm2yMkP3AQ7evIZiqaqIsHK/3q2CyxGJ8q7aNWC9VpLpUq/GIdZjVoX7A9xDmbzIcRB/Fufb3iRl+MFp+uOOwL2rJY+trsZ1UnT6vLUEN7bkmGuqnbed5S8SCZartGCI6sUYZNB8XS8vz3unnkig9D7ZEn7DPoUewUV+E1khu+X+WChtc3pBhzQrUt6r1D9Q7jchwU5khC3RY2iTWpnliBi3iaDpfP7CI+ef2syTfZpoGDK7WOvGEa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pass the maximum number of fragments into skb_append_pagefrags() rather than using MAX_SKB_FRAGS so that it can be used from code that wants to specify sysctl_max_skb_frags. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/ip_output.c | 3 ++- net/unix/af_unix.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 8cff3d817131..15011408c47c 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1383,7 +1383,7 @@ static inline int skb_pad(struct sk_buff *skb, int pad) #define dev_kfree_skb(a) consume_skb(a) int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size); + int offset, size_t size, size_t max_frags); struct skb_seq_state { __u32 lower_offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6724a84ebb09..7f53dcb26ad3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4188,13 +4188,13 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from, EXPORT_SYMBOL(skb_find_text); int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size) + int offset, size_t size, size_t max_frags) { int i = skb_shinfo(skb)->nr_frags; if (skb_can_coalesce(skb, i, page, offset)) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - } else if (i < MAX_SKB_FRAGS) { + } else if (i < max_frags) { skb_zcopy_downgrade_managed(skb); get_page(page); skb_fill_page_desc_noacc(skb, i, page, offset, size); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 61892268e8a6..52fc840898d8 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1450,7 +1450,8 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, if (len > size) len = size; - if (skb_append_pagefrags(skb, page, offset, len)) { + if (skb_append_pagefrags(skb, page, offset, len, + MAX_SKB_FRAGS)) { err = -EMSGSIZE; goto error; } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index cc695c9f09ec..dd55506b4632 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2349,7 +2349,7 @@ static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, newskb = NULL; } - if (skb_append_pagefrags(skb, page, offset, size)) { + if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { tail = skb; goto alloc_skb; } From patchwork Thu May 18 13:07:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EEC2C77B7D for ; Thu, 18 May 2023 13:08:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A40C90000A; Thu, 18 May 2023 09:08:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25476900003; Thu, 18 May 2023 09:08:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11BEE90000A; Thu, 18 May 2023 09:08:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F40A1900003 for ; Thu, 18 May 2023 09:08:53 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B03F01604CD for ; Thu, 18 May 2023 13:08:53 +0000 (UTC) X-FDA: 80803405746.10.B9415DF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 109AC40151 for ; Thu, 18 May 2023 13:07:39 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hSQ5xUd1; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fl+2zC2cOWTcM7FjMckfIDo5NeTUJ6H1d1XKuMzp1Mk=; b=i7cylSPJs0tT1ctblw/QZUm2SHLHnXsbhUit7Z44LkGaZWXTViMOJN9VsxHoRsCUtBoh/Y +uYnN+/Y6AkuRB5LWrWsHwxth8ZLiG6qTFb78Nw5Q/2+r9teGTmkVu5jRRFb2RmhrK8ya5 6Mzsud/lQZiTd4fFSFL9NCMlQIew/2g= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hSQ5xUd1; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415260; a=rsa-sha256; cv=none; b=6XJq0BR0Ww2etKVEqdMzHmMgXY9MSkUdeAINcmOekx/xx/8rHki+Z0OH02SbkqbLznJ//7 Zr/VAYP2UWOgSIFE+pDCpnAL32JlAEJUYhUDBd/wm5vxlDYa/AW1r0i229DtRyKiI2rxp6 yuJbkfQKpR+8bTn1iEyZs0IzWmbnA48= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fl+2zC2cOWTcM7FjMckfIDo5NeTUJ6H1d1XKuMzp1Mk=; b=hSQ5xUd1oSPxYpif/LNkgr4L+LzDKL3aJsjg/SQjtuYqEj1xuDQ2YEJu0Luq5zAPxiRMsk IP5ZYDp9fmHW53mdx/fMCNjcOEPzqTg4tTpb1dtGCldLSlvkQEmpPEFWXoxg3BOP+IUHLT 3wbro7XUpgUeYjS2tU5Iho6FFVxv35M= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-595-Nim3qSxMPQ2L4dUXo6bQVg-1; Thu, 18 May 2023 09:07:34 -0400 X-MC-Unique: Nim3qSxMPQ2L4dUXo6bQVg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D7F9E28237C8; Thu, 18 May 2023 13:07:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 650961121314; Thu, 18 May 2023 13:07:30 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 03/16] net: Add a function to splice pages into an skbuff for MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:00 +0100 Message-Id: <20230518130713.1515729-4-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Stat-Signature: cf1mfwh4cihjur31btxf54fy6j66n6xz X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 109AC40151 X-Rspam-User: X-HE-Tag: 1684415259-113674 X-HE-Meta: U2FsdGVkX1+AXxC6FfiWmYv70HczWWTt/bDzhtWhZ8B4+Z/D4HSxHDj/9O6wYkyGiT5LZjHWno6i+OJx+rH8UUz2FSCfz8OjQD5Mez3LS3OuWEBWIUf+NF4idL2R9NObT5+7Kh2eovRIjRrvg9Y1Q4ykFz+7wwu2Y5Yd/XWjZYD9PA72Ql76c7wmeH6P+nwPWjP1f9C6lDazCZAGa5HZc7XkgBHlbZYs8Zr3KiutlJVXJbsGrayZZXZphVWsCHKE3BgHh6fDpz1K6QvNctnedYNMN9d5unXmJLhyBY+J6n1ssce7rbDnRwH/1H3xnh5BgR/oGZCip1xd/Hkp1ysDVsm8f8Hwg7cgFw2WFRFZdiVZxGYT5TBiLFuJ8cOjavJh4yn7IgNpPf2zM/AOIEudN8V5yYrKkgOhd2OhcSlzOEQ6MZ0SdpyJuELza7i0fwV5jNYRGJcrWKQZvwIS2PqJVepf3JefiC2OGApTgZ5UE0w/CxMmVltNbzeYP9AXxZxBw3vmZZ3VnCfM6uJ5EvKf03UWixUTlY8wBywArMHKHr8/31FYvyBpkDy+pyA3/wFFBh1mp/i0ByGFwSZPpa82YN2mbsZjBVSUVenTFiEzSjN/OaQcmF0jcfqIMmX66feDdo7yHdN0xavXHd+ijZxHPPkpbS/4hfqTQDScd3NY/ua7WtYwNdVeVpCQCyBELkYmQ8IomJWvCY3UoX68lz2y3i4JIU1I8rgf/NgTKffe9mO2xBYhO2gc5zu1RgwhYiR0mATzKLBtRJHNY0SMQynjAVyB/+4MAqQz26TCmfl0SehJrN3RD/cJHwZWEOzHWNmHwuyM75wLI1vT5W8EVBJDOAhYCCWZkvPbUtGtBK3Px9r6DFtIfgfpbKzTkDTyhvugejMQ6zx0grjqCIExnJUxoG1EYvGxag1dX8/3oqLdrVF4oDq3HKXOHueO8FP19sIwUgz+1G5Wq1Aky4aRnsM AluE6evn TIBUqBjHJ+BawXIDdzbAFlaiycI72PmerOAwidLbUCN5bsk+UILvGD810zfHgF9iayfz+W+p7SZGfYm9VRhc9MvaCCLys1+Ei9j0kKoRvqEqc5Xb3Xhkhhhwr6MteBiQFBoBCfMH2DSbCoQnlDdRYpj3aMaHyJjAAzWQ1KsVq7l++QdaK0XFCYbzg8ZE+EKEJ9nqETWDBZo3PrWRyaynDP+BAElLGbE4nfhGr3gn+BMmn0ZyGPYIC29FXF9NihHyOlMvA1jZA0NuzPl4fUTy32tcIaFYBVsi3OItGW3N/fGfbEf6XbTNwFAC3KTqxihLzJXvQy0JijuWDTIqH4ZPHIgQsVicePvvQJX8SZTcmbOgDbbHaWkVGbxMZTDCRYe5+SwJq8eBXt+PcQnMnYJutDCGfHsq/oRUi245V8nOloSamPq4fbLVgFAhlIuCuXN+u99m8G6C8bEuxF+YMPGk0+/SDz3PqNolKIK0ir8t5c/QjUwQKbk3rYbm5OCFL8tfur1AYPoKaOWx85yZOVpqKUcCyb0AkdqudgpJIlrx9M7tS9dA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a function to handle MSG_SPLICE_PAGES being passed internally to sendmsg(). Pages are spliced into the given socket buffer if possible and copied in if not (e.g. they're slab pages or have a zero refcount). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Al Viro cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #8) - Order local variables in reverse xmas tree order. - Remove duplicate coalescence check. - Warn if sendpage_ok() fails. ver #7) - Export function. - Never copy data, return -EIO if sendpage_ok() returns false. include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 88 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 15011408c47c..1b2ebf6113e0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5097,5 +5097,8 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7f53dcb26ad3..f4a5b51aed22 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6892,3 +6892,91 @@ nodefer: __kfree_skb(skb); if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } + +static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, + size_t offset, size_t len) +{ + const char *kaddr; + __wsum csum; + + kaddr = kmap_local_page(page); + csum = csum_partial(kaddr + offset, len, 0); + kunmap_local(kaddr); + skb->csum = csum_block_add(skb->csum, csum, skb->len); +} + +/** + * skb_splice_from_iter - Splice (or copy) pages to skbuff + * @skb: The buffer to add pages to + * @iter: Iterator representing the pages to be added + * @maxsize: Maximum amount of pages to be added + * @gfp: Allocation flags + * + * This is a common helper function for supporting MSG_SPLICE_PAGES. It + * extracts pages from an iterator and adds them to the socket buffer if + * possible, copying them to fragments if not possible (such as if they're slab + * pages). + * + * Returns the amount of data spliced/copied or -EMSGSIZE if there's + * insufficient space in the buffer to transfer anything. + */ +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) +{ + size_t frag_limit = READ_ONCE(sysctl_max_skb_frags); + struct page *pages[8], **ppages = pages; + ssize_t spliced = 0, ret = 0; + unsigned int i; + + while (iter->count > 0) { + ssize_t space, nr; + size_t off, len; + + ret = -EMSGSIZE; + space = frag_limit - skb_shinfo(skb)->nr_frags; + if (space < 0) + break; + + /* We might be able to coalesce without increasing nr_frags */ + nr = clamp_t(size_t, space, 1, ARRAY_SIZE(pages)); + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + ret = len ?: -EIO; + break; + } + + i = 0; + do { + struct page *page = pages[i++]; + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + ret = -EIO; + if (WARN_ON_ONCE(!sendpage_ok(page))) + goto out; + + ret = skb_append_pagefrags(skb, page, off, part, + frag_limit); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + + if (skb->ip_summed == CHECKSUM_NONE) + skb_splice_csum_page(skb, page, off, part); + + off = 0; + spliced += part; + maxsize -= part; + len -= part; + } while (len > 0); + + if (maxsize <= 0) + break; + } + +out: + skb_len_add(skb, spliced); + return spliced ?: ret; +} +EXPORT_SYMBOL(skb_splice_from_iter); From patchwork Thu May 18 13:07:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B17DEC77B7D for ; Thu, 18 May 2023 13:09:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A00490000F; Thu, 18 May 2023 09:09:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 451CF900003; Thu, 18 May 2023 09:09:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 317C190000F; Thu, 18 May 2023 09:09:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 20027900003 for ; Thu, 18 May 2023 09:09:21 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E82E7A07AD for ; Thu, 18 May 2023 13:09:20 +0000 (UTC) X-FDA: 80803406880.05.A17567D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 8B368C00A8 for ; Thu, 18 May 2023 13:07:43 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=E6UpDNxT; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415263; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CbEUQCnChP/PzTidzzQWL6iO0yyE2e0IQp2l4Z4U9vY=; b=fg36yvFiTfWSr9+j5ZAt2RFNRNBOYd+xfoeroU7y76bigKZXhzrth/exeW9dPZ4Mw1gVdB fTxm4MJ7tYt+Nmz9vWc/bTAt0kpXa/2AvjKfq7iMYl77H5k8gFfTsULNLGCIGw6HF6UWE/ Ym/LrTXEWNGkX57x3BaYmv65PwV+GhQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415263; a=rsa-sha256; cv=none; b=DjyTZkBi8gEMqyLNKApaCxltjSdxFJ+MRJldm17dvIh+yDwxjOflzN+7LL6+BewhZyBmE/ gE4ODyyzZKlUFPG8BXitEkpMT/dpykqY66MaRR41dUGZiIsUhzA/1fPyIYSH1Iw6B5ZW8t 9DUwocwwuNOAFQQ9wyrI1JI/4YWAFkY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=E6UpDNxT; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415262; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CbEUQCnChP/PzTidzzQWL6iO0yyE2e0IQp2l4Z4U9vY=; b=E6UpDNxTcfoSZ8GF1CTenzwY/PxDN7dSOyX3xIkTVYZU7+yz3XrYHYiIYcCQgoiExz40Yc oCSrF0HPulJhOBBFHaXfP3eWAtctNfxo06m6N8iXOp1FcPlzbXsvYS83jVGczQjtquh0Sp ted3XVUU+AdNn+CMWE8HevBfTYmtaNg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-355-v9bQtabbOrGx9iDPjoFLmg-1; Thu, 18 May 2023 09:07:39 -0400 X-MC-Unique: v9bQtabbOrGx9iDPjoFLmg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 322A184AF35; Thu, 18 May 2023 13:07:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id B53DA40C6EC4; Thu, 18 May 2023 13:07:34 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 04/16] tcp: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:01 +0100 Message-Id: <20230518130713.1515729-5-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: jnqyyu939fpbtuc1bmeb6x7hhwefu8r3 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 8B368C00A8 X-HE-Tag: 1684415263-320880 X-HE-Meta: U2FsdGVkX18RlOtAiCGqqSjAYwI/oGVyPZjIE1o0z4x4yHFjVxcCsfNoiq1EesQPJxHw6BRYECJI+smZdE0yYReFZAn0IktD/9QDee0207Z4U5kEHtzStG8UAb2u3ZB2GJq/iGVjCThorwSDIqftNqr9NGw7C/Ma3YbwgEP7bqgmDr5wjgYyH1cLKfrdZ0XwvBMpbeXgGLUxc3GB0yB5LlU4d5Q4so4pUK7CasJdZ5bpYKiGc0pCSkJOREhYvXF3Ff6r2qLiqEXrFd70btzATGfDyaKNvxZ8B8ICYoddNYsMd6uFhgmVf4/P4ZK+H/hiaSzfqeOUbhco5Yd8U8YbMBMLf3eV4xqWd9xCBFrpv4SOi7km0cqxot6Tpl/evAPcCtJ9Z84Kb/HxoYSSPT3OOgVGW/VVU5wNWgzjKWeejQw5ZFPXnd/k1qEXQ1aFmLnmJMv2EnAhoLHUJwu09n+gHXt+lHDXlC6Po+W2Ol92Lnbtpwl9rgcyAH2jMoQfYtY84EG06xLRisoP4vlqlioqePr7vkE3eZKuVN2jn408SvErpyojkPggDezcxCZM//8Xtn5Sp4y1CfqofHqyFdxbqR+tkPfWowjkxsxT8BN9CFqOa2bcEB/rHRmQECmuGI2ra2g4M9nE0wJSSYymBLcnWFUe+DxH/yMCsa8klWlPtuyn6Y+F3wXbub0nfCwPxF1od5i/FFOgTUiSrkrjfvoYyrAYguD2P9Psszak+QP4iVc5EoCbcSSFjbv1xx0/DCj1WWL8tOt2Xe7rqPGWNbdSV1xmD62N5+94eLAv3B9bW9XkiYsIxUUZ2LLfQSBoXY1mX6nkICKXRsDH56mKks7ces7yNNj02NcopdUHgYXa+IXP3ZH+sQCValia1D+8y9tjbmR6gp+aDhWCYJGBMx0RTxGkfWp+6GB4Fa0bEwRZRSOAgv1JaBNyWKiTgZs7ZZ1oSrOpgTdrrgS5ZULSINa KUufY4i7 IUCeKXeKXO9G9d4l/IIL24wU/xEIL5tGM2/Zij9mbGVyRo9FpeFzPaExYf1auKIUq0p6xLPVKLQVO3PDdIa0YGIfrE2x8hJlW29yV8NdHT11S76SZab78ULkz46WC1u+1TUx4/ZLaTBJXdKqsAd1l/5xrR1w91ogveNWVbzZJL1qqSsOKah6rR5iXUcOvHpKkVq2CswDthGCmDWC+R5PsNr3hY8x3TVE2dLUfCABZ227/vT+tAyvNk29uqrL+oZZjpoqp+Z6kvxZPujWL2ylXpWPrkIQDWPe31xvEzFE1gqz5LPd2QFw0Hrz6FUuJPkaC/zG+q+ddnXPHyh8YSBsRmhwsO4IPbQQki7Eq/j0f7yov5VqDC9XHkYZyfvUAcYDL6E8Bl0wRrUDFLJIcz3ytnSxgK/pwh24PDiOF68c7IWhxxWte4RkeaOU0rb8/n6k4Z01NbXaZ52sAxB2vxfJSk/kUTZewW2Gu5ebTdl5/x2c8ygv8/6mqYVPZQA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced or copied (if it cannot be spliced) from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #9) - Fix a merge conflict with commit eea96a3e2c909. ver #7) - Missed a "zc = 1" in tcp_sendmsg_locked(). ver #6) - Set zc to 0/MSG_ZEROCOPY/MSG_SPLICE_PAGES rather than 0/1/2. - Use common helper. net/ipv4/tcp.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3d18e295bb2f..2d61150d01f1 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1223,7 +1223,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1231,7 +1231,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if ((flags & MSG_ZEROCOPY) && size) { if (msg->msg_ubuf) { uarg = msg->msg_ubuf; - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { skb = tcp_write_queue_tail(sk); uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); @@ -1239,10 +1240,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_SPLICE_PAGES; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1305,7 +1310,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1346,7 +1351,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1391,7 +1396,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == MSG_ZEROCOPY) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1412,6 +1417,30 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == MSG_SPLICE_PAGES) { + /* Splice in data if we can; copy if we can't. */ + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) { + if (err == -EMSGSIZE) { + tcp_mark_push(tp, skb); + goto new_segment; + } + goto do_error; + } + copy = err; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); } if (!copied) From patchwork Thu May 18 13:07:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39597C77B7D for ; Thu, 18 May 2023 13:07:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 933D2900004; Thu, 18 May 2023 09:07:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E50B900003; Thu, 18 May 2023 09:07:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AC06900004; Thu, 18 May 2023 09:07:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6D66F900003 for ; Thu, 18 May 2023 09:07:58 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2FA301407B8 for ; Thu, 18 May 2023 13:07:58 +0000 (UTC) X-FDA: 80803403436.13.6764003 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id E47AB80013 for ; Thu, 18 May 2023 13:07:55 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AUJSO9xu; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415276; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=79TjdqGEZWzMR9284kh0SsbKnhgq/mMmLDGgzzPhuR4=; b=E5yPuYWum8+frWCvZ5r3Udi1EpIieXNK538t2oPMryBBYX2sQqXITckJ2Qw6odPxOWf0wX esRLK9qfOHZhfCcFjjBYp4i7QW01U07yvRMoO5gxekIDbzbjvNFO8AINk5rzW+pzTRTiQr LXkAl7RnXY8OFV42UEat/I5y5+RG2rE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AUJSO9xu; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415276; a=rsa-sha256; cv=none; b=j3BVqCtYRuOks58JU1HXAeXirMHezP0TM6Vs7nU7YHry2zzs+jTL7qwPvYTL8gxDM3LSZz YUezmZcLGFuFUmcQfJv0CF4ikrbDWLQtO0LiPD82Sgad7RW8nXBvJQX0uNfwuqoRIPVPW5 QUoJUQ639R1ydonAt6y64MZzmvidFMM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415275; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=79TjdqGEZWzMR9284kh0SsbKnhgq/mMmLDGgzzPhuR4=; b=AUJSO9xu7OrroOvpYtiQ6O95qgc38erYpIwqJMm5RXw5yYCNKaOWEsoJqB/RIdGCJ1wyTH 1cc3hmEzj7gdUY93NnezMSvymDr+79KOAC4amVrOMvGbP0WbL7exFVEI6j2IVP8XrAAWAx gT4u9eDy0qsF0ys+maTn8zVstwslEEk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-247-KanpYTY4OHmNeX6aG_XJaQ-1; Thu, 18 May 2023 09:07:44 -0400 X-MC-Unique: KanpYTY4OHmNeX6aG_XJaQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 65A833815F77; Thu, 18 May 2023 13:07:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 51BE440C6EC4; Thu, 18 May 2023 13:07:40 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 05/16] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:02 +0100 Message-Id: <20230518130713.1515729-6-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: r4n46xmr3haxf35447ouyisxsm39mimg X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E47AB80013 X-Rspam-User: X-HE-Tag: 1684415275-824579 X-HE-Meta: U2FsdGVkX1/Ro3DnSCAgbUnK4z6HEfkyBQWlhSimdyD4VfBQqTNfXiNeJqUDunzo/8pRbhMjz2XBqM6OFJs5NRbWmVJCISsxGlDkIVvh1VAqh7hZGCv5zEGgsk0OktNEBGYqndPeMCIZA76HCjqpLW2LlhXVFPzZIX7RPNpmnigb0jmi4nKRiSjZlD4vNAbAlJyl3O5bstUOQPtuJbm1qcHKvxfNl4ls50SWPQcEM5D8OcT2fmdW+O2SIK9mX0y3+RZB2v6nse4Io+hfvFfVslyqcy6LBUqifnyZwJj9lLralig4JT0nb4qtKqxvanpHi+3mZWaD6HjpIIxEZ6vTF9kbZ11SVKwUh0oM1MixETvUSKZNyD3V6zCEz/Wml/BRtbprsMjnj44OVvJFclFrx347HVK26QysWuwG4MWtek7Oi22vkaIAQqwF4TT4yNWLSiJpVXRlEHqeNcLhT0+G+X1lZlqh60+DmoWL1Dh6SU6FAfS0GrRhqxATyaC3jyIrS7rBUI9c/jaYWYynzcyG1YmAng2Hyxc38JOPkKkCATHpMItCgMXYkIzOeWY/qKVI9fOhAjF0IKdrwi5oJbI5tuXVYS33hN21Ln1tXvR3zWl+ZSCZfSPvJKfNbzi7PPemRtzm+af6kVq2NQSfxjJfp7NKXYiGDt79PPKovHdVDR9G+BQgHDi17sbXOzpjwQeqRl0FQlN4bmUnrCZbHGYdyvZHc7psNuw198hf+jNXyguMNADSJFbHnSUfWNfrUHvp9RyqlD8W/KoN4UxAHa214zm9QrIsvjQas9OVQFBG2vEnZ4zBDWQOt+OUo8YhRNi14gIUbegG+ETT19fQC8+4Oyd51y/EeElnBO+VzoATyQhynI9/k2g5LyjB9Od5s4anYr9hOtYKfcVca+OtntJy8zmtJZ+UHRSTqz/Xw9FhR/yDbTiTSXs0Y72gEESmK7jS7ikWiPM9ikQW6YFKO4a Gl0c5Uzb /S0GydXgKOpvsNikDJAqPdtSc9fxALGB2BMM816W1rCNOJavuCjtR1oZU5oLfq71FFjZR1ia9WNtuaxc8n43HO09YAn6sf/W3Y+kbrjC0kCN8M5sDzuhSBIRV3UyxCvUZSQ4jj4/qYbhVYSWduk3miZLKW+5kRg7ogBOzPcdLlKYRXNwF5M554kx2S2YOVqciZU2i6G08hHdwS9R11YgTfnVEdfuPxflTCJA6jdKDRQ5sp2mltTdHBBGbJIYoskoC+CvZpjTCwHIyW8yKTp1na8JsTOfNVA+uRVDsSOz6tFZfPd4VW/k7oSWAuunJ55slKFM3/rX9rqafnSxSgHJC2LQuONPOIanuV5/c3hlIVcKk5NJHtVoiWFXp9lNVggYV9bLHcXDv4QD4hYUt5GWFsbJgm7l7TFiqvtiMd2tcNaRVwlkr1Dkbk6e5R4CmfpfhXVMTWh9Fnt8lny+/Wf1qyNNOnHMimcOHjUlrCFVq1VOfXRWDBS0OwPfkfQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 2d61150d01f1..f3a0c02678e0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,163 +974,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Thu May 18 13:07:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A81FC77B7A for ; Thu, 18 May 2023 13:09:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB5DE900010; Thu, 18 May 2023 09:09:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C199D900003; Thu, 18 May 2023 09:09:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB8EA900010; Thu, 18 May 2023 09:09:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 94906900003 for ; Thu, 18 May 2023 09:09:37 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 658CF40848 for ; Thu, 18 May 2023 13:09:37 +0000 (UTC) X-FDA: 80803407594.22.D368405 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 371EE14018E for ; Thu, 18 May 2023 13:07:58 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EkC7VsvO; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415279; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=ouJcqTk/sxBVLvtkvRj9aJ/qE1dB3wHYtJeurIuARHOU35FmrIPz8BTyw5jzy/ndrkCaba 4b7ZqLorLLcjdXgiCayacTFbUgC6tcfR/hmuZxK5S7JHYoDbT/+P0ZYzvdcOk1qy+6GsO6 SkNzsPlde3jB2bv3LwaVhNimy70rZ4g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415279; a=rsa-sha256; cv=none; b=H3gDIE7e4+XUT1Eldv6xPoLkAWaRBrv7Ad7EicDswiIpDIL8F8LFlgEtjzMfq37w+57d/O fezVeZbgeCTAsB0pRWbATfGNuKnaQP4yU2aiH3C6vSEUvrDfURuQHFoEibf3GUqD96AUSO bjeW6tXYFoYoyK9nSwQC0a5vqVrVHt4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EkC7VsvO; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=EkC7VsvOGxY/5VzhnL5xHTuZWV1Yjk5RIv5FqtbIFQtBncdSf41MgsMBNaIGyCZf842Rbf Gd0chgxEBOQhXDC3e9WjL1gKVYQgs6GUqUCzXAZf0vvgGKHUfrb2QLRjZnrNGSKEVEXOtA KqqMkRBZL1spgHaE3QwUTe4ZQe01Ja8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-209-s4xticGiMR-62gp7XbuBGA-1; Thu, 18 May 2023 09:07:52 -0400 X-MC-Unique: s4xticGiMR-62gp7XbuBGA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 38B6985C07A; Thu, 18 May 2023 13:07:48 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 155802026D25; Thu, 18 May 2023 13:07:43 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH net-next v9 06/16] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Thu, 18 May 2023 14:07:03 +0100 Message-Id: <20230518130713.1515729-7-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: re7rij8ojs3pre5ezdxufjciaxot167m X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 371EE14018E X-Rspam-User: X-HE-Tag: 1684415278-996464 X-HE-Meta: U2FsdGVkX18R8EjGDLZx7jKVzRSmwfI5MsqyBdJNfzOhhLHqmGF+hYKA5i+gF27iHoTIAP9cD2R3iAbpdiiJUeObxu9B0lLfSsINxm7fmX3Z1+f68lXLbGYKFsqOmW7FeI1IpvlkJ15McYtUD9yn3feVUfJ8WgvO/zGpjQFYRNNDAAP5kt9DFxcJtgmhanQ5sO+EPc0LoBuMInhE4fTmkOLrhfKC9h5ctYCywwL9PFWHrwEbXg5sTcrF/9pHOWAv7FaKllsNPagFa73gKVCWPUG+cOmU3/xXFY8N5PdxDqmDEPfoiwtPQNKNKKVbrc1KCgXDTGnMFyGd0TTS0NqUCh9sMi8mJpy52EzyDU8KgWBlc2iEFmmxSsnTOHlD/ndHbzuq52KLgd8DbIY7/0Nk4CGCHM2NGfgrZSpGYr+hXgu0IrhmT/Mhe6D6ZrapMMXXynfF5EBoXW1dnvkf84Gjm2uE1Fou0WkUSkUoPPmWmFLwBQKucoajdEybB/QkN94ZkPZ8vn5hPt1V2/IVspRcd9Ea7ZTyckggolDVVFoezfsky2yTNhmo8C2rAhNsBn+w1jD2A6m0HXybZ7UxUKjuylZa8Iiu0gkWzGQAlqmPmzc+h1EJlj2QG8exyPcIev1y6e6bRsxk54VezN8b5s/dg+dCNXUu5Ki5sq4YjXKm6372Gq5dYh7HFXRITACJ2/xKccK2d2H5BkyNWSm4NdeDYphYrtAjzFBRYNydHj+84fGNR7gkCB3YNuyBCUEDGoKAcKbcSzCApAQOw4EO+pANdZ2hPTceHt8D7KmmJyptfI9ST2bTOTvrwAWcFs0JQiQKA6ePNFwuMMsTfh3ox9VAfHTugST3Qt4Cu4U6zDkwKrpil1Lnx8Uos8wvgIRfKXOoSQGuspyTkeT4QGvIok0WswRVWetW7rQtHB4PhotBj8f7dTrMjjkv2cQ30bN8yBsiAQq0o5pRY/fW4kJ6awX nimjWjav CfsNr7+ffN2LKNq7zYM3OdvjYiBblNz4zINZoOnSZPnxWf9N2P5yUB+MlS86/WTpkZcvVg/NhPmZvm9MtHNseP28A4Fv0lVokrYi9IauT33mU/zEOKavM+9Yu1IbBBc+KVrB6XoyAU0VJRvPwguOTdZ5NQ7T7L0wEdfmE0rlBusLfi55Vpydtxv+5zqUNjGkzl4/xO5bkmCdyZX/jexBhsl4W4osyMXdwovPlx8fme3e6HvHqXIni/gG6r6EES9OeXNv040XGZjghDWGrTd03qxE3Ka1aFUH1SpRQqSbZvXvo0kJYkjCsyl2FAfXFyy3qxL164zyTvhbru+iNcICJTAty8MwgPWjnhJx7agaq+fBRf173QCUeXnL5clhCiARX8w/3GOQkWmkyoTSWnNWeWcRIji47RSky40IqJWHJEursi0SuQxQkBPkrHFhpo1wOydRVZ6tvZ2m2PenK66RfsYM67NKIyyatFZWTvSnTk5SVZKxeouEAcdojzKQCkVbfrfV7KoJgCY4de2TgQe6/Un/NgtbDRnTH/As7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 2e9547467edb..0291d15acd19 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Thu May 18 13:07:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91809C77B7A for ; Thu, 18 May 2023 13:08:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 195DA900009; Thu, 18 May 2023 09:08:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 11C73900003; Thu, 18 May 2023 09:08:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6444900009; Thu, 18 May 2023 09:08:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CFCE9900003 for ; Thu, 18 May 2023 09:08:27 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9C18B80817 for ; Thu, 18 May 2023 13:08:27 +0000 (UTC) X-FDA: 80803404654.30.10D0E3C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 9A0CC20009 for ; Thu, 18 May 2023 13:07:57 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G92AMwAH; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415277; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=lTEtgdAROHn4rvhZdhpteFFT1CXmDlaUtHzS5AvQDR2/qYADS6TtkXHQpk1o8HEV32uzhs k0jEWyplDtMrWdD1RItrLo5l0ZL8UdocAh/dVrjQVvwG6p/2OK23114+5LAa09+k1dGdVd 068ElnhSWZ2wvXfw+3wNQybXHOhRu0s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415277; a=rsa-sha256; cv=none; b=k5cUGETSo3132eZwobbJnvVvGHussUrV3n3aMxiXzgplkjBN5PjhoD3eL7tjpkcX5yyFLx Godr8zHGDH4QxjAwFKs+/bCu4k1Dx4dC12gRJwHTP2STNySbBX+F9As7NHK0TVJU2N9I6Q 8NwbTXTGjuCDFbAXQcQ4BSUshCWeb7Y= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G92AMwAH; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415276; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=G92AMwAH/wuYua3fMjsrEKSHeal6tIWIy8oTDPYVMKBzPHA+GcDJJgZTDFsf2O6L9fTTNF EbF8gukG4kpQsZUxz2VA7brFGMDJXHYNACeKnNsB8BrcgClCxQUIWfJ5QF1uO+9QcLP6t8 F76ik4OuOzaIeX6WBtMRTiSDCz9RLaA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-539-6BlD0X51NFmY4rfARQLhyQ-1; Thu, 18 May 2023 09:07:53 -0400 X-MC-Unique: 6BlD0X51NFmY4rfARQLhyQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6BC7328237C1; Thu, 18 May 2023 13:07:52 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14D062026D16; Thu, 18 May 2023 13:07:48 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH net-next v9 07/16] espintcp: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 14:07:04 +0100 Message-Id: <20230518130713.1515729-8-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 9A0CC20009 X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: izesg9dhthfdq69kwepjmh4oq6w41ruj X-HE-Tag: 1684415277-219745 X-HE-Meta: U2FsdGVkX19wE+CELb4Z6DxXey4NvAnVPinvvR+bK5DCA0q8AgXRL+1dykfm4WPWAlmDzkCJ/AHy4LtjDyuOsmKLemQX7HjmovEOu+hJ2C6iXVRtefQK5EnwMLbUIgKJTuwEMOvZdza9C67el0Eiv/DirOs13YskYZK8oanSNHwZ4XMRSCEoVBbCPQZh6DGQSQDVdH9N0FUgoW9yRetsx6PRaHUd4Nkz3DWB9bKxULIkRnDIijOdyBmRI6W6e0RRWMc5Pg6l4d8CRBUlIBBzUwsB0IB4JlNfNwQaJjuUNIS/Ynd8exZRyWVew6BzG84+IOU6vyvXNzHgGxYm/g3pL9fBwGV6tP0sUbbi+Tjjs9JXablOQWM7fatBk+33U3+sUsZjcoIez6h22j/fkwNT8IySmturGr4WeS3TlGzL6myD8ue+3tvyv7GYqb/YPPkeikd/ZYvjx7VIE4jubR6dkVkfHEnX1HTW30CUxbkSk/4j0EgCRleyhNMOqALouEd4XenCuqhyKggj6V+gf/3N2STK3GNjfWmQuBUc8cQmP1YbPuX3LxLjiV7oZHeMb5DezViYOVph+epTBX37ahhAWW5yBic7/SgPphZz2f/q13xxzvLAineYletQwDZ3FvDYQOAb874pr6fxvyqFmoFfqKoo8g9AC3ADWH+30NKk7UEzop1TfBgmXHJ+IZkBk59Mtw1cLX8A3mEP1z7JP75mQpWWB7NPFWHtUBy5oqwD/siPvw/Nhgw+QAzOXY/K1orwaPrUQlnLlaAHLFy/qBiwU6SxpYhsmh+T7faldIPUAL1R8/GXmCQZZ3iqfsaYlwFopBbxEM/R1Ic1pQ0erXnTtoG812FNElpDEfSJ9lCQAzCMn9oQU6ehNOkUEaHS/Jd41D8QKGDITsei0oYrr2Lkn7EG2kqWJihRspROKStn5sKUhhsRmHo1S+KLBN7dn9h0Jrhv2JU9J3bGCYdZ+46 S9+cLUYF RmEtUXfy6u2eQAxsXScA4Eydc2MDOepgiEDZVlISp8FxQeN3RLXJqEe+mNq6oYp+i1GMFVGVF8FJDZWuUu9Xrr56LbzLkgbRzEzcMiJxZWYntBOOCPsKUCi8oQBNZ5ZabNEGtC3PhCxrlEnK2ZHYjjsT1ToBEN/zKnaSVgYnG2pNMQpcyp8jicmI13Sc4YCxlRiIOt+XvDCigJJ8bEkrIZsCS71TLwZgBhJCTGSmlm1pGVy5+NSB1Phpk4UW7o52h2pkpM9PLEIV3ruIjXIe6dCQvBQWqbp7aqJqIzqbzwSbwVAriPKRVM5wJ6hmFjXRHiauLOGQdl6eB1LazopFWymfLiJz1D+F/OE7Q5aTTZjuuL//EemJ8mPk85kD4P52/pZeuNaAsVF+p9q3qTsjIJE2dTg0f6pxpZziJLpG8Sc2IsTv9V0etyNziIcfBjKsNWKaYAxYwqWOmbc5jlSkeq3+2jeeX59bTvqgWQTRLAbTYPz9wsgZkWgAV9CKKiSJpbQCHUuftrZ2G7o89k8nSKLYJsaQ6RvaUMPabcYjAqSoe3pVGW0IBunaSF4oMhQP+BCw2nIkyYpwVs5hO3ikJhOWIfckuFAJoWjuHK9kTz40R9oA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Thu May 18 13:07:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2067CC77B7D for ; Thu, 18 May 2023 13:09:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A938090000C; Thu, 18 May 2023 09:09:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1C4B900003; Thu, 18 May 2023 09:09:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 897A890000C; Thu, 18 May 2023 09:09:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 75AA5900003 for ; Thu, 18 May 2023 09:09:01 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2D78E80817 for ; Thu, 18 May 2023 13:09:01 +0000 (UTC) X-FDA: 80803406082.22.10E8EFF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id CFC9240027 for ; Thu, 18 May 2023 13:08:01 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MeAO6wyl; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415281; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=uCOM95ckOBCstRnBs/chpDx6VXyv0aI3qfIf/0rsryO634RqZgKDvcZc+kpihwxxoFcSxp fw59/S1otCZmn9ISzXHjfJJ2IIbLqh5l6+sQcD2H3nnJE1SbCfLNCnuc8svtdMxEsBPTUX lINnf2kY6ExmQDu4IvhARS1uUxlwk6Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415281; a=rsa-sha256; cv=none; b=sOkAxnHNUu5uONG7V3CKnXNCOoFhFXFWTj2pNddiuTriuUsj7/9a1R9K+obIKSZX/0JNXD EAwz1vLogdl5780MccCzLntvdU4CNEahrV8eisYHBXcMWdWYonm7SZr+T+7uUWa/2wRSHt agHRtVp9yxP0irfwtB8l2jojp/a4GPY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MeAO6wyl; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=MeAO6wylu7gjkgqeuDQf0g9Q8UMSi4t5Ua/3qfJizKKEYYZj0w+sEoUUZ5xNBFvS1zBgIq CNHWQcvgv3riAIuRewAPZOprF8QsdDLwPoHHh2hP3qydK+3H2889Ns2D5jtSdnrRObIkm8 8uEWFN0hpWRkCahg/MVyD7eYmT9qx5k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-551-cLgcPFrJO0us3MfDI4Owqw-1; Thu, 18 May 2023 09:07:57 -0400 X-MC-Unique: cLgcPFrJO0us3MfDI4Owqw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 540B585C075; Thu, 18 May 2023 13:07:56 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 36E3E40C2072; Thu, 18 May 2023 13:07:53 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v9 08/16] tls: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 14:07:05 +0100 Message-Id: <20230518130713.1515729-9-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Stat-Signature: bs7srw4oomqfjs8nxgacjifehtcxcdh7 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: CFC9240027 X-HE-Tag: 1684415281-576679 X-HE-Meta: U2FsdGVkX194R31esHPzwzuizxkEzUIopTSLMB9mgvUKMeQLGMRmq+icpYr0bvHgig290bo/3iM56EDz6L+s3plM7A4Kc38hpFnghCZ1G2fy4SnULZz94sW4XlCEE2+R5drP3opKdXd3oIub3A70mVi31ZP1fJ4MGRuCqq0Ye0m0OcAAmspBWiaiOW+MJkk7nzi8dj2aCw8cUXEmK7wtmqgX1jyxsMZn5y64vn62+CtBw41lq5+VTIF8c+tcfidtD81Hj68/HSjkly5j4K/fNEQJWPpdwW165opHP20H84RqhH5jpEGe7Icnp6wCLni9bZRXYuCs5IhsMWtR14Y/Rapzhiz0wymf58wlXnUElwt+4PhgZHAnOnwxbAHwK0TM8g0E56jA8bzeRom6pA2kc7wB2WEsROpnThCPNLhPDaP09vbi4inPcq9Z6hFZJ2nymXQ1ptUKLluo89sw7/xiawpEVx3fgvcf4GGFnRPN3lrsMu90geJEONGECqASNe3QEl+5NR9aYgcw/MbUTT/Gq6HT8VU5jMuQUdkDJ3hErKIwExQRVqEC7qg17cuVlucf/jFyjzlwY/woX6vqaABWn12GevRmkauGgkuHvFWN2o4XGTk0brp5Ju28TSNxXc3vfXQyWbZ7cc55EFkPF17fnVUVAuSNGsQ9JHYWVreKK1WAaGc82RHF+Eubq6zGZbK42s4d6kCdP+CT0pLwk0VR+V6+I0nujVCUVlVkk+QEUnbUvDe+BDBRzDDj2L9/ExIwwnmQhHLN0wJnsjjLbLLKN4zs+S780zqQ3opHkpazrPIh/NZySpXOF9PzQ44mvm03zTPV4INODPanHrtUzZxetBo1I87K9cQljP6bgisOaFrSU6F79dr7B7tcdmDWWxk4xwU8i3xHo4KwAoRXxACmKr6IdWjRP2zq31IF4nQ2EEYf6I2snWWNJqB0RsWaZ73KDFBiyd/XpE3AZSYsq3R fIaRBJzf Hnp1oOJ4HQFVHkH7lIhdiAsliZj+xVcQfByIzIhE4hvO9sJRoubx1UT9Pizdo9njO/b0Z9AlBHk0fzawotvnmKKlgheEGQJO2ls4+FDjKdySaN5wTgwJh46B1tu9enXOQmMvJjkSFOLTVix/obvrYboSsB4w04nOVapOXGW0dBQtOJ7yfhoFfyYmzCUmvfyYS0b0+o6SwhyjXx0wYQXp7jVOHAtqgSOyINYUud9gUua3K6b0Li4i8GmwJIcKwWW8FVD/G5hkbuemsqHnRVjMX33BR6jyQ1wejry6iOQYbStWMQpQFFwl3X1yN14aXsTBiqEP8sMOhYWGwtA/mIFC7LaYqvy8kKmyVgRYry5fk4D69gqR3g1+UguSU3AwTvwznuhn6yhoDEN+BYDBKloBg9kNO8Zy7aazsuYfKr9wFzpdGFYm9v3c6bIyZRY8qYO0uhkwzPFIE9dbi19C9FAcjvLf0LCRtvDrSIGnRrAXLq5duMH7yXMMZs3Fm8ZwRjx74fuNhOhtr/Lhq3N/JQVune2uqs3bz2DVEwEi4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 6056ce5a2aa5..5791ca7a189c 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -258,7 +258,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index f2e7302a4d96..3d45fdb5c4e9 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -125,7 +125,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -134,16 +137,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -155,7 +161,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -169,7 +175,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -247,11 +253,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Thu May 18 13:07:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E11CBC7EE22 for ; Thu, 18 May 2023 13:09:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FE2E900006; Thu, 18 May 2023 09:09:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D282900003; Thu, 18 May 2023 09:09:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 699DD900006; Thu, 18 May 2023 09:09:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 56C10900003 for ; Thu, 18 May 2023 09:09:35 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 30CFB1607EB for ; Thu, 18 May 2023 13:09:35 +0000 (UTC) X-FDA: 80803407510.09.63D51BF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 66C8840054 for ; Thu, 18 May 2023 13:08:11 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XCUeT6b0; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415291; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=Gtbrj4V9fGY8ezu/RBjlHMvpa0NvvPym15rrNXPjm6H8HDNoOW2TM70XeJ4ycFq4z+bvl0 ufixvowgDPQ26xNDA9SQeCpL+yiXxDjYH8yew/AS68LCtB4md2jmqEpTSoUktipxSRy0Cc R2XLVhud0m+Ps6xeoU4CLFXHxB62EII= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XCUeT6b0; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415291; a=rsa-sha256; cv=none; b=gdAUc0hnELHJOSmb5krFi7a/XqS5kn2ZaKPRnswpmuGyYQpTP9IzwdUaMxKks0HsCL86PH +VBTiQgwmlLO3B2p9MOcx2T/bX8+1Mk8CqPrJc3ooHPCddEzAR7jUhhWN6cHth1HxSnZg/ wBtH/mDF34uKK8pdIRoFIO1IedgdqlU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=XCUeT6b0Tp8is9iTB0w181Crn2QgYjMfVB6U55XRZ36yejHqAmhSYBiN7o+K65thiNlR2T sfblU74xQ5nZnD3Jz+MBngR40S1G60htz5U/F80p6Uvt5qwqweYf2qciYx6d48AQFewIsA PjVDQd5uvPVMUWP8jdTawBWnsCFkoFY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-rWJ6E0svM2ScwbkcEaLtzw-1; Thu, 18 May 2023 09:08:03 -0400 X-MC-Unique: rWJ6E0svM2ScwbkcEaLtzw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CE5503815F79; Thu, 18 May 2023 13:08:01 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5DF3B492B01; Thu, 18 May 2023 13:07:57 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH net-next v9 09/16] siw: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 14:07:06 +0100 Message-Id: <20230518130713.1515729-10-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 66C8840054 X-Stat-Signature: wkypfeycbj43ex4g8eysxwjp1ajxbdgt X-HE-Tag: 1684415291-436076 X-HE-Meta: U2FsdGVkX18ddZiXWSE2KxLeiWDlWjL6LTc01Y7b4aurkEivMq65wLFf8/hd+CDG3Ep4JS0ojQ3odpnkqQIgW2woM1Don2OGyB3kIxu5VMvBhjMZHWmHmRktWlUJtiDtxftvB0CrBx0jGK1hv3/ZAGrpnWc9WgwXyFZIsX8oveC0X2VwQpq+xedgMPS3U6cstw7LCr/XUt/Jd4gkfOS+jIiqGnf5KIrELmDeguhUjqLLxBtYDaEaiInhTim+5e1axrJktV7gC17bZMrqySWB+qSH+A0JOkrgdRL/+JCBPWVxTTNFHMbbJlQ0ZJtVmdH1dXwv0sB0+ZE0o6RkXfTzUP77l6JtZY1SzEZxYS9jODwa3aSqbxqkeOOfLIT2kvaLTo8h/dypl8glynBjKxRTmJuWseCYQhAKAVKrM02gQVM82RDGEGMDajXNTZ/672l77s8rizeX4HUl/eGq6TWd+cTl9K93Ha5QTGTSwmVBu9XoJd2PkXR1hPz/u0ND3AS5eXspwJecZc+dyy5chxeoi3Ys3c8g3fgNPfINZQHvB2PCbiotI2jSj8FCN8D+0wruuxvLJStR8NgekEee5kklIc2OZKDafdYOEfxYP6YghV/Eh7LDEEexVTjvJMaJ/HUC2Yfy5w88jvJDSBpATslmTw8IFe11w5QmrXUo5mPWg4uhlGFtd9S62S+YtL+I7MFkE8d1bnPfWzV+hiQE7htBw0ettEPEBtqws1ij0SYO/03qmjPBPDA6+aLwne+Uo5cmnLUU0KA0gsmh9d+4ynt4nOWzThzzgV/l3QRPwoTSFufpXjdqxxv0h9Dcy/n3205Ye8mfumoGt5p26e+yNDip60/HIXNpw87LfceMt9Onxg2wqYnATM22X/fXcrmXCc2oklYkp0gzaAUkYSBEEOxqouZVOfLd/q73kA1MMCYpDx/jLHeSp6nh85GJ4EkwXc8w+NmtxKvQuvQSErpjZdZ 44/vYbaH xYxuyOiYHyEuPxwCakdwmmatdO2pSAYsptEf14T04U7AcjAkd6D5ZgQY7nSfIRHnWQrzHv1nL4BtHt6bblCT2ltIFM6NiW8LdmuhWu8Q5ICxq0reUgaDtndqG/pvqOD3ApmRbbBcLwk43/oOLHQ1v4IRQL2++viW0KWE8WOEWQ3Zm/6CXosJt/0mLaLyJhdJjAkY75H2TQ6WJG2ojKmpDyJQLplCmgjI60UXUSH5R+MmLs02DBhBvBldMDkDSh4kcsu0BJGTj1DjdZKKPw9UatBouxTGtcMBKjZ3AmrvCuFdN4BSvNaTkVN8Co6lI0rpG57edTBmj0Ayeflv2zWZaRLkalbjukW9JhpdGCtb+HEV+v43fsGTjdUoK/XvfxLLQE9Jl2wfy/CyfjkXumrh2F7Ub3qidel85/J+4T9azigtlO51y8cvDv0Bh5MvLNbsXQi3KVukuka0zh79zcM/GRKiXWHCpAtqPSyWduskAVSlMqDsdnRiB04a+QIK35COF809J9p9akdXrIXk/tE7YWKj5lt0mfDcmeBwFRc4xkf5QBtw34Lpg72JSlLEvoBEn/HwXXvAQDSnLB084pU+uIzU4w0fZXHZdubA9ulxyT5etIbqyQmcOjZeDCIFpYkVyS/Si X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells Reviewed-by: Bernard Metzler Reviewed-by: Tom Talpey cc: Jason Gunthorpe cc: Leon Romanovsky cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #6) - Don't clear MSG_SPLICE_PAGES on the last page. drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 4b292e0504f1..ffb16beb6c30 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,7 +312,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -323,20 +323,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Thu May 18 13:07:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4B2CC77B7D for ; Thu, 18 May 2023 13:09:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42A3A90000E; Thu, 18 May 2023 09:09:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DA90900003; Thu, 18 May 2023 09:09:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27B4B90000E; Thu, 18 May 2023 09:09:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1326F900003 for ; Thu, 18 May 2023 09:09:14 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C9BE540846 for ; Thu, 18 May 2023 13:09:13 +0000 (UTC) X-FDA: 80803406586.13.DE7AAE7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 539794004F for ; Thu, 18 May 2023 13:08:15 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hzAxwfjR; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415295; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FkLIsuWcxLdm5m/RLKw/6bFEvudw3Dbf8/jbJKlIa0s=; b=iaJ8XXeH7swEF+QH3XDMW5dUgIVUXaXq74C99KMZ3mwvRjZZlFtv5d1LFlJtHsJ8KLJlSz ubsBtCtDRhzjgWIB4Ix/CkRndgQ7BZrXEcjGlXin2fNin8jJKSsamINkflo0E9uD+fuCL3 KanW5zy4fAH2uulHE1VtBmzURZCBiS0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hzAxwfjR; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415295; a=rsa-sha256; cv=none; b=uCYDPvcyXwKJnLIeY7iZjv58w9EF08M5tDPzL6p2NOYknBCQQcSWIMMAkVgN52VMBWTnG6 ieIt8m1wYfPWjXmzEdoWqplsnXTPUbLpwmrMRIIKphVZzQzcJlFTNhU4mK2bw/b7OVclaT 5t+YbLr8si4O47M1ErebOGCfKdNFsUg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FkLIsuWcxLdm5m/RLKw/6bFEvudw3Dbf8/jbJKlIa0s=; b=hzAxwfjRfrKPHQz6EigEiQWKCeE40uh2UozJPCm54MMfb0j1GM5jLs6zr7gH10GoCGQ54l ikXADZNUGpIGkpdKXx3B8fN+x8nNp7drGuFM+JVlCLPW8JI9JOAXEirOlYfebiQxu02TZn T1EZF42oApjSzwOli1pewmto/TwIOB4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-459-_LTtSiUiMLObar1uKYQXhg-1; Thu, 18 May 2023 09:08:08 -0400 X-MC-Unique: _LTtSiUiMLObar1uKYQXhg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 594D0857DD7; Thu, 18 May 2023 13:08:07 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id A93852026D16; Thu, 18 May 2023 13:08:04 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 10/16] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Thu, 18 May 2023 14:07:07 +0100 Message-Id: <20230518130713.1515729-11-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 539794004F X-Stat-Signature: 1gdws7gzxjtj58rwehrie7ytxt7drkto X-HE-Tag: 1684415295-795023 X-HE-Meta: U2FsdGVkX18hAdK+VBARYmjEeWQcr512QMNPsOCFt17C3JTf7pPQGbHUs+AZ312n+ixR8uQAAUGjLn6HkyPH2bs1cXzwBHu9qADt/PDw0VWeGdO/523X44z1KlIC8BJnW1ZgVxWy07sVIF6XByf+Au5MACrNnI+WBpXwn2juWRoj36xW9V3W5Ltk6FmlmQWP3OxK18i6Iov1zq11aaTZnm/TFirtkni4fLVbbt8w20WhYwRc5mX3KfusctTxHPGQh4+RkhdAnSFu8it5jN+Rn7W3TgWDZUMOwySpQSOBd0I2bkvSsMCwnJuQUN0I4iPNGDJWWSEMHVdux3Imz3yWEF3ycdTopsZ68wA+ccdQ6G6nRmzM1+6EBpUMHT6DNfJqJh2ZcluCp4EY7CGAfwCzf77tzFQDDVoyb8XTNf4tme85Y/W/ZawOaTnLW7pT3FJ+5uapQPM6H/rhlj2vHqA6NpCBUanl5eUGKqjIUOBDivIrU0Yj9RDfKvLiCQNAP8P09NLjWjg+S6hfdww+ENH1c4kgd8hhiTi7R9/WbNC6JVn2xL4jtTnCotw2sqeo2fNIdqM6VIcCzOediPoVEca+2IPzVGZt7ibUUhmLH/KIvf2lfJL4d7hzy/l7ts4dvk6XYHnJGRLwu6gIYdVkz6CnexUSAdxRKAW+iBY9OPPlia9OIA6HqpKcUV1v+aWMcODNxtJQsjkgSTaFgvcIeo6n3hFjGK1RlxU+Teq2M0oavkHVKBrsnNuGsBPSjkuc5R8OZ+G5hv4M9x8gEO/cjYXK11aHdmOlcEQW2wzqNiQQWO1zetMOm1rU5IBP8by6UnGDhFdYuaOkVfHQoKEHhAf/AYtImANANjUv6qV01d0Y0fgsl2jt6p85E7EGQ0TIhJ/2rqFsqPF9DE19wnlvJBQU4XJL0mo3mOaXU2MBwBKrydGVkhu9+JViHzza2Wr+SwE7axvpXmH8rlRCXZ2zni7 BFEuPYIO QK9iOXRdCK6N+XkUl4YxpHaRrZ2ieyjdVapN+6WFtwe8iJowVt5zlKG/+IyN0ad3fDCgvdh6ER0zeYOwlAmGX+qH0TUiLCHhIHjjZe9YLolHO+LcDb5cZeLY1OK/SsrH9WELjxksrOgWR0vAerF5QpsB57VRhia5FbFduBzMTFJ0NhYYTu8eBiKvmRCFWTWiZiXV4a258MIpxQHi1cgiye3c0i++9yY11WM870eDAjONXChdRX/UsWcoVNKBlbFw+znVaa4PN0Tw66Omvq1Z+YuaIVyLa76oBAQ3HTF8oRBajqayfG5U7vXHeAIMfyvxlp4K2n4NDUwniri6NJYh3U2jzonQJT4J+YcsXJ5R7GJV/A/C3ItGh9hfrfnCE8KdTp/niu+ERlPSHuPtDIA2N5EDVN7BVOu/iRpv1A3kwIz7pNeWcQ9fzjVtsP2iAz8wWOsBjxNjG9rBqeoocvs2Ey0SNwX6l+YufKHg4i1K15vgIpIutax0tsglZAQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 04a31643cda3..02a6cff1827e 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index f3a0c02678e0..e9506cebecce 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,12 +974,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -988,18 +993,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Thu May 18 13:07:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC5BCC77B7A for ; Thu, 18 May 2023 13:08:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75F9A90000B; Thu, 18 May 2023 09:08:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70EA1900003; Thu, 18 May 2023 09:08:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5895190000B; Thu, 18 May 2023 09:08:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 473A6900003 for ; Thu, 18 May 2023 09:08:59 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0A5471C764A for ; Thu, 18 May 2023 13:08:59 +0000 (UTC) X-FDA: 80803405998.30.C6A5966 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 8460D80161 for ; Thu, 18 May 2023 13:08:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IZPfQphn; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415296; a=rsa-sha256; cv=none; b=2MBRpXnlcbJpUujml96Jy9y7GFOaiQ6WXUd8IM1dxdubA1kYr8acTmsXM6wCjn1OHiOD3M thqxvHfJIa8Lbfz5CJZAnbYSxrcfH/2pYmkHBhgwzjB0ItTDQpsM+tW78pQObi8hnMZ50F UyyldkR8AdCHUuZX30qQP+4H17Km0jA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IZPfQphn; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415296; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=TxfQZBIaVYN5Ww5L9z7Bp5BeAsSZ4kaUNwqnOhX5dBqnxmrg+g6c702WksF5h/0S/P12AS qWL2bzCXqa9hW0qQDO5y6ciOcAmOHCgIjCMevpxmvFVZhsNknDnqxkHbB5AXVNCmmZ8D5P 4QOuzYac9//vF7zlb9WnmdgdWhVtR1A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415295; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=IZPfQphnxvTxMtTbxTtjCHZJUQrTcR7Dv8VJuWWvOzvw7GkbZ4AOnljsqz4quz8vfHCiSd IN64I9ELnjCxoQUPnlclu6phuNyXDseh9aygXcQoh6glXYz0YyStdNpRhmFNeqQzXVluGr QrA5KqJZ2DQRwHgCcl2bNMEfxpEO4iE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-255-IupZrLRDOgiU6UQOydG1Gw-1; Thu, 18 May 2023 09:08:12 -0400 X-MC-Unique: IupZrLRDOgiU6UQOydG1Gw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2621484AF2F; Thu, 18 May 2023 13:08:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0967F40C2063; Thu, 18 May 2023 13:08:07 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 11/16] ip, udp: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:08 +0100 Message-Id: <20230518130713.1515729-12-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Stat-Signature: wf88wtuxkdk5em6n1skgimzcczmz18f1 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8460D80161 X-HE-Tag: 1684415296-371341 X-HE-Meta: U2FsdGVkX1+XzJCmcK0HMQFCrgYo3lVP75pLWSvqphIfoFAhImXaWsNeJkZ+MGtYfpUCbzc4Dc2QvTLtErTjXn5WYsJeCClrE25SGwNhhK8vEbHAA/aAfBtvsD8yJ4xBRU7h0ukRGq61p+zNSbmUy3E11O+EaYnRtiVTXqXWvH+Z0vCfg1f7Dq7plJXmXc0c7mJOexPKTzSwX+nD4Qalgd027IG/SdJdkgqk4j4WsRsdhzbiSQC3nu7X9c+PA5rkOmqmBNHzdCJ4JODQAaC6s8ysF9119qFm4Gu5HiPx+LYRumjvu0p5hRBP6lwZrGkb201HVrG2eHl4wXs3jZkNkHZVRIZIf5SxTctk51LsmqZD57O/K8dhq/anHv8zaewIByo75PhOtZoM6Rw+c0tqXOTKkz7m5+WRIwo/5Z8EYPrNCQSdXCEn7j16TT+gcvOSU+4Lq+orOIWdJ4zqPXxDh45UDAaDLfsrRoav8dKp7CdavRdFeixhrVho825vaufOQC+jGOXu7UbLAn/f5aAkrHkhuGfzrYj1RYzES6ejo9zCRXVwD0pAmY/lWfOyL7haBXoRKGr5ZiuDApT6eGyUAfMNteMKn2aP+dr8eWvUF2LNyb/H6jPKQ/afuG2jCDY8+oMYV/al9kZ3zhSRAQq+Qf939XStbOfLiPDfu8FfcfsawvMSgp2IblkYdXBBieXodu2EsJfUN0BFvgCBv/gKTUhSQ5XusoMNgWo7ruYOvdDshFXAj0hPSFqwsZJYnjQfRWDkYk8zIFB4ynA9dU4sr2fowpFwRi6m+DUbLbpM0jTPhlvWEjlnWl1CmAUbyvPrjvRllX7wRv+BZhj0Dn/NuIusU8RS5anWMqwWdnJT5iOMl/K2Ay04FvNSaCqq31wL/ByoxLyRQbfqFeu0o1B5cKRCduwZFbF5iXoqc4xppLbTdj8OVZ1ZKU4HRUZtnTYybwyzFf/YBVixeUFvqWs tZoNOtF8 m14T5X3sCoh1oteKaFGk1SGE8SOYkEApNQeFVv827NMjZW9OSlJeEojAWpV6GevSUq/ACuRqk6bKXzAoiECJ/tSevy/DvejNAzibInC2gFNvSP3y8WJ8WplQBksIEip50jta1QVQH03la7WkjUE9s5bI4f0y7U9p2SfsjpzIoaUDWKQnr1lm7PSi6wLjfxeXfung2qRZwLEUh/etqWSaF9AZ9iq/cUofAxXvqBCDvQZfwQGJUDVyoF35+K0PeDxyKLYBASy3O8i+uN5DMnht+pNdByaFtkKUukrSZn52uXIaA+DUVrNcUDMZZLvbLCmuakkhl/HjxFGCsVZeukadJrFLleurA5d2uhWVardSkYldufeuRSL9YFRIhfGos3hFnrUOGoeoHs2qAqjSdCygLwnzfdEeKJXL5K+8W5aYriSCXhKiYnfncbO2QWGIXgcd1bTDCTMi4qWE1eUsuP4XyAwD3r2kCnXefc38Xv6EGcCTutRFzhE6y8NPNA5D4orSL0oHHhmgaYSShYw0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/ip_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 52fc840898d8..c7db973b5d29 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1048,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1215,15 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu May 18 13:07:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 711A9C77B7A for ; Thu, 18 May 2023 13:09:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14386900011; Thu, 18 May 2023 09:09:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F632900003; Thu, 18 May 2023 09:09:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFE40900011; Thu, 18 May 2023 09:09:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DE99B900003 for ; Thu, 18 May 2023 09:09:46 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B5BD31607EB for ; Thu, 18 May 2023 13:09:46 +0000 (UTC) X-FDA: 80803407972.15.C6E0FF0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 337D8140259 for ; Thu, 18 May 2023 13:08:21 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XHi4mq+J; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415302; a=rsa-sha256; cv=none; b=YeOmtKI8Cu85w1tcAWwio//JvkX469+gDXdFb8BYSAmHd70ixXNoXeySEXBqVyuZ84vVYi NgqSRsGGpUjmQZ3Bx1BquUqkbdkSMare6mJv+v+ZyCFB27bWekigKOJX2qKY2IVxcaVK2v OOzMX/iq6iF8qEKkQCRQsoJgFJXsFmo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XHi4mq+J; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415302; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=dMndARiqwfkDKXHcokZeQvxe22SCqRjMliS1l/4wdthBiVxFsdaWHxbclJFNxR7/gs+DHd VrryNTQxqTzNlkGNllRmpzYfqbXEvXTwx2fg697aM/6aFLiv3ogx+negnUK0L1394J7gx4 4xR2NwFerWnApCqVp3urRfVLMqW6JRg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=XHi4mq+J5nzrw3HKtAV2E8dTepBd6HQ4m9q0gGwErz3J7Rawtw1OnSk6uShLxcrPZyBpNR 5ea56cLriBTAxCNSqTV+qKg8eQG69AJnRVCnZ2XqGvR5X39ruRjETH8dqwEsKvju92gDz6 tOfL/pjPzda/ByIpbpfazR7fvJIzH8c= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-lV58l2MBPkWSA52mh6FUSg-1; Thu, 18 May 2023 09:08:16 -0400 X-MC-Unique: lV58l2MBPkWSA52mh6FUSg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2F4D128237C8; Thu, 18 May 2023 13:08:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 397681121318; Thu, 18 May 2023 13:08:12 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 12/16] ip6, udp6: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:09 +0100 Message-Id: <20230518130713.1515729-13-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 337D8140259 X-Stat-Signature: x8ure1ey1uujbqz53adx1rumzypy9kui X-HE-Tag: 1684415301-36948 X-HE-Meta: U2FsdGVkX1/vAH8Hff/fgT0+0M6Gs/p0U6tOu2hc6ftIY/9dnrd2wHGQq0xUi6vX5oe9hb7lfRhpxC9Q+wW4x7T6KqbdkBRt8bWTReW4N+8VsCXWKtljJDLgjSy2oR+7e7KNyxoDpmCgktS7JxOmUxSOUx736D/pzFdxHwban6OEJ9+OVgI4BPpf2fYuTMzTaagmtYNRsLr2Pf0oo8mfJTS7W1Z7HM44qw8xuAlbMQB4HCnULtSiSwG9E+zI71ngncIi9mT95InxW/pvze1YPZZDNsnLsZCKSyLU8qy2EamxTJ8I7Fl2fs7TIKO9zMBuDs4RbQ8b0w+Q7wbRURPTDVAUrjR4N56UdGh7mi4heYAQsVDQJ4cU0Pt5+nOMo2HqODcYc0lq8DLInXMIOVQK3anlOUTY+0xJqTF1c8+N87RbG0NsOVNPpBSDP03Zu41M9bGknnx5yVljPdzjmKvrS6JqMyH70NjHzQNsEraEi2QPy8X4sRbf/4ORN73X7iUcOEu4LzRtxNvBz2fYKNb1wUtVxa9QbNh4PEn84GkvErwRiotyXVKrbksJg3/vGhNLvRQB+yLmLtyZ3Bck9uor8v8NjbRWGcLM4rUKFQbqP5QqtllohOVVthlsmZscaV2mIWGRoBekqUD/YLumoxJSw0T/KnVPRzbZbGGaSzC8oE5uOMe2El2zRDGzC5z5frv6Bir9wOu2tIOaqvpraxq4CK2RPeLU77SdQRKIUc3TkXARA5MCSDljHhNJZCeTftZTaktNIdl1s+NbKyKHrpoWNYvQtvth1Bf872sAnxkX5NZbxCQH7PRP/30XHGThP2Ac66k1WIo/i+/WTIVNjaRXzEOoS1+AdSvQu07pwVeQz3XBrMS63Sn7XhOS7832W4/N0QYIiiJ/EuiSkYIirlXjSQqvqzeHJI8HNRvvZAGdo83fowU6jRbZBRI0l4GK34XlLixysnkUe2XC3Jqq7PW fyFGTXqL JwSqXjZazLS/7AeNJZN376tB/9OrqEYmik9kmJoRBXpc0Y/Y+q6KlwgojGTgJGMsMmRQFRSwaodNzfReRwmaxEya4kqmIFEsc4fmAGzdeodUEKlL7iukoFCB7f7mjvMgK/MvgJLLusD+XZYtC3NPvvWzQgtDSs29wXmfhoPjDeu9d1u7SXG+56ty0geOM+Rc1JaIo7noTfT206D5hKb+D97mFxwKYpXN30/aKNvSbkE3UUpchMruFIuAowUgF4XRAdSGChwrV3KtHcpwK7qG2p5S2lKZ4hW/spD4a2hyrkb6+7kFl/NZcgk1JCONIoHNRylZ2wuKAhastV9hDDYDpqEUUZ55jQ1aAy8AIT2f3tpcP1nUEJQLtGM/NcPU2nG76dZd7XMh36na4Xl/xRN/WEVfZNwhkND5pcDrmUG8zBWt85GOwzNcNG16lezA7QfjSepNpQ9iDmcwbu7tOo059so4QKe9wCShEiH3+KdlmWVUo2hpbRR5QEOo2d42vQa3IrIrJ32tKIcZufHw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv6/ip6_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 9554cf46ed88..c722cb881b2d 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,15 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu May 18 13:07:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC39DC77B7D for ; Thu, 18 May 2023 13:08:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D61A900008; Thu, 18 May 2023 09:08:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75FDA900003; Thu, 18 May 2023 09:08:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58E3A900008; Thu, 18 May 2023 09:08:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3E9C3900003 for ; Thu, 18 May 2023 09:08:25 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AEFA1160837 for ; Thu, 18 May 2023 13:08:24 +0000 (UTC) X-FDA: 80803404528.05.E2AF4F9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id B178C80002 for ; Thu, 18 May 2023 13:08:21 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ANeiowsk; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=qI8KGC2W12EAkBMyfsn8UdagKfjDNIti+oPDzbBDciR90OMECEw51fqhff/Egu2zSyldwz AadcxeaRtl2njRi2uCpL+3DNuCn5z3em3E9DLEF+jb7au6e8t1SR+ebTebUJNLXhkGhX+S wV30R/9gzeYs2jKKirJI3p2FV4jVZ7U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415301; a=rsa-sha256; cv=none; b=cIa1V8Q6UnahixRgv/bkzGQLVoctBug6Rf0XxACdKlAK89NL6eQmF6AmXG5DuLPMqIJYo/ AkDfJQrAwj3ur0ExxP/z/vqkALrqiDz72DEEUttUM3imp7DgqW0pa8IEd2zv6CpmxQ419F NgwZy6USBpxwkQwW67zYzE1+fnzlcSs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ANeiowsk; spf=pass (imf30.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=ANeiowskGyQlAkhOj2dfJRBdmPyn6iP1WBG+oUsgizrCVnazeGxEtCmA/n4F6v+MrRqpPh +80Alo0gz7ScVHQN6l3yVrBlNZ9zgu7UvqyZJ/piOpbmFYhcqbQTGmW7E0mgM08QEgP4vP Kgp/7GK8m6a/TAkzRpgV9Pza9TAkoL0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-122-MLznaae9M3inxRR4K92iug-1; Thu, 18 May 2023 09:08:19 -0400 X-MC-Unique: MLznaae9M3inxRR4K92iug-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7E93928237C3; Thu, 18 May 2023 13:08:18 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18F1E1121315; Thu, 18 May 2023 13:08:15 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 13/16] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:10 +0100 Message-Id: <20230518130713.1515729-14-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: B178C80002 X-Stat-Signature: qx9y8fdadms4cy5rnx4dird1o7j5ouhg X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1684415301-205372 X-HE-Meta: U2FsdGVkX1+lo4Z/wYkrxSWIFcbGx+v89r/Ux1YDDkn759J+04z4Za5L91XOSMGUqEZrfVpI9K3eqcQE2QItGRHwdsL8DcY7okhnZx0aqSSkyG/eVj0rXPCT6jlejLLOGXs5Y2zGpCo6YUpnYFJPtzbmOlmr++ooeVGJ6qLHAU3X50bjCj7wABWKdYoqVGPyK2PiHrtd2ls11Uh6bMGL4AsGhhByDxF6mYb+ql8PV572IEVGAQLl/tFvmxYBKDfudPu1WfTjTOUXe/1XgstsyQmZspd5bOpr/OP982bpW+WSIApiHI5ZeODAvzLcCdJ8ROoPbnIbaH1a8gWMxxCyaDKVOGS4WqbSHS/PKFzraWkBos0VR9ewUOZyEIguk+g6aMUYXR19h2PyCNwGuZKCC6aWapjp0n4k6PaBBRwpzTjQj8wouVab1FnYLMbPmOtUKnuxY+OENSMJ91eBDItlSDrORRS6QSFrfEpbOvi5H/gjn7OYiY8vMbGoJNkqF476Lvk8lhFCbRPhjtbrAh+TtiqVoWPOJoNCEDA/xFMo/6zzuKEBAW2BSpkIBdWwteSFHZBaYJA5o3LtXKq0ckUVbfTBziOKI5W8aOnnwxR4xT9spLksxi/YH5s70Ke0/r43c0/14SjsLNjrcJu2Qbl6ioUNqGG+dFhdlOYyMsNiP25bfXkUTgDgXfp+qiCn7qjzw5ECHhUbnQlZ87zL6o0GynR4j1PvivKM+zEAHW7sZ9LNExtcaYhX/tqzwMP2eWygFh40tXh3Igm2muqLVuPAOhXJjIzBmIGFYSup9pucat/6VWkltgP3N/XE64I0u6sXMvHIlpq+a00D0vFFQmY5g9/msa713SgnucmFtrIOcn2FejzCeEqQDrTiZdwVyUwgDuzq2DRADdtNObveKbxjqySzc3mSc/q8gKziGrdJ5U/Zwc6dIJm4/ZVlXTspvbt1+0BnVV9X6s+4Qu+bytv JTEHLb1x 4RQa+1RE+phPKgiqGKgDRur0oEUm6v7EBMnSlHb2dw0IR9PpRz7a8CpSwAh38MMjq37Uy7Q5aJ5/u4KVffrnoUZ90GknEavbxxMRIgN0TuC2WRRqDL6HJf102Jc1vWL7fpPE6eTzyNvakEK3PBfQmf+ZR/cKRftyPcC6DIV37w/C3V0IvRo3FLasAUo63Njc6frRSuiwzDwUgO6I24LhzWg60O1dzCsXaADw7LLoj68pj3u2het+3eeXFHg4Pyjm5AVSMgo40XNo6RG+V/nYGDQbJkg4m5o/NFFvI/fBmqTytTil4+p5z/AVdyMLFcSyHRrZ8nBjEH49fPI03fjnW5J8bTQU74UTL5x+yKo4C1LeGUnADanBkL7lN+PROq5aXilNlpzpbF92vZL08uYXqAw8DnxsZaj300Kd2EqzoRvilAbyIH2TbM0QDWytdOAFRH36qu7HJ++ZRH0UvEBUR/3tK/X9/iVzomeUIObjSVedFvUHrGYZqxCrAtE4ZKGlu9PL7LnjmhIonYm4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - udp_sendpage() shouldn't lock the socket around udp_sendpage(). - udp_sendpage() should only set MSG_MORE if MSG_SENDPAGE_NOTLAST is set. net/ipv4/udp.c | 51 ++++++-------------------------------------------- 1 file changed, 6 insertions(+), 45 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..2879dc6d66ea 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,54 +1332,15 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } - - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: - release_sock(sk); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return udp_sendmsg(sk, &msg, size); } #define UDP_SKB_IS_STATELESS 0x80000000 From patchwork Thu May 18 13:07:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9F0EC7EE2D for ; Thu, 18 May 2023 13:09:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF02F900012; Thu, 18 May 2023 09:09:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2B19900003; Thu, 18 May 2023 09:09:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF18C900012; Thu, 18 May 2023 09:09:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9EA45900003 for ; Thu, 18 May 2023 09:09:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5F2CF4081E for ; Thu, 18 May 2023 13:09:47 +0000 (UTC) X-FDA: 80803408014.06.FCE353B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id A9F6720061 for ; Thu, 18 May 2023 13:08:30 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gh4FA9QC; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=KzV6RdFtGvmD6wNLCCSpkdYb8WSc3EvlgX/pZL83//YOCmfbrc/ytFXw0l7q7MZx5s5+rX RTGH9Hdn9TefMVZ90Rrpi2vbIGB7jtEZ1Mdi2N4Qg+Xu8UzDFniY1AlAACKHv0BIsoFQLQ Ps1GaU5GOeq6Ohpr7rw7yv6Xj1XsA98= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415311; a=rsa-sha256; cv=none; b=74im0JeKKewXg1WLUZajVecs1kiDw13RiD0W7lvJJQ8nPTScrh2j0mySUn9L0F+T/krwXQ 7GGBsY3zINUBVCBLTvSO9qhs0Br69SsgEX4i5+JS7Pop1bk0GKGrrbKOMSp8khy0aFMqly LHGBeobeYDn80xM6Xttw0FEifp8ekmg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gh4FA9QC; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=Gh4FA9QCzFKufyEzPncgsCHtWSv4OikTYUofsRK6+rIKXXSiel5tN2qfplrP539xlbgAuR QTNxN8WoAnX3lK1GWVlEeWT6nt+cgCjry3ASuUfV5VPZpYmdta5mGN+Ub9iHHjApp7LOAI E9sMfuzzh//GGxIAU5tyMLuyQvw6kEE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-p1Z1QuysOXekhrovt4BGFQ-1; Thu, 18 May 2023 09:08:24 -0400 X-MC-Unique: p1Z1QuysOXekhrovt4BGFQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3684385A5B1; Thu, 18 May 2023 13:08:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7D030492C13; Thu, 18 May 2023 13:08:19 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 14/16] ip: Remove ip_append_page() Date: Thu, 18 May 2023 14:07:11 +0100 Message-Id: <20230518130713.1515729-15-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Stat-Signature: f4hc1gwcrnt4k81gksj7zq4933k6ix5p X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A9F6720061 X-Rspam-User: X-HE-Tag: 1684415310-875417 X-HE-Meta: U2FsdGVkX18ylzT0NW4dbG1F5Ntt4XIWsKP0XkHAeHU2sMv7ewxLomEtkWOz+3Jz1IkCbf8DOXdK5+MMIWcbfM6G6GwO/6ajG5SmDtWtEnWHVbrDROB8sAJUxTqkZeV4GHqjwQfF6EJM/ud0UQ9xZV9b66ZkdDpOY4jQpkk10p0xFnsCEZKDkOLjibAa1vtkzWzx3YbdSLVRIwMjrTCJAhbbyLfVQLU5NQbI+lji49CdzxigpsXxsgJ93TeMccem1MaJs5B3sQGGIzIPewMQCBFG+Fs2dytKr+PBnlJvAadmTr2it3AOFyiBr03b+Bcb+7fuNQIajWTalHGrqQ3xEEPAWCvfoW4qXdY5/JUhPQKWGbrbVRlQUp3bU2A+CepplYWkY0BypwgzIrTfJxVWKj5wzacAv7Hm98xapOqUtDe7EKc9PhbTUUqSdwEJ0M6GXJladleWCL8nh7dCKNWGJEyquwk62P+2Qt6aQB6bBm3kaJq4UnOoILcGBp5No0GrHbW4jBKdjBv4wkc6fdISunJZwadP0c7Ane6/Cm+FcHJgV6plXP0pA6UEu4WErDfh3waEmOu7ulIyjuruwM06q4JLur6oOpm0FkvwcsDpCUdXlBGsDjpLjWcfuIVBZ/cG6BGJTzcYKYi/oVpRF9z361qPl41Y8VbJSYcokoonrdPR4RBVFQN+VQCVvV2ydvRKJH1JsqwnqQvKK0qor8nsy6KQABFpzEtGZjh73GBIIiF65JEPEayGexExZsWMP43j78DxbynOFP0de7RFEMFxNAejbAPoguiZpPa77qLQHh9JYQG3kg+hju4LBtoVXcIKpiGtz1rftPC3L5i4o9lT47DzjzlFX7L+VSaHl8/AjNbDhSgQvBUdfw+5bh5goMC/qNN/xOUG6He7DJZBQXt+1WAgEVe9T1nGO2IXNVgmu1Flp/pzkfsuERReLt23bjPnX9/EZzaLwiCaHNtGq/r gCerMpjA h6KgBhXB4evWQWxT+cvmcwYLUEG4ptFMZNBnkcZty859Qs1LXMiOz23El+zPtm/T2CD3XQ7vGurCEw1z/a3/QcURU9DZpn8m88Y0h4W8x5pBvCuSbdTE+tQWqSJbTO2/VvDyj1kkC9WKSC+gRT+kD+q4h1QndgiSeNYzlQx9WlltmAV8koXA/UdU10Qf62UWMBwqk8murVBnv+uWxf2EW3FZIZefiZLN20SxsZH5WRCBcg2CY7TfOAU3VrzKo2fjC3qKgnnChOpwVMiKJL7s59gLuAeSo/TJ67m+y01z5ZKZKf9e+HupVXV803BO7ya9Y5jgKn87L6L6OQlpXiQCE8VHkfonh2B1gOV49VNHfJ+zxW42C4wn2o7SnDxjDVQz8GCBOwcsqiOl9QVZbuag2hGGKuTHij4KV8vYrKzu3ZuGI8MhQBhFofNuzQrlbUG8lC5ERD4NFfoDsRGppQ9HCcWGTSMz4cBchZ8okpUoL0/e4kAmwx2GLzaHNrnfoBUbccpVBOjoGGjqmryA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Remove now-unused csum_page(). include/net/ip.h | 2 - net/ipv4/ip_output.c | 148 ++----------------------------------------- 2 files changed, 4 insertions(+), 146 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..7627a4df893b 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -220,8 +220,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index c7db973b5d29..553c740a6bfb 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -946,17 +946,6 @@ ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk } EXPORT_SYMBOL(ip_generic_getfrag); -static inline __wsum -csum_page(struct page *page, int offset, int copy) -{ - char *kaddr; - __wsum csum; - kaddr = kmap(page); - csum = csum_partial(kaddr + offset, copy, 0); - kunmap(page); - return csum; -} - static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -1327,10 +1316,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1363,135 +1352,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len, - MAX_SKB_FRAGS)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Thu May 18 13:07:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7239EC77B7A for ; Thu, 18 May 2023 13:09:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1EDC90000D; Thu, 18 May 2023 09:09:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F503900003; Thu, 18 May 2023 09:09:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8488190000D; Thu, 18 May 2023 09:09:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 68504900003 for ; Thu, 18 May 2023 09:09:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EE0B5A085A for ; Thu, 18 May 2023 13:09:00 +0000 (UTC) X-FDA: 80803406040.30.DAA0E9B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 4A6521400B0 for ; Thu, 18 May 2023 13:08:30 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aq+POVWb; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=lvlSLroEEg1VGuCDCeQ4xc3xFi52IK9mce9FvjeI+YDPm5ysx2CrU/OPLhTg1wHBlrZkEm ARMxflLv475tL+Iu5B62zn+o7mDzc5alYbpQEBsEi+CUmfraNFR5Yq0KUp5bTRMCmJVdpt qhPrKnEtixsMMYD4PoWz3hytLHNDuqM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415310; a=rsa-sha256; cv=none; b=wND1JMGZ9J7M2NAwn98RegPxvodFWDadT+u4fvPK68IPInwwpAMonta7LIeyDKidVOsG/U N3ewk779dtx7cydRTQr5clIPjGSCnchsOHfR+9gtFWSqgsNLKL0k9hrnzL/eAtBqJPsubG aPx4cZXy73x/Xoyb3FB+7bzu/fURzC8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aq+POVWb; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=aq+POVWbHkFInjUvPgnDwRjNUmXiO5qO+7Bj2k358/i6HJ/c2vOcGXhHYjcKDHb+nSBDlo iFFjfYEsLVskcsh3ZbfOtAOdVbqesegTE/U8EWZd+kQDWuaAWhMLqrihZCRNes1ELlmUkE J6v1ruAQQAnnhvkYZPKk3bjRzkCzl+4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-189-jUFF3OsoPBixAzERtA42UA-1; Thu, 18 May 2023 09:08:28 -0400 X-MC-Unique: jUFF3OsoPBixAzERtA42UA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1BD65101A551; Thu, 18 May 2023 13:08:27 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2DFAB63F5F; Thu, 18 May 2023 13:08:23 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v9 15/16] af_unix: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:12 +0100 Message-Id: <20230518130713.1515729-16-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Stat-Signature: 5j89c7z4rhqpf7e8d8f5ognnozscgcga X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 4A6521400B0 X-HE-Tag: 1684415310-321145 X-HE-Meta: U2FsdGVkX1/8falWH6X5/2RSpQzqTpJsPEK2OStMvaFRZCHFdQA+AKaufYssbnefmAz3NPGiRJC1EbVxL6FlEorAmhhZPPLC4C7jLyIM7WhhHmI1RtgT+wSBCn28/87hNIHVTHrAYxdORL4aUz9nHvK5MYZhCOoXAzH+d/FpFASJIdn4DvnOOi59/or+pYANnmzG2BYGj4NoFKv4nrD9jRGNihz8KyTQzMFrc25pXZcFM1p6a2Zl4D3Jc0PkzeyRRAy8WBjfpWueWaBMmKKLABNqWb5TtONIWpozrR+c9ZAEOqcDSisPD5MuRkX83g3k6CMpz7xjgSuhM+laofLWcG/85+uUD2t/tCpgDtk4QpFiK+EaFHY+8503G46o46lQeJutVZVYg3Fvkizk3z6UAusv9+1xnt8vyP/KZFWRlEDWp8cf4hgCe6gPDvcYxpJfK8JWbFSM/As/z3G6z4T1/3sREtZHbarpaMXXpTA/se+x4uYNMAb3VM6/lT8XW60kfMFE4EJahSzo+eE+rSH60MIua/cMnKjvw21K+4Zax3GHB+H6wXpPh+anLbcM1MKtVbs7krraR9tUJZzO9YufzE6mmNLISMPqX/K+bsyhQyyAEX60hgV0MAOBKfKDNnGqLwUowCHSD+WzULywSL6j4iBHdjsbFW4EnKgrtKQ417vCH25IPSpdhWebKbvoEMdM7dOp1jia1ZHUNxo6RAyytkC1wTcHVrdRzWMvdubBsiAuXWCRZpSF/Oc61FWMqFWF0WfnEc/+igveH/hKXs5tRqVkfzb07pOdNK8/jl3D8Q1Chec5+7BIY1ypMrsNGQhGdC/+vfKZeOAblROEq/E/MpuaS+c2goSeA+uKVh14lJgF6woArQ5PiIy/iPS8pS/FVjVsB55NGIU4yWKsuQET/wZpYdEa/evWnb6/lW9VpgNE9QmFcjqzaB+tqKi1wcc9Fa1BmVQ5UZAEe3hTBp6 2KtIgjVE jLPGIwBNZfK/1z3aFyYgU3vRUhasdoK5Qb5lF0CkA8InzlHr05/zEmc6fJBU/THZCJ/BuhSgPC1lsai3S9vnfIQ1DaEWerBk+ieUk3WVNbJRwaazpUuA520CsZOlAAi80KGwRbRsjAFF78my40wUykUxt3Sjn17ef0jMhQRXlMBjsWetLHQRonjzm5YQzCilxLXf3Mi5XrARr3attjeXg0iiaEnMDGQ9n7Prc1sltouZYM3aob8OKalyggC7WGhpBgW5oBP1JbKjUibdZ7qycVDktkCGeCdqWjRkVFV4OXcfvq5k8Ww9mHS4v0fiD6B6s0cVHAHm22Ii9FOImjitJmO5CfkYcDAcU4fRzFi0m3tP0cOl9og41ydUngITo56OCYBjxNxSEJB7uuwEjZXApqZ/6Xiv57p0aa5gVPfiqON1b8KtbRqKbME13ZsQlW5XWKADWzR64M3oK0krvHs6Pzni+Cy67GNeiOqOxZPm5ao/Qysc4s4t3RVj+4/kY1xd4Uyc9twHfgCRfYsI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/unix/af_unix.c | 49 +++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index dd55506b4632..976bc1c5e11b 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2200,19 +2200,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2230,24 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + err = skb_splice_from_iter(skb, &msg->msg_iter, size, + sk->sk_allocation); + if (err < 0) { + kfree_skb(skb); + goto out_err; + } + size = err; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Thu May 18 13:07:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13246756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9CDDC77B7D for ; Thu, 18 May 2023 13:08:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C268900005; Thu, 18 May 2023 09:08:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6733B900003; Thu, 18 May 2023 09:08:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53AE1900005; Thu, 18 May 2023 09:08:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 402FF900003 for ; Thu, 18 May 2023 09:08:46 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EEC581406B9 for ; Thu, 18 May 2023 13:08:45 +0000 (UTC) X-FDA: 80803405410.13.96B60E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 4A4192008B for ; Thu, 18 May 2023 13:08:37 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YLvX3OOm; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684415317; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=QKHZ8kQkMCrfo9Ae5SrTmCJCqzInoUeiqouYNBfKy2Z9cJBYUNrmkWuXuC8PgPLY/22fga EGhh5gz/uPX0gFleb4wyu6HPp+ioepvYonFGV6BHU/hqghreJ6a4K2Tp7OEUtOaZaMvzxU 8lwpEslXMjFceBhPvKroa9sD0qS7p1o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684415317; a=rsa-sha256; cv=none; b=vVnvi2itHAyKI47pquvlHkZU3BL3F1B11ee/eZW8+0VykT704NTjXi+vgJkuF+5wOs8MkG DRUq+EUzh5/558MQXV5q8Vzdsu9uLsqqsrZtg7BivIz1Ean8zpxfcBV7zrn+eufSmDalVO HcMtqXDVYcj5iUVGXf1/BIGHUEUXegY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YLvX3OOm; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415316; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=YLvX3OOm/mDq3qH2WsXSJxG/suIVJrmDMyAvl6PVd9CtEkB+HaFsXpznJa3BBTQ5Aeteyv exfxhhpDKtoLoWw1W6MyvmBG+GKz/1eAVaYQxECA1wFXIu3GKeeAN2xy+ZVLIEIqy2X+Ji sky8j4wpRUD2kWszUNyLgz6aeuDEh+g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-434-1jG8JoGMMZqLc75xYcIWrQ-1; Thu, 18 May 2023 09:08:32 -0400 X-MC-Unique: 1jG8JoGMMZqLc75xYcIWrQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2CE1384AF32; Thu, 18 May 2023 13:08:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 09D6E1121314; Thu, 18 May 2023 13:08:27 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v9 16/16] unix: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:13 +0100 Message-Id: <20230518130713.1515729-17-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Stat-Signature: u1w4c6rjumpxbpbyff5gczazasguump9 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 4A4192008B X-HE-Tag: 1684415317-596320 X-HE-Meta: U2FsdGVkX1/yfGP0HEJiZFgVBA7dIyd9XO5CzjuePOxT3g+NOZFQ4x3NQ6FoMWsGJEUqU+flxLNdVf/cRJ5IldP00icw5DAwRMmWzGuvTEWW8f04LaiSW8BHH2U7O78Jw0I3AzkIhL42nTQGSNZJvnzgriDASeYwZM/VlRYxWpvzL75Z2MG6qm8cqjrqdzD6GKRGvYw6hgpOMQF8MQloUer9T6LBSc043A4R2/MxJLDC+K3gG3CG1QJyt7K6wOfb2WjQ5u4TllBPKcgvdFCuTsQ3jgIx1Q8AZx3fq267nODRFGwu3kFY9mlN2Wz/c428m4uI9xFo56Vnvnr9x2ylcPOTHXpr//HpNzdG2BzTz9AtzTuX4BAdtMmDgx0XNJeBV0cdc2xgq9JXWyvzucoEqoIwhJwAhzH6N543omx0FhjmZT9eZ/LgeT5YwVWYtNxnnbqzGDLXfmR3IEg+U8hB0yPf2wbYukCsZdfJQUk+auEpOomiZQY2kDl7/IsRPRgwzHONafXCVjK4m3Lbp4oo0+qI+uMOlZXr7GqtWAa7YVnGe2tTDFButLogdL1XboEtAZ6FhPYGys59mvbcmacoVIyb9RlqcMcuvngo2UuI+ub50H8eOMAVBDanYoYpyzcVLt4heCWzrrvIo3/IdaumMUPpKTiBwWrOSvlAJdRpIKGMX7adHVZ5jrRPucvrsB8lhX6Si8miRBQMQ4pVSGs2G5+IhsMswXpWsZnxxUGGB5QZi23PliwFL6nuO3COkaqcQJh2MOPAd8hi7ib/G+TarMed/VEiPoHCOk+BwRA5KA/NLn0UWTsdlF0iWvvHdd/12N3VHdsYmqDhDtsY5ANe2R4oBmeVHJll65vC/mDwDH/dR4zeqbB4bbTWZOTCQN8HNRBB/lVYDRb9VNOajq/uGaSa8Z1NVeFoNF+fXfgpmgrg+9zF2jkYoDAqsBxVkZ1aOf1zMEgydrN/564YlBW EJZSlcXz eFYJYRioVD7OUunCqChVlFtIb0qXhGeFVcKVLX5oPMkxmyKvFDYsqMDAPxEAybtNhrdxqe5V0yzmYnhznM9TaI78FcHaxbaKfEoc2/CLXjhkbegp8FYzwW1vJK3qO6qKIX1Oc4UYeR9Oupy+fZZbU3Yg9GniEnprR0BkSXSeeGDDC2rgBOI72joMk5JVEO+0zxSWirfVNt3uwh4LqgtzQm3IyWjeoBnTKT02Q70pa7iFbIqG/BVncqsB2yAjmyS2JMUVZ+4eh31mQZRiYIB2ZCX/dhjeZKGW+8EwEB09bBArVPRHr8vuIO2wVaYn0jNhR1E7yUxGVZ4vnJ3x6xpbX6EPaYZ1pzhR2mUjIT/C3b6aUAAbTCfh+bj72CgwXAshvRqm6qS7YembSPD38620o5yXerhl/i4caBOZolCrIU0/9vid2LRpL/7ZFuc1xu4BP2z5anb2qypVNtFvsv4hyugQREa0uXncDCRCWGSaLOovtbE9++GydMRq7hNXeU+026pa3q+Di/Hfyp2A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert unix_stream_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 134 +++------------------------------------------ 1 file changed, 7 insertions(+), 127 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 976bc1c5e11b..115436ce1f8a 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1839,24 +1839,6 @@ static void maybe_add_creds(struct sk_buff *skb, const struct socket *sock, } } -static int maybe_init_creds(struct scm_cookie *scm, - struct socket *socket, - const struct sock *other) -{ - int err; - struct msghdr msg = { .msg_controllen = 0 }; - - err = scm_send(socket, &msg, scm, false); - if (err) - return err; - - if (unix_passcred_enabled(socket, other)) { - scm->pid = get_pid(task_tgid(current)); - current_uid_gid(&scm->creds.uid, &scm->creds.gid); - } - return err; -} - static bool unix_skb_scm_eq(struct sk_buff *skb, struct scm_cookie *scm) { @@ -2292,117 +2274,15 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, int offset, size_t size, int flags) { - int err; - bool send_sigpipe = false; - bool init_scm = true; - struct scm_cookie scm; - struct sock *other, *sk = socket->sk; - struct sk_buff *skb, *newskb = NULL, *tail = NULL; - - if (flags & MSG_OOB) - return -EOPNOTSUPP; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; - other = unix_peer(sk); - if (!other || sk->sk_state != TCP_ESTABLISHED) - return -ENOTCONN; - - if (false) { -alloc_skb: - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - newskb = sock_alloc_send_pskb(sk, 0, 0, flags & MSG_DONTWAIT, - &err, 0); - if (!newskb) - goto err; - } - - /* we must acquire iolock as we modify already present - * skbs in the sk_receive_queue and mess with skb->len - */ - err = mutex_lock_interruptible(&unix_sk(other)->iolock); - if (err) { - err = flags & MSG_DONTWAIT ? -EAGAIN : -ERESTARTSYS; - goto err; - } - - if (sk->sk_shutdown & SEND_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_unlock; - } - - unix_state_lock(other); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - if (sock_flag(other, SOCK_DEAD) || - other->sk_shutdown & RCV_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_state_unlock; - } - - if (init_scm) { - err = maybe_init_creds(&scm, socket, other); - if (err) - goto err_state_unlock; - init_scm = false; - } - - skb = skb_peek_tail(&other->sk_receive_queue); - if (tail && tail == skb) { - skb = newskb; - } else if (!skb || !unix_skb_scm_eq(skb, &scm)) { - if (newskb) { - skb = newskb; - } else { - tail = skb; - goto alloc_skb; - } - } else if (newskb) { - /* this is fast path, we don't necessarily need to - * call to kfree_skb even though with newskb == NULL - * this - does no harm - */ - consume_skb(newskb); - newskb = NULL; - } - - if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { - tail = skb; - goto alloc_skb; - } - - skb->len += size; - skb->data_len += size; - skb->truesize += size; - refcount_add(size, &sk->sk_wmem_alloc); - - if (newskb) { - err = unix_scm_to_skb(&scm, skb, false); - if (err) - goto err_state_unlock; - spin_lock(&other->sk_receive_queue.lock); - __skb_queue_tail(&other->sk_receive_queue, newskb); - spin_unlock(&other->sk_receive_queue.lock); - } - - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - - other->sk_data_ready(other); - scm_destroy(&scm); - return size; - -err_state_unlock: - unix_state_unlock(other); -err_unlock: - mutex_unlock(&unix_sk(other)->iolock); -err: - kfree_skb(newskb); - if (send_sigpipe && !(flags & MSG_NOSIGNAL)) - send_sig(SIGPIPE, current, 0); - if (!init_scm) - scm_destroy(&scm); - return err; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return unix_stream_sendmsg(socket, &msg, size); } static int unix_seqpacket_sendmsg(struct socket *sock, struct msghdr *msg,