From patchwork Thu Apr 6 09:42:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ED3CC77B6C for ; Thu, 6 Apr 2023 09:43:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C70166B007B; Thu, 6 Apr 2023 05:43:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C1FB16B007D; Thu, 6 Apr 2023 05:43:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE7636B007E; Thu, 6 Apr 2023 05:43:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A15F56B007B for ; Thu, 6 Apr 2023 05:43:06 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 712611A0D34 for ; Thu, 6 Apr 2023 09:43:06 +0000 (UTC) X-FDA: 80650477572.27.7C3E54D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id BEF42180016 for ; Thu, 6 Apr 2023 09:43:04 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dDASBm9X; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QAACn2mr7ooi1DdUlemHLPNq4itk2LGg3Qvic6fgj7U=; b=gZXkLhX0bPTLYTF+b+umP/LkrGCkdxFbqfP1eY5H5FNbQ4LScyAWTs+347GlmlQpkJdhLP PhhERu2zFZYa2iVF4k7D0/gOXa2wtKNCi7HGyuUKCkRUygR7Zoamml4xkZZXgnh8y2kvE7 C8XyRFEjtV2o8mmIWDHbSyKe8S7H03M= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dDASBm9X; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774184; a=rsa-sha256; cv=none; b=L/8uxhVhmzp66PUdtFaNYWqk7tCpZy6xucbRbHYrclbINdKcSjQOQXBdwmZ9HEmgXcxqKv vz9awGzM2FkJ1yJI6DeBJ5+Fd2mm2d2kj4bWdZDGJnS2rIXx/aqx4Db3/yBc2s+N5RrGeZ F1X4TFyZtAImzhwhQhfIjnrHry96gaw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774184; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QAACn2mr7ooi1DdUlemHLPNq4itk2LGg3Qvic6fgj7U=; b=dDASBm9XXEld27rIX5q3MIsbSoWgHLQBluwP3tYSzENIBldXIpKaIhVkr1Zhwgn3aLZ5RY ZSgIT9FSMa9UNaE9fSKJUrNoFZBk12DFn9L9uYYXUsCdZ9M978zUh2jcKAPIk4/mSwu3Tr sZ8GPRcdhtWGa2xYWZgJbyif0Y0Nmdc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-oFbn2Db-OmiMAeWnu6JDQg-1; Thu, 06 Apr 2023 05:42:53 -0400 X-MC-Unique: oFbn2Db-OmiMAeWnu6JDQg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 16D458996E2; Thu, 6 Apr 2023 09:42:53 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 06D00492C3E; Thu, 6 Apr 2023 09:42:50 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH net-next v5 01/19] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Thu, 6 Apr 2023 10:42:27 +0100 Message-Id: <20230406094245.3633290-2-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Stat-Signature: tr4ecds4u3wti7tew3rtnwwq1wr5kedy X-Rspam-User: X-Rspamd-Queue-Id: BEF42180016 X-Rspamd-Server: rspam06 X-HE-Tag: 1680774184-811549 X-HE-Meta: U2FsdGVkX18Yk8+wTqwkSDA+oDKCpqgAHwCg0f+qU9RYj3ellSSIqz7sI85mGaz+/Hqt8XI7hVNb8jgflZRZubyOtF4LEfesldyd8hbJYI2LxqRSWx0RVnrWjLSVqXamt7b9BeL2x/cf/iZlx1YRqKK4FFPcvot00LYzpc9iChj+jzkAFO6QY+7qQSm1qMttcb06Tx91TL+4YmtYYBXm1BztEUHN++wMzu13H3FvzGiO2W+ru7LnQyFda/E13RevsF8e5OUuk/cYGZR3nnHLL9vEov9+0m34rbfBENSgNTaVHrfdUkiva99SkkE39IqAwwoGHd7skvRzGFx61NGAJ4pGWwxVQrT2rHpiXQ5oFW9HRELRxmZM4AS3TyZaDA+nyOsINlJeUEKFpUQHBzeYzCEAvsK8AaNtAOJSbx3UWe5A0OHRmSRc7n5/jRR2rJDIR9hTfaPVUNEM0CrC7kdKlhJZYPGqcXd96fWf4w6zg/l43M1ptv9cZQUqR6uVA2mT5d7n2Xq8V2dHSVZkgcfACOJzxjxS2sqpBFEndgxZUaJf//XlYTBTKWOPlozbkSg6yP88q5ZKqhiLSIySMmsAWvruezscyI6NdeDN/8OeO9NAvE1inkCGrqT9teGr5yEKu3e3jNCzy/nJ9ZGORqmQK/8vMJbs0aknXM/0kh1N4VJdxmLM1h+T7JWZnsI9kpdVsSNZs+0OE6wzYKqgSN+5ZliZuRBDgRjQdQg8XqKLdY+cpiiDqgw/OQ0JIZ5FeKoVuFm5gQiK+Mh6E+1/9fJizFDXvf5CGsnYImJMUIz4UlsebzfKd5MezRdtVeOurt2cdwbfEtjdGTyIsD/wYqqlEX9Ay7wb4/T9iB3KOvxh7oknVGvaTWvONWTojs8XgyRE0gcvvej6Pa/kI23CIMgjNq9mpFb44Bo2hUiULmxu6gfTeSbnYFAmPQ/rMNHys/HjEDM8EIHLjeynbh7+4Oy vQ9BLazz lRNN45zG6iIEZvEjo4n3bPHZLCr+becc7EuYIqW7SD0sHKxLWR1wR49LbJHjEpKZd4P/AVxATWbo8J9tt1bBmmbxcOF29JadRcvx1GDNyQUwGhWqEzMPqu1RPjvv0anvXJ4W6G54HJu8f1R3nkcaduzHlIIPuyTxmL33BuqcNQezt37DhuytQHCaubqx+2FrXc94CIRUAsW/nTpFLIXEYp7PsC5wkcRLNF2WI2WpRFBXPCdXf9vZs7DmuhsJyRJce5mJfrkc7Qjpf3cNmz+xHzw38P5GdQ2JFT1Yzafqi9lQIle6AaP/82ciMoCNMyGmMbAMVOj1I3lWHQmO+x1sooF8dCw8nzMRBGvyVuf0M1PsFXJf6/+4S4rYxGq5gCchc6uPl/bbdBmYXW423dMVavwJ92Lt/yB0jfc4CiRszfj/IiVsYaEnAFO1VRxruQQYqNxb2O9zETAFPigi4juTXfJpnVk5/lj/4P6U/6KEDEfi5TI3NB81LW2FrZothC5ZhHHHo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/socket.h | 3 +++ net/socket.c | 2 ++ 2 files changed, 5 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/net/socket.c b/net/socket.c index 73e493da4589..b3fd3f7f7e03 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2136,6 +2136,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, } msg_sys->msg_flags = flags; + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) msg_sys->msg_flags |= MSG_DONTWAIT; /* From patchwork Thu Apr 6 09:42:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38314C76196 for ; Thu, 6 Apr 2023 09:43:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBE936B0078; Thu, 6 Apr 2023 05:43:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C6EB36B007B; Thu, 6 Apr 2023 05:43:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0E6A6B007D; Thu, 6 Apr 2023 05:43:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A1E756B0078 for ; Thu, 6 Apr 2023 05:43:02 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 709CAABF2B for ; Thu, 6 Apr 2023 09:43:02 +0000 (UTC) X-FDA: 80650477404.01.48ADCEA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id B348240013 for ; Thu, 6 Apr 2023 09:42:59 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IBV4M+Ia; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GgIr9+xYtjWCQCpR/5vL0XO1pMxYzzO4LkH470whdw8=; b=X9ZjGKT18puHzObSzQMKbqK4UNt/c1b1DFkFJCSM5PBW367qtg5jNw43DY+Liiw5ujG5b7 H+SdoX5QIivBXLBrshLZ+q2ZyburiOCNemQpWFZM+9+RV+dMFRvhsTGXFCrc/9srqNchB/ AbmQQMqYlOh/SmFskAW9xVmgMkOq1iM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IBV4M+Ia; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774179; a=rsa-sha256; cv=none; b=ZusyuTfzAz8JWxxVNI5FDvR5p6fhYVFEVNyPw+b8LMbIm2GYmKJ3aX/cLyukX5I/mhU1+w e4De55QZXK52UtN5IvnSN5SfKrkoH5QZEleQ/byAQoPxrNhQywokP93fdTcktFC+6K5kia nIJ2UuNP4H0Tki1ZjQWLXiLRO4cm5mk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GgIr9+xYtjWCQCpR/5vL0XO1pMxYzzO4LkH470whdw8=; b=IBV4M+Ia896kNzwgvm2MCK2+gyTrll/W2X5urSV166QePMAfb0lWscJkWVH96OW0yI6cpp jwgMdTGY98H3ubEtfnyBMzLN6hYbcbnOWEjcWtbUfgDTRk9xjmBWaL8gCfsNwzFEPL2ged flrNj+DI/Ylce3iZNmAEfm+13QILyao= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-451-1ShmMRvZOOedJ3J350iYHg-1; Thu, 06 Apr 2023 05:42:56 -0400 X-MC-Unique: 1ShmMRvZOOedJ3J350iYHg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D33758996E2; Thu, 6 Apr 2023 09:42:55 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id BE81240C83AC; Thu, 6 Apr 2023 09:42:53 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton Subject: [PATCH net-next v5 02/19] mm: Move the page fragment allocator from page_alloc.c into its own file Date: Thu, 6 Apr 2023 10:42:28 +0100 Message-Id: <20230406094245.3633290-3-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Stat-Signature: yz1hmx9krjua9mfgd97ddsy5nx9wxe5g X-Rspam-User: X-Rspamd-Queue-Id: B348240013 X-Rspamd-Server: rspam06 X-HE-Tag: 1680774179-673910 X-HE-Meta: U2FsdGVkX1+e/qI6DjgUaSZZe94N4dVVoGkIYD/D7YZef71RtGVDKsWaPKPDahRbyAjVwaXvTFicePgtBEVBkCRZ/Du5wvnxU3Rt+mXpms6gBxnG+cMHI65wW7W4u7Vvv0uoWpUkEPrwwTav2wed/Z4Y0klCoQkq7sH7p2BF2Xkz9tobh/9Dx+LoCxKiolCeZ/wzWZzVEW8SfgHYAgMElsL/3wkxAguP26Ob8HkJess+4X1ssFZQDrDJlqkxwnBjv0FHxYBc3wZVrI39+XY/b0LO3PCRvbrAPQU/mASb5dGSqsSjF+zeBi9C1oq1LfbycT+9Y63mAyI3CeHVa8TngYwBBa1ZAJgzTrkbpOMV1gYeZIYlrDjQGSGX0e3bBr9mTIQlNMQw7axEIhQsNgoJLRtjMD+0GbfmthS4KAox/FCT563JVPrGCefPlLmvFqgzCqwxgqrerglGttkkhTZK2qZjuA9PTsLGYlpsKuawRWTc0qbD1weiwCiLotyXHdChMlSB9Yuv9HpKaYxejNzFpflIG1q2yNm5NDEBmmlylpqfGV4ykJCbicOLK09Hk+ywUHl51Hvah+2D22YKkFecT4UN3Z62yAsX9EUdXIyMZW2f20IdiKevPVk0qacbixXud4Wl5LBSyrMBZoHUDsr6+40FfretHKlrlsdbRixP1w33goErauGzhAvaxW5O5YGzAbZQxCxQazjWzDHTjdHYZKRY5I33IQJGcq3X7C9aRVcWif2yjVp0N66OI12z1AO4hXRwPDAApr483wn1NPosG2mPeM5Rf3PfdSuoSiYAnesgDOgV4z/DJFKbV2Y1D6KT+b8Ij0i1lD8OxFwvNq/vNmOciytYM/2WzvA1AgUEfNXOMHDi4SRphafuiwYx+wwnTwYoaKhXxt+URuLx9SwJ4VVlOImd5IPP+7ONRF7ovpX2RT23ipy9yuOpw/Qc00+ilnMJM7no4x/3o2BMjup acnH6joH aJlGMXE2wIeDlAXanrNZYJX+ZaJsp4ZbfEUcuXV2TFzn1wNF+cQY+vELFoj4LhAWS/S/qkt+ovg4Y6rGH/I1TrZx5YsKs55tS2RmHM1bXA2967UJsSPx8FD+OaCtExA7N9jfKg8QDjpQtvGSMrr4DZaO820hcW8n4WK4wmrfY2IklzGey0htPH9TQTw2H+S8ZkTsuLtISgVI9XTp7/4STnQNUqkC1+PbKFX56SFynGJFnTuMKS9/U+xb+Fy1RksJg1ZjaHbwWruVrEKythPArNpCkUebGZabHwoRGl4DiBm+LETf1+IzLz79+/HydaanHNYw1cfS8omaodvP0yqj2Y/AjiQp6xixjm50MNuV6kSXAE2CVHW7qQx6ejQKNpC1zJrcHqIaYR5auJurwl3HLA+LQ23VBhegQ/lVAqefdfZxUq84V45dunBGH5de8a1tIjbBzVF8BIND4GtTED0qFGspB+U236dpqkW2Gpj/dg/WkwcO3oREEzd9AyxcV5a9PnP/Ixv5uzxWpiv5Z++VFFxI0rHK6zzZNwv5tBB5eYhCT9kYq3pmSYuWs3pYiSLzErm/n8OYFn0dpbWUVdyRKVOg/a0vexCdOnvlzhEGrNUBjaOkU3Td/BkfUsmcpdwSUx7Vz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the page fragment allocator from page_alloc.c into its own file preparatory to changing it. Signed-off-by: David Howells cc: Andrew Morton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- mm/Makefile | 2 +- mm/page_alloc.c | 126 ----------------------------------------- mm/page_frag_alloc.c | 131 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 127 deletions(-) create mode 100644 mm/page_frag_alloc.c diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..4e6dc12b4cbd 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -52,7 +52,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ - compaction.o \ + compaction.o page_frag_alloc.o \ interval_tree.o list_lru.o workingset.o \ debug.o gup.o mmap_lock.o $(mmu-y) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7136c36c5d01..d751e750c14b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5695,132 +5695,6 @@ void free_pages(unsigned long addr, unsigned int order) EXPORT_SYMBOL(free_pages); -/* - * Page Fragment: - * An arbitrary-length arbitrary-offset area of memory which resides - * within a 0 or higher order page. Multiple fragments within that page - * are individually refcounted, in the page's reference counter. - * - * The page_frag functions below provide a simple allocation framework for - * page fragments. This is used by the network stack and network device - * drivers to provide a backing region of memory for use as either an - * sk_buff->head, or to be used in the "frags" portion of skb_shared_info. - */ -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) -{ - struct page *page = NULL; - gfp_t gfp = gfp_mask; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; - - return page; -} - -void __page_frag_cache_drain(struct page *page, unsigned int count) -{ - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - - if (page_ref_sub_and_test(page, count)) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(__page_frag_cache_drain); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) -{ - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; - - if (unlikely(!nc->va)) { -refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) - return NULL; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); - - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; - } - - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (!page_ref_sub_and_test(page, nc->pagecnt_bias)) - goto refill; - - if (unlikely(nc->pfmemalloc)) { - free_the_page(page, compound_order(page)); - goto refill; - } - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); - - /* reset page count bias and offset to start of new frag */ - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - return NULL; - } - } - - nc->pagecnt_bias--; - offset &= align_mask; - nc->offset = offset; - - return nc->va + offset; -} -EXPORT_SYMBOL(page_frag_alloc_align); - -/* - * Frees a page fragment allocated out of either a compound or order 0 page. - */ -void page_frag_free(void *addr) -{ - struct page *page = virt_to_head_page(addr); - - if (unlikely(put_page_testzero(page))) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(page_frag_free); - static void *make_alloc_exact(unsigned long addr, unsigned int order, size_t size) { diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c new file mode 100644 index 000000000000..bee95824ef8f --- /dev/null +++ b/mm/page_frag_alloc.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Page fragment allocator + * + * Page Fragment: + * An arbitrary-length arbitrary-offset area of memory which resides within a + * 0 or higher order page. Multiple fragments within that page are + * individually refcounted, in the page's reference counter. + * + * The page_frag functions provide a simple allocation framework for page + * fragments. This is used by the network stack and network device drivers to + * provide a backing region of memory for use as either an sk_buff->head, or to + * be used in the "frags" portion of skb_shared_info. + */ + +#include +#include +#include + +static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) +{ + struct page *page = NULL; + gfp_t gfp = gfp_mask; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; +#endif + if (unlikely(!page)) + page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + + nc->va = page ? page_address(page) : NULL; + + return page; +} + +void __page_frag_cache_drain(struct page *page, unsigned int count) +{ + VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + + if (page_ref_sub_and_test(page, count - 1)) + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(__page_frag_cache_drain); + +void *page_frag_alloc_align(struct page_frag_cache *nc, + unsigned int fragsz, gfp_t gfp_mask, + unsigned int align_mask) +{ + unsigned int size = PAGE_SIZE; + struct page *page; + int offset; + + if (unlikely(!nc->va)) { +refill: + page = __page_frag_cache_refill(nc, gfp_mask); + if (!page) + return NULL; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* Even if we own the page, we do not use atomic_set(). + * This would break get_page_unless_zero() users. + */ + page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + + /* reset page count bias and offset to start of new frag */ + nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + nc->offset = size; + } + + offset = nc->offset - fragsz; + if (unlikely(offset < 0)) { + page = virt_to_page(nc->va); + + if (page_ref_count(page) != nc->pagecnt_bias) + goto refill; + if (unlikely(nc->pfmemalloc)) { + page_ref_sub(page, nc->pagecnt_bias - 1); + __free_pages(page, compound_order(page)); + goto refill; + } + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* OK, page count is 0, we can safely set it */ + set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + + /* reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = size - fragsz; + if (unlikely(offset < 0)) { + /* + * The caller is trying to allocate a fragment + * with fragsz > PAGE_SIZE but the cache isn't big + * enough to satisfy the request, this may + * happen in low memory conditions. + * We don't release the cache page because + * it could make memory pressure worse + * so we simply return NULL here. + */ + return NULL; + } + } + + nc->pagecnt_bias--; + offset &= align_mask; + nc->offset = offset; + + return nc->va + offset; +} +EXPORT_SYMBOL(page_frag_alloc_align); + +/* + * Frees a page fragment allocated out of either a compound or order 0 page. + */ +void page_frag_free(void *addr) +{ + struct page *page = virt_to_head_page(addr); + + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(page_frag_free); From patchwork Thu Apr 6 09:42:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85DEFC76196 for ; Thu, 6 Apr 2023 09:43:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 173536B007D; Thu, 6 Apr 2023 05:43:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FCD36B007E; Thu, 6 Apr 2023 05:43:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6A3D6B0080; Thu, 6 Apr 2023 05:43:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D94036B007D for ; Thu, 6 Apr 2023 05:43:08 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id ACA361605F1 for ; Thu, 6 Apr 2023 09:43:08 +0000 (UTC) X-FDA: 80650477656.07.2BCEB8A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id E946E140005 for ; Thu, 6 Apr 2023 09:43:06 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Cvl8nkD0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774187; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nVZzQDZ8oCv/Z/RSubd/I0Q5tsMRDz1XKi4/e/H81VM=; b=M0OqU7CecAAoGT8Uzy3Vk8qgxdlDFncQ+LqxtxGwnVnav/odAeNrHCh7yMJ24Tyjo/FV0+ cdg5N4+RLa+25e91GhUftXosvhROnIgUxDNKW3g2V8fisIaRGKpVppZh1H7NlGyq8dNcNC RtvjnTe9RYPl1R17gu2weCK8R0Ob4yg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Cvl8nkD0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774187; a=rsa-sha256; cv=none; b=SgsqlVOtYdfgd0qIrjtAKYgNOLzGHLfhV8NlSkiKLGWHSIFZ08YsdCruWoHLwigsl8VFsN 6m4TLLsuj0YHeQ5v/tsepICcPyj/wr2lnWEr8QjzlICSHtlXw6sPJGff/OjF8lgzz2n9MR z69E/+zt30cMvgc09GXS8Hhfg7HPzB8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774186; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nVZzQDZ8oCv/Z/RSubd/I0Q5tsMRDz1XKi4/e/H81VM=; b=Cvl8nkD0OfDwLKpexBdjuNklI7fZAJv0YlAPgAS8ghnJ575EjQFc53nZtlnO7y3DgsSeUW 6ZZ6dBqDtk0kL/nUHFbmNlf7bm+8DV/z6eXG3FedBq1ixLtDNzg1Tqfr33ixfz+OU6b9A2 JQ9rn0ForKSpWwP2XXj6P4Bfnp4J+qU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-385-efyz_S0XOm-fe6KgdgJSmg-1; Thu, 06 Apr 2023 05:43:03 -0400 X-MC-Unique: efyz_S0XOm-fe6KgdgJSmg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 873EC185A78B; Thu, 6 Apr 2023 09:43:00 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 71734492C14; Thu, 6 Apr 2023 09:42:56 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next v5 03/19] mm: Make the page_frag_cache allocator use multipage folios Date: Thu, 6 Apr 2023 10:42:29 +0100 Message-Id: <20230406094245.3633290-4-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: E946E140005 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: mfq7863nyyyz5cm8wnodmor6xasontsx X-HE-Tag: 1680774186-199708 X-HE-Meta: U2FsdGVkX1+eW7bw/oXPCw3Pd4P3iX0Xn4I1tPtYy8ul0hW58P8gmLoD3F7UI+WdJ0Wp/06ne+GFH1bTO8qCUmpGPrwDQkXWy4fl4c6e5L9QNmA6mMEyLe5TPvARHbRT8nTdIDsZmjxXL1GgdC4xM+pysNiKj8lMrsjsQfTbpFwQVZbQ9I86K9h+sYv568WMPP9cIa/lMnG20CyekcJDsNAoXZFKC0hTppqFf4AWzQXY8JsKeJwyiMPRjhXnYgOpk/5GFLWUBXGyMfiR+8/UXnWGExMpDWbNbADDtIOU5OjmwRKYUhlWIJ+72+oSKGloTvMrsi0aS5TK5WPyHRWuJ3Qi3OKrC4CYcAaC9jahGT4LFbp5OH3XKfIuulTTkf7/RlbU3JHDod7hAr4UPmV+IOecTdzLPNaQzVsgThah1kVKplc+FBiVYOu4Jqqsi4WcVJWw7Mf5t/i/6d04p4xSRkI9IykYBLiAf32fuLuaM8WSeDC7oUuJU05CyWgIdnoCxzyvKonSsoql3YpVa/OpvTfm7TeYZcvd0Wm7lypuppQMH2gkaWYYS2Qkzx7GjhAr0Ln/iUdEOZ0RxIivnpugY7A9IYevMxKCereucV0kv/Tn8yVI+j8kZNdV3B6O2UtQapHQc5aDb5kH++MjrzXugXUxVAzXu8YqnH0Li85HgXQtLawnXK+M+mgT4hjIxD+p0tas2hLffs+lO8EyEyjgDOmJsqVbl1tPG361dbm/Hck++fT/6kkPgBgyFS67vxdv+lArrbkSIrdbhfxCKeFvHFrPpWTdotVU/iiyf8pJtZkm6b5IZRs8Dlk81YapbIPBpe+44L/AFIBAzDG3bI1SexVmezHQUJ4jertrJtI1ctgBilxQW9qv3f0LI/egoV3EIP0OaxO5xqSMnt9dS+nAcs3r5+2PpJzxJ3c15JaWLQT8ckI+1k01nUnQF+Mq8xsBxpkG5tVvxUgClLOCtvV i5lrc81k SnvY/DYpZA7I22nMtURnrLDuLNiJE0yr81MmO3Xm6psJXb3wiCyDDaDshiG5rRyolfwARp4cjChn8BGmEQYk/2p7JK0ztJUtFW7etEYOnGpYuPEsi2RnR3lFtoV9Y38wAdM4e6YEwpwHc+0pqYPJWRJuuR1yv5ITDWYsW8tq51Gno2rfX1Te5CyLr94JFAnPpAHRllSZ44XfzV2lbgSU0hEcb48ClUksTDvwPZ8dvGK4Vabq1ZyEpeKrRsvs5fPM4oJUcoeoPvIrBimu5euDq1pWHze3pqCLGLumzQ2JKNc93OZxxqJtLCAyvEWveUaHWbewvDZt8eXEQB6amtiv+6DrkxfSTDo3TxuOIlygauwh4zBAG2q9GSggsV9gkhNcbSAEqVR4IUG9GPBsKZIlYsA4rOLqsjA/F55KHk6GzUcWB5e1eYjPHdYGT7vyztIj98d9xPFROlP9/bfF8RKtgFjkHLiYQlddpW9s1d/eEve0J2v2taDD8GAc0f3cltqDrqli5sLc5QEEkPl4K6/oBeyk/ueJuYICRbNT8gPYY7jlqK3U1LSUrPcNxpQkFQ5ASrJp6rTKUcmnBGnLUF0B5mx12xu4gEtV9GuArfBL9BqW6qA0MKcpUTviC+AO4YNNnuPD6JOeAlRkk4C3I3arBt8xq6so3ouxJpPN63vWzPlpT/+f/ceqNzGBB7dcyft5c0PSAairLsFJIC670TzlIp74MYqLxQJ0nf2ciJa1V/GVRAZK/l32zR+mKcY2Gab0BJ5Faa9eyG/rGmqY+A2C6uQVWF/mqt0mntyKtCwCy/3+l5Tv0sHaFOvXt0AtfFPeiLGmhmU9H5PvPoU1UBwSxtLsZR7KgGpL3MjLgHW4WZ/qRVVHDBCdq574CAaJdjbFqqfD7IZivfMwurv52yOl28/hAku8pM6mGQ5LZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change the page_frag_cache allocator to use multipage folios rather than groups of pages. This reduces page_frag_free to just a folio_put() or put_page(). Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- drivers/net/ethernet/google/gve/gve_main.c | 11 +-- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 15 +-- drivers/nvme/host/tcp.c | 7 +- drivers/nvme/target/tcp.c | 5 +- include/linux/gfp.h | 1 + include/linux/mm_types.h | 13 +-- mm/page_frag_alloc.c | 101 +++++++++++---------- 7 files changed, 65 insertions(+), 88 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 57ce74315eba..b2fc1a3e6340 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1263,17 +1263,10 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) static void gve_drain_page_cache(struct gve_priv *priv) { - struct page_frag_cache *nc; int i; - for (i = 0; i < priv->rx_cfg.num_queues; i++) { - nc = &priv->rx[i].page_cache; - if (nc->va) { - __page_frag_cache_drain(virt_to_page(nc->va), - nc->pagecnt_bias); - nc->va = NULL; - } - } + for (i = 0; i < priv->rx_cfg.num_queues; i++) + page_frag_cache_clear(&priv->rx[i].page_cache); } static int gve_open(struct net_device *dev) diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index 69fba29055e9..6ce532217777 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -286,7 +286,6 @@ mtk_wed_wo_queue_free(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) static void mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; int i; for (i = 0; i < q->n_desc; i++) { @@ -298,12 +297,7 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) entry->buf = NULL; } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_clear(&q->cache); } static void @@ -320,12 +314,7 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_clear(&q->cache); } static void diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 42c0598c31f2..76f12ac714b0 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1323,12 +1323,7 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } - + page_frag_cache_clear(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 66e8f9fd0ca7..ae871c31cf00 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -1438,7 +1438,6 @@ static void nvmet_tcp_free_cmd_data_in_buffers(struct nvmet_tcp_queue *queue) static void nvmet_tcp_release_queue_work(struct work_struct *w) { - struct page *page; struct nvmet_tcp_queue *queue = container_of(w, struct nvmet_tcp_queue, release_work); @@ -1460,9 +1459,7 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); + page_frag_cache_clear(&queue->pf_cache); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65a78773dcca..5e15384798eb 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -313,6 +313,7 @@ static inline void *page_frag_alloc(struct page_frag_cache *nc, { return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); } +void page_frag_cache_clear(struct page_frag_cache *nc); extern void page_frag_free(void *addr); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0722859c3647..49a70b3f44a9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) } struct page_frag_cache { - void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif + struct folio *folio; + unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; - bool pfmemalloc; + unsigned int pagecnt_bias; + bool pfmemalloc; }; typedef unsigned long vm_flags_t; diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index bee95824ef8f..9b138cb0e3a4 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,88 +16,95 @@ #include #include -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +/* + * Allocate a new folio for the frag cache. + */ +static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) { - struct page *page = NULL; + struct folio *folio = NULL; gfp_t gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; + gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); #endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; + if (unlikely(!folio)) + folio = folio_alloc(gfp, 0); - return page; + if (folio) + nc->folio = folio; + return folio; } void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + struct folio *folio = page_folio(page); - if (page_ref_sub_and_test(page, count - 1)) - __free_pages(page, compound_order(page)); + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + + folio_put_refs(folio, count); } EXPORT_SYMBOL(__page_frag_cache_drain); +void page_frag_cache_clear(struct page_frag_cache *nc) +{ + struct folio *folio = nc->folio; + + if (folio) { + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + folio_put_refs(folio, nc->pagecnt_bias); + nc->folio = NULL; + } + +} +EXPORT_SYMBOL(page_frag_cache_clear); + void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask) { - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; + struct folio *folio = nc->folio; + size_t offset; - if (unlikely(!nc->va)) { + if (unlikely(!folio)) { refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) + folio = page_frag_cache_refill(nc, gfp_mask); + if (!folio) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pfmemalloc = folio_is_pfmemalloc(folio); nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; + nc->offset = folio_size(folio); } - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (page_ref_count(page) != nc->pagecnt_bias) + offset = nc->offset; + if (unlikely(fragsz > offset)) { + /* Reuse the folio if everyone we gave it to has finished with it. */ + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { + nc->folio = NULL; goto refill; + } + if (unlikely(nc->pfmemalloc)) { - page_ref_sub(page, nc->pagecnt_bias - 1); - __free_pages(page, compound_order(page)); + __folio_put(folio); + nc->folio = NULL; goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { + offset = folio_size(folio); + if (unlikely(fragsz > offset)) { /* * The caller is trying to allocate a fragment * with fragsz > PAGE_SIZE but the cache isn't big @@ -107,15 +114,17 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, * it could make memory pressure worse * so we simply return NULL here. */ + nc->offset = offset; return NULL; } } nc->pagecnt_bias--; + offset -= fragsz; offset &= align_mask; nc->offset = offset; - return nc->va + offset; + return folio_address(folio) + offset; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -124,8 +133,6 @@ EXPORT_SYMBOL(page_frag_alloc_align); */ void page_frag_free(void *addr) { - struct page *page = virt_to_head_page(addr); - - __free_pages(page, compound_order(page)); + folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); From patchwork Thu Apr 6 09:42:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE848C77B6C for ; Thu, 6 Apr 2023 09:43:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F6366B007E; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CCF46B0080; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F91B6B0081; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1CAB16B007E for ; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E6D81A0D00 for ; Thu, 6 Apr 2023 09:43:12 +0000 (UTC) X-FDA: 80650477824.06.918D43C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 1DD7C160005 for ; Thu, 6 Apr 2023 09:43:10 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XDNsefxE; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vT0qsSDBYT0X8eSrHsK7EpxTL/1ZdmjGeRARxkvOlLA=; b=G89ABDGqhGGGRv9YyW0174q0mytIPr0jB5Sn8nThEp2v0anOhozJNur0eC8E6K7Wkwgvam +2Wc7zS0fTsqKC94Cv68ylpSTFP1kFGKb3UiCVgb9lqBDHm4dXydnNv/5RSSfbjuZwRxcq qiKsxMYS79oT1OKxU0AJDRljhQF4Se8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XDNsefxE; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774191; a=rsa-sha256; cv=none; b=5TjUijx0Y6fhC4g6aUe3TXDXx+/3gvxAozT4KLWp/UkU2dDeeePnME3FyAYHZm4e0gSFkP DstBhNyjQ3rpJKm+r9Qv06HkwFBV7frfplTH42XUXm7MziMKZLl20uIW+Rp9RPklAvOQVn VijAKnwezlkHUb5I3gsUGOng8TX6Lpk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vT0qsSDBYT0X8eSrHsK7EpxTL/1ZdmjGeRARxkvOlLA=; b=XDNsefxEFN2stmi869+BgCOBQuDOoNMXvELjwMrQQANtuhO8dl6UxYamQI6x0ZFA4bNUE5 nmqwedEOmYzFV6+cGNCikzI2rekOelHmc8LULQqoY4mX/PKEId+Za9XiNdxDtdL7vzm2M0 hoVVAotNVHilyAQ1LPZNPe8OpjZOvcs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-382-A7hI_nzRMuG-vsAxsqekbQ-1; Thu, 06 Apr 2023 05:43:06 -0400 X-MC-Unique: A7hI_nzRMuG-vsAxsqekbQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 472DF185A790; Thu, 6 Apr 2023 09:43:05 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BBC840AE2C0; Thu, 6 Apr 2023 09:43:01 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next v5 04/19] mm: Make the page_frag_cache allocator use per-cpu Date: Thu, 6 Apr 2023 10:42:30 +0100 Message-Id: <20230406094245.3633290-5-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1DD7C160005 X-Rspam-User: X-Stat-Signature: 1t869boscdm4c193kp4jokxxpq4o8edn X-HE-Tag: 1680774190-314292 X-HE-Meta: U2FsdGVkX1/KQhhHLC1SsATEqahiFybivhN2Gku+CAEZhl652rq031WSwWqlWI1/qvSILOW1nNsXzRrC7ZO2Es+Z+ePrMCZYGTc0IXsIL4jI29YjfvKfpqA1JdxC3BMhqBRdX5UG450LkFzHhon+pRpMoJeq7968A4G3DfScTMzl46M6sgneLEsV6mSavTwB60ION1b68ECXtxUVMmpRT0t+zqfujRVdcgL5oIcrRoCvqAdLiU3g0HTRgNvKM7ZBPhluV3QXKQshLh3a8MCgk4zqP/S9wXFW3c3ZsHFn5O/vVezuLEWiQwaKyfroz6U8QpmUJmUu/Sc50FjitnHV8N45IFny6u3cMAeVphD1wDe6WV9LDvY8cKmUPlSEmgQ50PKmVjWjY7sHa686nfVH6jKltZAjcLAsRUHgQ3J2r7OYYtjFrOULsCyC9aF87nz22bElh+wU3Y2d6lItybL9tVZVtA4iKlROo+XfUroWb6R8OhtMo3ud+NLRYA+eQ2L6BLochdyYK/pIbbjkNAqiJ9Z8851f7AuGREAfpP1D3mKCyggsJpzjmpslUOWvODsMxsbWrnZxDx/NuomNCm1t5nhkq2DlriNNux/eB8XW+2A3WIf1QGdkfI/qT2O5mAfEmjWAT74xV+QMGfBH/fiXTU/QE6hR8jQGf8626NL42RZ9kT6wD0E17no6Z+ZwByLLL+8iwXD6GplJqzegi32QrJZptpp2JW4lBnR+gdDKIIsOsd1MAZKVda2ENZycmVmRelRZyPMxH7ghBV9KQpcQIAVxfh61rOCInuNe79sySFDee11ieLLI/+fcYRZIShpTpd5jL1ePoAbkwNoSPbhpukaRBOCE0LwWwnEcsUIbRtVbUU81ciEuW3SCm7wE0a+VsNstDMYktqLyu8tt2MJRY3eZ943L4eL7BZVeKsjlgP+zxzI9QFiv1y7vOYUMTHIE1cqq4FgkbFncst9FNp3 s57/jeUT nsRkUvzNKwFQZQIikHsx+2uV19aifS0h/PTk1FMiefj/TW8rDhXVg48POKwVAGUWFd3ySmOc9A1QNTbk9HQRhxmdQiZB0nKyP5HNkzrZuTQLZIrS/X074G6Tk/ZA39MRaQH1JhL1yAErl0gSYJopWs0W22EikpIvqbFHH8tkJCwNH8a5L5c2Px0hZEaYyAG6i9dF4DoRRH5CYyCLzt2ocmyxIwOt9awsNQno137pyO52VQK66H9/MePocmznQdtfU9H7++bAoKxTCyt//eGdz4qZRNL3TmJcotdB6y4dzeKz7DgaOy9goobwbHhHA4vv69FJCAnkFhhSfWJtMddqBkNuQUoqqE38FhymIDRFNjkAY21uGCdsU2av0Zttng58kB+D+4pRoezOC2wHg5a5rrMwqQPDvqL0ENrtr0FhFSGykjIfDMjOaNusCZ9Fe1QYpX6dvIt3WWfZIboz2m8zHLjePp6kCW9gHkre1Xwcqq5Z7QfWFAJPwB8Rc4OCBKSB6XWtv5eLrDPjtinM0gjt3SgdRw1E/XKqRw/nyctpXk2W1TrWXUsi0zzRSSsTK9Q5W+4OR0lf9mDm1o8FBcEbi2FRO19keTfkiHnKUDWBaOqAOs0Q2MeDsnuqlDsnQyRI4/QmTCUK4K43tOWJc/TYCcK6fBI0wqtT1HX7UWlbYKCjwAMZ+csd+S5+Q0JEwHK7TiZ9WRGka802fQ2KCzONiThkzvFPmpBxCtX12Ok78UlKWvBT2SpAAMc2z0JRjgGHJjb2wp1iK5u37VsgJlZu9IiCdagWITypPZPh7T4Ou/JmDf/xS+DbdzR5zCv+eUuKo8YKe1Xbll/W0K5G66h26ZJQ+D9q9MglATwQC30aM6G6Tvx8ku/xO6tuZnIa3obmcQ/T96vbigqa//QSCz1GPjkCETr8/ocIsN9E6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator have a separate allocation bucket for each cpu to avoid racing. This means that no lock is required, other than preempt disablement, to allocate from it, though if a softirq wants to access it, then softirq disablement will need to be added. Make the NVMe, mediatek and GVE drivers pass in NULL to page_frag_cache() and use the default allocation buckets rather than defining their own. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- drivers/net/ethernet/google/gve/gve.h | 1 - drivers/net/ethernet/google/gve/gve_main.c | 9 - drivers/net/ethernet/google/gve/gve_rx.c | 2 +- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 8 +- drivers/net/ethernet/mediatek/mtk_wed_wo.h | 2 - drivers/nvme/host/tcp.c | 14 +- drivers/nvme/target/tcp.c | 19 +- include/linux/gfp.h | 18 +- mm/page_frag_alloc.c | 195 ++++++++++++++------- net/core/skbuff.c | 32 ++-- 10 files changed, 165 insertions(+), 135 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index e214b51d3c8b..5864d723210a 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -250,7 +250,6 @@ struct gve_rx_ring { struct xdp_rxq_info xdp_rxq; struct xdp_rxq_info xsk_rxq; struct xsk_buff_pool *xsk_pool; - struct page_frag_cache page_cache; /* Page cache to allocate XDP frames */ }; /* A TX desc ring entry */ diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index b2fc1a3e6340..6c835038a8eb 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1261,14 +1261,6 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) } } -static void gve_drain_page_cache(struct gve_priv *priv) -{ - int i; - - for (i = 0; i < priv->rx_cfg.num_queues; i++) - page_frag_cache_clear(&priv->rx[i].page_cache); -} - static int gve_open(struct net_device *dev) { struct gve_priv *priv = netdev_priv(dev); @@ -1352,7 +1344,6 @@ static int gve_close(struct net_device *dev) netif_carrier_off(dev); if (gve_get_device_rings_ok(priv)) { gve_turndown(priv); - gve_drain_page_cache(priv); err = gve_destroy_rings(priv); if (err) goto err; diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c index d1da7413dc4d..7ae8377c394f 100644 --- a/drivers/net/ethernet/google/gve/gve_rx.c +++ b/drivers/net/ethernet/google/gve/gve_rx.c @@ -634,7 +634,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx, total_len = headroom + SKB_DATA_ALIGN(len) + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - frame = page_frag_alloc(&rx->page_cache, total_len, GFP_ATOMIC); + frame = page_frag_alloc(NULL, total_len, GFP_ATOMIC); if (!frame) { u64_stats_update_begin(&rx->statss); rx->xdp_alloc_fails++; diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index 6ce532217777..859f34447f2f 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -143,7 +143,7 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q, dma_addr_t addr; void *buf; - buf = page_frag_alloc(&q->cache, q->buf_size, GFP_ATOMIC); + buf = page_frag_alloc(NULL, q->buf_size, GFP_ATOMIC); if (!buf) break; @@ -296,15 +296,11 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(entry->buf); entry->buf = NULL; } - - page_frag_cache_clear(&q->cache); } static void mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; - for (;;) { void *buf = mtk_wed_wo_dequeue(wo, q, NULL, true); @@ -313,8 +309,6 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - - page_frag_cache_clear(&q->cache); } static void diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.h b/drivers/net/ethernet/mediatek/mtk_wed_wo.h index dbcf42ce9173..6f940db67fb8 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.h +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.h @@ -210,8 +210,6 @@ struct mtk_wed_wo_queue_entry { struct mtk_wed_wo_queue { struct mtk_wed_wo_queue_regs regs; - struct page_frag_cache cache; - struct mtk_wed_wo_queue_desc *desc; dma_addr_t desc_dma; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 76f12ac714b0..5a92236db92a 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -147,8 +147,6 @@ struct nvme_tcp_queue { __le32 exp_ddgst; __le32 recv_ddgst; - struct page_frag_cache pf_cache; - void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); @@ -482,9 +480,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx]; u8 hdgst = nvme_tcp_hdgst_len(queue); - req->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + req->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!req->pdu) return -ENOMEM; @@ -1300,9 +1297,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) struct nvme_tcp_request *async = &ctrl->async_req; u8 hdgst = nvme_tcp_hdgst_len(queue); - async->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + async->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!async->pdu) return -ENOMEM; @@ -1312,7 +1308,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) { - struct page *page; struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; unsigned int noreclaim_flag; @@ -1323,7 +1318,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - page_frag_cache_clear(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index ae871c31cf00..d6cc557cc539 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -143,8 +143,6 @@ struct nvmet_tcp_queue { struct nvmet_tcp_cmd connect; - struct page_frag_cache pf_cache; - void (*data_ready)(struct sock *); void (*state_change)(struct sock *); void (*write_space)(struct sock *); @@ -1312,25 +1310,25 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue, c->queue = queue; c->req.port = queue->port->nport; - c->cmd_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->cmd_pdu = page_frag_alloc(NULL, sizeof(*c->cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->cmd_pdu) return -ENOMEM; c->req.cmd = &c->cmd_pdu->cmd; - c->rsp_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->rsp_pdu = page_frag_alloc(NULL, sizeof(*c->rsp_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->rsp_pdu) goto out_free_cmd; c->req.cqe = &c->rsp_pdu->cqe; - c->data_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->data_pdu = page_frag_alloc(NULL, sizeof(*c->data_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->data_pdu) goto out_free_rsp; - c->r2t_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->r2t_pdu = page_frag_alloc(NULL, sizeof(*c->r2t_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->r2t_pdu) goto out_free_data; @@ -1459,7 +1457,6 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - page_frag_cache_clear(&queue->pf_cache); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 5e15384798eb..b208ca315882 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -304,16 +304,18 @@ extern void free_pages(unsigned long addr, unsigned int order); struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); -extern void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask); - -static inline void *page_frag_alloc(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask) +extern void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, + unsigned long align_mask); +extern void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask); + +static inline void *page_frag_alloc(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp) { - return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); + return page_frag_alloc_align(frag_cache, fragsz, gfp, ULONG_MAX); } -void page_frag_cache_clear(struct page_frag_cache *nc); extern void page_frag_free(void *addr); diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index 9b138cb0e3a4..7844398afe26 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,25 +16,23 @@ #include #include +static DEFINE_PER_CPU(struct page_frag_cache, page_frag_default_allocator); + /* * Allocate a new folio for the frag cache. */ -static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +static struct folio *page_frag_cache_refill(gfp_t gfp) { - struct folio *folio = NULL; - gfp_t gfp = gfp_mask; + struct folio *folio; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); + folio = folio_alloc(gfp | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC, + PAGE_FRAG_CACHE_MAX_ORDER); + if (folio) + return folio; #endif - if (unlikely(!folio)) - folio = folio_alloc(gfp, 0); - if (folio) - nc->folio = folio; - return folio; + return folio_alloc(gfp, 0); } void __page_frag_cache_drain(struct page *page, unsigned int count) @@ -47,54 +45,68 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) } EXPORT_SYMBOL(__page_frag_cache_drain); -void page_frag_cache_clear(struct page_frag_cache *nc) -{ - struct folio *folio = nc->folio; - - if (folio) { - VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); - folio_put_refs(folio, nc->pagecnt_bias); - nc->folio = NULL; - } - -} -EXPORT_SYMBOL(page_frag_cache_clear); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) +/** + * page_frag_alloc_align - Allocate some memory for use in zerocopy + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The size of the fragment desired + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + * + * Allocate some memory for use with zerocopy where protocol bits have to be + * mixed in with spliced/zerocopied data. Unlike memory allocated from the + * slab, this memory's lifetime is purely dependent on the folio's refcount. + * + * The way it works is that a folio is allocated and fragments are broken off + * sequentially and returned to the caller with a ref until the folio no longer + * has enough spare space - at which point the allocator's ref is dropped and a + * new folio is allocated. The folio remains in existence until the last ref + * held by, say, an sk_buff is discarded and then the page is returned to the + * page allocator. + * + * Returns a pointer to the memory on success and -ENOMEM on allocation + * failure. + * + * The allocated memory should be disposed of with folio_put(). + */ +void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, unsigned long align_mask) { - struct folio *folio = nc->folio; + struct page_frag_cache *nc; + struct folio *folio, *spare = NULL; size_t offset; + void *p; - if (unlikely(!folio)) { -refill: - folio = page_frag_cache_refill(nc, gfp_mask); - if (!folio) - return NULL; + if (!frag_cache) + frag_cache = &page_frag_default_allocator; + if (WARN_ON_ONCE(fragsz == 0)) + fragsz = 1; + align_mask &= ~3UL; - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); - - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = folio_is_pfmemalloc(folio); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = folio_size(folio); + nc = get_cpu_ptr(frag_cache); +reload: + folio = nc->folio; + offset = nc->offset; +try_again: + + /* Make the allocation if there's sufficient space. */ + if (fragsz <= offset) { + nc->pagecnt_bias--; + offset = (offset - fragsz) & align_mask; + nc->offset = offset; + p = folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return p; } - offset = nc->offset; - if (unlikely(fragsz > offset)) { - /* Reuse the folio if everyone we gave it to has finished with it. */ - if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { - nc->folio = NULL; + /* Insufficient space - see if we can refurbish the current folio. */ + if (folio) { + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) goto refill; - } if (unlikely(nc->pfmemalloc)) { __folio_put(folio); - nc->folio = NULL; goto refill; } @@ -104,27 +116,56 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; offset = folio_size(folio); - if (unlikely(fragsz > offset)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - nc->offset = offset; + if (unlikely(fragsz > offset)) + goto frag_too_big; + goto try_again; + } + +refill: + if (!spare) { + nc->folio = NULL; + put_cpu_ptr(frag_cache); + + spare = page_frag_cache_refill(gfp); + if (!spare) return NULL; - } + + nc = get_cpu_ptr(frag_cache); + /* We may now be on a different cpu and/or someone else may + * have refilled it + */ + nc->pfmemalloc = folio_is_pfmemalloc(spare); + if (nc->folio) + goto reload; } - nc->pagecnt_bias--; - offset -= fragsz; - offset &= align_mask; + nc->folio = spare; + folio = spare; + spare = NULL; + + /* Even if we own the page, we do not use atomic_set(). This would + * break get_page_unless_zero() users. + */ + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + + /* Reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = folio_size(folio); + goto try_again; + +frag_too_big: + /* + * The caller is trying to allocate a fragment with fragsz > PAGE_SIZE + * but the cache isn't big enough to satisfy the request, this may + * happen in low memory conditions. We don't release the cache page + * because it could make memory pressure worse so we simply return NULL + * here. + */ nc->offset = offset; - - return folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return NULL; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -136,3 +177,25 @@ void page_frag_free(void *addr) folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); + +/** + * page_frag_memdup - Allocate a page fragment and duplicate some data into it + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The amount of memory to copy (maximum 1/2 page). + * @p: The source data to copy + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + */ +void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask) +{ + void *q; + + q = page_frag_alloc_align(frag_cache, fragsz, gfp, align_mask); + if (!q) + return q; + + return memcpy(q, p, fragsz); +} +EXPORT_SYMBOL(page_frag_memdup); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 050a875d09c5..3d05ed64b606 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -222,13 +222,13 @@ static void *page_frag_alloc_1k(struct page_frag_1k *nc, gfp_t gfp_mask) #endif struct napi_alloc_cache { - struct page_frag_cache page; struct page_frag_1k page_small; unsigned int skb_count; void *skb_cache[NAPI_SKB_CACHE_SIZE]; }; static DEFINE_PER_CPU(struct page_frag_cache, netdev_alloc_cache); +static DEFINE_PER_CPU(struct page_frag_cache, napi_frag_cache); static DEFINE_PER_CPU(struct napi_alloc_cache, napi_alloc_cache); /* Double check that napi_get_frags() allocates skbs with @@ -250,11 +250,9 @@ void napi_get_frags_check(struct napi_struct *napi) void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) { - struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); - fragsz = SKB_DATA_ALIGN(fragsz); - return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align_mask); } EXPORT_SYMBOL(__napi_alloc_frag_align); @@ -264,15 +262,12 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) fragsz = SKB_DATA_ALIGN(fragsz); if (in_hardirq() || irqs_disabled()) { - struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache); - - data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&netdev_alloc_cache, + fragsz, GFP_ATOMIC, align_mask); } else { - struct napi_alloc_cache *nc; - local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache); - data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&napi_frag_cache, + fragsz, GFP_ATOMIC, align_mask); local_bh_enable(); } return data; @@ -652,7 +647,6 @@ EXPORT_SYMBOL(__alloc_skb); struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_t gfp_mask) { - struct page_frag_cache *nc; struct sk_buff *skb; bool pfmemalloc; void *data; @@ -677,14 +671,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_mask |= __GFP_MEMALLOC; if (in_hardirq() || irqs_disabled()) { - nc = this_cpu_ptr(&netdev_alloc_cache); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&netdev_alloc_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } else { local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache.page); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); local_bh_enable(); } @@ -772,8 +764,8 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, } else { len = SKB_HEAD_ALIGN(len); - data = page_frag_alloc(&nc->page, len, gfp_mask); - pfmemalloc = nc->page.pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } if (unlikely(!data)) From patchwork Thu Apr 6 09:42:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB9FFC7618D for ; Thu, 6 Apr 2023 09:43:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D32936B0080; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE4316B0081; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAA536B0082; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A8BB56B0080 for ; Thu, 6 Apr 2023 05:43:13 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 756E4ACA35 for ; Thu, 6 Apr 2023 09:43:13 +0000 (UTC) X-FDA: 80650477866.05.651D95F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id CB2DB180016 for ; Thu, 6 Apr 2023 09:43:11 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=STGIi76j; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RIVP3lGRuM1KX32IU3V3PATNWV0YJv7UKTSfgC7+z8Q=; b=iaTmUrOWLqegCrkRnkgdd9jZ6NdheXhOaplwYOl7NsCV6RPTswUp7wF0Yi2h2RaNsD6Alx Ba5ixrq4M3bNsR4B+rFjxf7vcNsdts92o0f8u0l+vdm0MGNDQz4vx3nR7V9S/yu0Uc+PU4 AbSK4JxZK7mqSeofmu4D8ZRFD/vWitQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=STGIi76j; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774191; a=rsa-sha256; cv=none; b=X3i9WBcjXk9xzLSAiZkr/aQc6mjpDt/x7+t4MjYxFdoCLMABDIhRJtclMRMdF8JpwMhsiI SmxFhlAAZfYZWsDOfa/L+jmTvLVstdvh9UZhyLNXL+XR4yPO/+sn9wqt8WELUt5aRNWG7K 3pzIVITbgyydK61eEUwResBXcQ4cuxs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774191; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RIVP3lGRuM1KX32IU3V3PATNWV0YJv7UKTSfgC7+z8Q=; b=STGIi76jlT9RPl2SOaDizXjzu3RvQ9Nw42I92O5bjXLUGbn9Pud0HUhvEeEuW2k5Ojaem8 IlVPx+lYDvoYDmk/5CTToP5fkh29VXZHlp+1pFfga/cjks0PYjLbSCylMHQvB+W+YiCTfm /hyYFESWTKkcZEtGJg0xq9R3OyHky1Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-6-FyvdLlUmOc2cPKFSu5UMHw-1; Thu, 06 Apr 2023 05:43:08 -0400 X-MC-Unique: FyvdLlUmOc2cPKFSu5UMHw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0036F185A78F; Thu, 6 Apr 2023 09:43:08 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DECF11415117; Thu, 6 Apr 2023 09:43:05 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 05/19] tcp: Support MSG_SPLICE_PAGES Date: Thu, 6 Apr 2023 10:42:31 +0100 Message-Id: <20230406094245.3633290-6-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CB2DB180016 X-Rspam-User: X-Stat-Signature: ugw7kmcd4exgt5pexuqxuhk5h4nmf6mm X-HE-Tag: 1680774191-410006 X-HE-Meta: U2FsdGVkX19E3xD6JPko3GhP+EX03+SJ9kaBbI/SKCwa5cufQnfks8KJrgz7FcyHZ7QLKd9MJaoaMkfsG2gbNvaAz1+GnYHKxTODDIf4GboLxcYIzqlIvHZa0QJTFKhq+g52ejvRM3jNWCmcZ3YmAKNbvpns49FWwZTuLA/4gwPpgdnj1Y9HOgpoVVKwomTTp+R0e9BYLdUsINwUuRaag9c1ECFDi7QduhrrDFMGI3ZdKUaOUUPwI5wJCrb4rIXCUU7MoPPsZuHqhS+uEjGVqwAo+o5i1EbNJiT8/QZV8pec3Nb+OxVbyrXQSrBpj7jV+BDG+prjOo1Xb7hteC91sMBIx3pXOtjikVSgV/Q5An1yYeXKvxbDGc2sx3w3vl5RH2quXgYeZPer/iUpBTld4Dzgv4BhtXHUpZfEeEDuOwrrAht/GzrdT3ntdozPvIsY8UjoqJa80EdXP9X5wj8pddTcQz1Ihrs3lG8wxF+AN9LDKEHxdXvFe7xWM9C8npWjbm/lQ9i0+5rR3ZlF3y82kWvwSzyLAhfcP5N8W31MuW1JZDbKlZaOkj/CM3KeDdYRAWEnhHdWjxeruYhNgzM6KDbqoab6PPcfyGTQExXv/M7Yty9UIRD0lprVkBjzzl+h3BEtZ2uQf5AAtWn2+hJQhW8p0esacztJfYbFvI0xs44X9Sj85SsGh8xvP+QlSvCZujmFbY/eVQukVoRbjjmRA13BGFvaIaYLTtjC1jReRQra2u1Io4wauU9OpERt+z+IE/eCopNcgx7YFOVxY0xvs0ZoQM+RFw7idaCbGUXKimhVV8NONJkndz+Dq4zPRoIYTfvlali0IB1O+FIC6VoqdXJOew2b69HouX1JNTllDAua7Ot+Sqao92H8DZZDk99lwdE81aDkP3R8za2uF1PN67GVWdoxFKUDRsOYM149cBjeyToRqRC2nZDy009PbH5hCnHOKSGn/yapxv9bjeA wBw2mpX1 AQpwKQu+zWC5A0jC4RHnkWHiE/Rr+H1GAxmiqe7mZfKaHQdXDkTTQaGD/MRlh2hIx7XZB/EGhIPj/Xn5mdATs5B5Hbh3hj0sLLw93WDBy79Exg6VwHEuS3J5LP7X0caTjWmq3apefskkawqVzZJotTgbI78NEHrj4q+qDTfV/64hhjCqFRj/Tp7Dpx4h4r3209KwgCJouQQcWy5kpXQYobtoehWAQFUP3g2rVpGWwFxiVuy/J4t9W2DpDFP75TPJpOvPxLZsScFLzYStSAEwqikU4j04PM1kbhXpylKZR59yFZWjUuscvSHWb2+yG+WwW8KoeXCnxGyqAAOtX2bCcx0ZlI21TIMMVsbUOF40uPCl5Ge2uEigrUv7SEBmu5suvm5WvlXCPiB86gIgTlTGUTWpgobBrOyeTdhYemRPkhNIEMCayk9w29/NB7jQNieJymIhjJELX/91+sV4VryB/MQ0TLSTgTDs0p1WthP/IzvssTTlXuawkKTmHJXJ0J6jmS+5f X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 67 ++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 60 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index fd68d49490f2..510bacc7ce7b 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1221,7 +1221,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1232,17 +1232,22 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = 2; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1305,7 +1310,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1346,7 +1351,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1391,7 +1396,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == 1) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1412,6 +1417,54 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == 2) { + /* Splice in data. */ + struct page *page = NULL, **pages = &page; + size_t off = 0, part; + bool can_coalesce; + int i = skb_shinfo(skb)->nr_frags; + + copy = iov_iter_extract_pages(&msg->msg_iter, &pages, + copy, 1, 0, &off); + if (copy <= 0) { + err = copy ?: -EIO; + goto do_error; + } + + can_coalesce = skb_can_coalesce(skb, i, page, off); + if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { + tcp_mark_push(tp, skb); + iov_iter_revert(&msg->msg_iter, copy); + goto new_segment; + } + if (tcp_downgrade_zcopy_pure(sk, skb)) { + iov_iter_revert(&msg->msg_iter, copy); + goto wait_for_space; + } + + part = tcp_wmem_schedule(sk, copy); + iov_iter_revert(&msg->msg_iter, copy - part); + if (!part) + goto wait_for_space; + copy = part; + + if (can_coalesce) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + } else { + get_page(page); + skb_fill_page_desc_noacc(skb, i, page, off, copy); + } + page = NULL; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + skb->len += copy; + skb->data_len += copy; + skb->truesize += copy; + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); + } if (!copied) From patchwork Thu Apr 6 09:42:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C035C7618D for ; Thu, 6 Apr 2023 09:43:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC3C26B0082; Thu, 6 Apr 2023 05:43:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A72746B0083; Thu, 6 Apr 2023 05:43:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 914406B0085; Thu, 6 Apr 2023 05:43:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7EA686B0082 for ; Thu, 6 Apr 2023 05:43:17 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 580CCA0CB7 for ; Thu, 6 Apr 2023 09:43:17 +0000 (UTC) X-FDA: 80650478034.04.3EE4F36 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 827CC4001A for ; Thu, 6 Apr 2023 09:43:15 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IaVgFzp8; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=srNjm2pDsZ7bySZjGb738UmToW4CeRBJkACWB8iZ9Oo=; b=xAFckvK1h2nfHabl+XYnNajVG7TXBjKQ8ARtJuP1nsMvJuL9s/CtX00fIaQ69Cf9rGrDVu ONZrBZfYNWza1I4dgeowhlJBm4eWv6H1BBaWxRCQgiLp5XHHGI3zdTi6ERTQxXBTDsBzav mlu1bCMkPcbhLAmfp4e8rMjlgQyHqA8= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IaVgFzp8; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774195; a=rsa-sha256; cv=none; b=hjg2YHF58A2p7Cp1gYS16TM1z9jMilYOWOTNZqZk+2nqVCaosyoj7VNfn7uFIkO515uFEO vgGKIMg27roYmMDJZ8nNNrrLxWlvoOR71pDp74+FQ50yIUhr4ZCPqDe0qVONbKW8aS9gl2 +wKfqwugPPqcNQwlxX40dtz+DpuK63M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=srNjm2pDsZ7bySZjGb738UmToW4CeRBJkACWB8iZ9Oo=; b=IaVgFzp8oCASI2JULY6NPOutU8DimJ1ScIxFRsj632xKc2RJktkDtWqsEay6XUHSDGToun Zm4vjntvctYnF5Rku1bSYuS4sUasDV4RTRQ69IYMtV1m3qS5rTRNpQJMW9GfGnqueBtSI3 aK+9ngfZPyxs/4IbRgG96evqO1zbXyw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237-DOKvdq12P_Wm-Yg8pakj4Q-1; Thu, 06 Apr 2023 05:43:11 -0400 X-MC-Unique: DOKvdq12P_Wm-Yg8pakj4Q-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B992E38601AC; Thu, 6 Apr 2023 09:43:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7FC6492C14; Thu, 6 Apr 2023 09:43:08 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 06/19] tcp: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Thu, 6 Apr 2023 10:42:32 +0100 Message-Id: <20230406094245.3633290-7-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Stat-Signature: 1cwr6i8xha3h6u9k7pwc1acsfz586amr X-Rspam-User: X-Rspamd-Queue-Id: 827CC4001A X-Rspamd-Server: rspam06 X-HE-Tag: 1680774195-397146 X-HE-Meta: U2FsdGVkX1+xqSJH0hqm3rDi/+4wwc8MvcUyLpQTYKlOWb3oGMPjYkRRXD6b7nWG7CQ/3hs9iqcccdnwSqKcKwkMaqRv9S5wr2rh2JUBvCV+lpnteV/UDi9BBxoKTywXTv1YwLeB4B167Xstx7eJwPiKZowUCH+Db9ECCzlndQtQglO8/TcDW66YK+HNSHwdQJxL26n+JB5V9eBfrXyEP87cHxZZk0ECIZ1WeK5LeHQMchDJbk0t9FaprGEieK7Pg0w9m7wazR4etnwOVry0pqaTg3mmnYO3xCVrcpTh8rD2ZOpYB5BOdCPOx4n1VwDbnr6zJ+kGL5kKfSw65SDHkurI4fJmAFJY7ibCLWY5qfdTOcXHab5e8XCaobM26l0qY1TJfQzL3otNGTZtjvJUS3uD/fEOXCfB133Np/AVPoqCAyRoF69TWLOtaSX/J3gemrbmmLj4Bgl4thc3utFCYOHSd/5tY9tRYXAnWi1xT4rUiTholKEnHv7iYVjd2chui6Tb3wSRx+7c43fviWoWJWgEaJ7q1RbCTPSgMz6qWSNDNKFMNsZ+eGPAS/AjHGf8E8qPO0LS9YtFtit35mzZH39hyiFSE6F51BNeh13J8MuBPfSNYcQsgGsJpfOKG4jiuvvhdVvhJhRhhZxtxQnIVhXgII0MIO/ADP/HXrt0SUBVRRAtwr500eGXE5E2wTiFJziCYnvgjoE3qbfyR43HRYXfUgfsrY0QneetQtdnLSsI1zKNiZY44/QlMz75A6rhgXCnWX18n553R5U2CtLXVPzy19xx/7B2mp+acqdfuxZdZht/rcFkpCcFLyjTNTkcb3DVCImPizr0pae547zaVzF/aVjzshS5Q37eBwu2NTg26GIot3IpYs9A+1kNZ9cCD3FCitPsl6+zS0OVSOxrCGd1nGE2WUvh9BxV6wMwvTXivnqedHcAJ/hbJid2A9T8Ju48ppQClX7J5kHnuaH GS+PA/Np ELkRlzlxAZdPZR0DczZGvG7ByYkTS03O9XV0GuROajZXi/MU27LqCa3nb1WsUKl46kNZjG2ghDlYeIDTqNrkK6Ez6y+X83OS3ciIjT9vKYjRB2dKYgM9qXw9T0SDg4fq/zDByL9tlVDsAiB/f8i+swCbfp54z8ymwFGU5Jc9GN0scSwg9190nrFGtmecrTzkKySZ4wJjY3vW90YBwrNF5EU1hdIw/z4D6F6IEfUnsKfNtveRb+in66Ul200zx4SJZtIqJ5+GdOVLW93g11L3m7MeqchJ56rJfWloktKQAgE64/9lnnCi9eQWmbbxQeurhRXm6B+hslGQWwRVqvlH3J1KE20Y4G7D2y078h9Ql9LdfpVwSvlMInA2O781v3AqA92EiO5biMU0xyHp9g8vdXnFkiROjhMC/G65uaMYK/Jdif+T9VXX6hqThI66IO00yTlpmrqk9G4ZTONhIFMkiQE6iw8tBxtr37lP1maTRvmP9PocJOz2Gur3UeW6I6omnMd99 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make tcp_sendmsg() copy it. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 510bacc7ce7b..238a8ad6527c 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1418,10 +1418,10 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; copy = err; } else if (zc == 2) { - /* Splice in data. */ + /* Splice in data if we can; copy if we can't. */ struct page *page = NULL, **pages = &page; size_t off = 0, part; - bool can_coalesce; + bool can_coalesce, put = false; int i = skb_shinfo(skb)->nr_frags; copy = iov_iter_extract_pages(&msg->msg_iter, &pages, @@ -1448,12 +1448,34 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto wait_for_space; copy = part; + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, copy); + err = copy ?: -ENOMEM; + goto do_error; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + can_coalesce = false; + } + if (can_coalesce) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); } else { - get_page(page); + if (!put) + get_page(page); + put = false; skb_fill_page_desc_noacc(skb, i, page, off, copy); } + if (put) + put_page(page); page = NULL; if (!(flags & MSG_NO_SHARED_FRAGS)) From patchwork Thu Apr 6 09:42:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CB29C77B6C for ; Thu, 6 Apr 2023 09:43:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F8D46B0083; Thu, 6 Apr 2023 05:43:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AAC46B0085; Thu, 6 Apr 2023 05:43:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 123C56B0087; Thu, 6 Apr 2023 05:43:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 01FE56B0083 for ; Thu, 6 Apr 2023 05:43:20 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B4DCC1C6421 for ; Thu, 6 Apr 2023 09:43:19 +0000 (UTC) X-FDA: 80650478118.16.4017905 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 0F7D5180010 for ; Thu, 6 Apr 2023 09:43:17 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Wmx4DUDm; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Ie/FfVlAXyyE3lC0L+B6Yd0gloGht93FV/bBiMRDfo=; b=FjPyKwZnESgsVGVIZxyYVD6NqE/OwSJdZtYv2p5/A2QkoMisbvylY1yOgVniZvc7dm3cnB BzP0ziFdAFwIAKqxyTneSHF7y+SVRcyrULYeLDssuPc1J6OFkcOA27UrVruXoG0XIjrRFT lR8X0F0lTZFQdo7evmOUjJs+4LNxT0g= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Wmx4DUDm; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774198; a=rsa-sha256; cv=none; b=RZGeb+nCW/rX/S6VT2gg1uJ2PqVbpAJ/KqBE8qStgWvw/2vB7UrKHrfVEvEVLe9AqDOorP Ux/RxH6hsr+VjcxlS0iNtuhGegBwJZnX605Xv23uDlezV/LWBk8rs5pUuxgfY1ZSz/6/HZ qbYIUA4LTflLeqlVnUFUb6eJ/AnzUTs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Ie/FfVlAXyyE3lC0L+B6Yd0gloGht93FV/bBiMRDfo=; b=Wmx4DUDmx1TMpiql/s2R+UFE7pA6xBqAZspaoqv1/3MhvGzKmXupyJeVk2n/QiwKoqPy96 iz7zCb8RLss5gcg9zagFX7DasQSUARAzjSmB0JwbjIBXuRlDjKN8v0xMBVajYUwTubsWiB wf1RdeSX4nMY/V9Dp01VTq8+fa7AUE4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-jsdlZo8rMCGqC7FKxuV6Bg-1; Thu, 06 Apr 2023 05:43:14 -0400 X-MC-Unique: jsdlZo8rMCGqC7FKxuV6Bg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 684AC1C0433B; Thu, 6 Apr 2023 09:43:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 55D182166B26; Thu, 6 Apr 2023 09:43:11 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 07/19] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Thu, 6 Apr 2023 10:42:33 +0100 Message-Id: <20230406094245.3633290-8-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Stat-Signature: a7edd7wsrc4n7ybmgrs6apo9tfeg4qxi X-Rspam-User: X-Rspamd-Queue-Id: 0F7D5180010 X-Rspamd-Server: rspam06 X-HE-Tag: 1680774197-667873 X-HE-Meta: U2FsdGVkX1+ESz41o7yIKagHHj0vqD8eAUKDfo8/0fyjjvAaARu4s9s0meytz0Bm71F2G5gB6tbzTvTtZ3NFZQvBDgh8VPfg6CvInYmaZXHa318ujEb8wzs7oYfIaylJbaBbbWfRa/MQyPeJTWc3plJDKiOINhtWTT2IEeUGeJ7exRomna0S8ZHenE30WNnSW1QaV2qszR47N2JNw7mHAKtO9PXHOTO/ZTAU+wjPOAAfAqm0hCJ++9SMdLNIQtB23uhjgm91GKms6sMoNtm3vuOFRa0mbE6Dlw03W3KRABnb0Jmn+lsqIAm4LCiHghfpHFeibpUhH5KccnAUgWIJnbx8yqbD43mVkL9T9wkgMgWKRshqnuQZivvvVhCWe5C5KP5tiIPRS5Uzv6F4/I8XSYOWVuAqkNCT4NuMp8JXoQzbrILm6BLgG5wCvuwDVB52JI6mzwR0sg0v19yH67xjg5SAw5Ad3snEwoDCDVZdPt6noizLm3coFPh409ZJhMvi2eY/gLdhPE27Wry198edY3G7IbYdGhUHX6gb9fsQlI0irJKlN28wZn6UFssZL4K55C9KO/Ed0CW4BoUf6PAqqr+ma3118Wv9Y/xgSp7YmG08kmfTrH4kIKbd6LQcpt8XGRY8QwRXWGwXWGjogmKB1ykQTX/bFfeMXFjYU8AgCnmaWNHGvXxigKZlP99u/PebW5DUTuufP188wjQw6Ga+ZCcQJblNPNroYkh7pXYfENGIfxOQIftvI2d7tfmUrtJFsGY2S1yNVYEm5jfz1m5Lt4FqNK2mXYSUBvpPYBFxFdqoZYrizepIRDGUAU0yJxm5+xo9uk6kaSCqs//u7kdjPJqnSU0Fq3YLcH7VZK4F0W9DZj4pUYQzdZshAiWfyVcAw8wh2Q4FA2BoI4KLL4oV7u4HTVoy6Lzm3YSDxWMqiTrv9xSEtqBNrQYH2oKRISLPrW1w2rTNQho/OR+8OJG VfJoeglN u4SE0ri5wz+/Km7bIvP2spY3ln4tHzCDAGhy7W3B5HfDYqXyD2su0ag4ZjlMtxFeWNrZjKl8l7AoQVGBpNvm3Dkmqgx1LY0J3kMw1HO+yAA10R10J39fnjL91rXbTwCfWgNNOW4xb7LIzcaBpU9dOSsTacknWPqAoWp0IvBLd1/xuzTSKZvOe8q+OngSkFwVu8VFuqjlfC/L8FpQjUm3BkCnM1kkyErP+iEQlbmE5qjqwALiu1PuY8XJZyjJPSOZZ9SpIDJxaoNEwbLZ8fpxmAwXhbiw/V9cj28oEyl3Q+NkT94xplxoMLq9sJe5T8AS6S5FsVIkLHsTrYxLJ81gDZDvZImzUZYtmMIiu57muFacPbcZ8AUpjEypNODzxuZxQ80TQE6TNxpeeYphOQddKQE5kvFhk3Ej18BTe0/bNmQ/i7MtnaaiVVvEWuiAJiYrHQj5IbVZ9CU4KwlQ+lkV2cDi+nXXKypLY5gGPl4prCXPoxuYzfpGXjKfL44tovVxiNdmt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 238a8ad6527c..a8a4ace8b3da 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -972,163 +972,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Thu Apr 6 09:42:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D58FC77B6C for ; Thu, 6 Apr 2023 09:43:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B37FE6B0085; Thu, 6 Apr 2023 05:43:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B0E196B0087; Thu, 6 Apr 2023 05:43:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AEC56B0088; Thu, 6 Apr 2023 05:43:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8C8436B0085 for ; Thu, 6 Apr 2023 05:43:25 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 666B6A0EBA for ; Thu, 6 Apr 2023 09:43:25 +0000 (UTC) X-FDA: 80650478370.04.74146F6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 95D10100002 for ; Thu, 6 Apr 2023 09:43:23 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AFGlLihX; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rjJlYTxOmtT7wdKdf7S5Ju9QN7AtCPTA0VxjqyvYpeA=; b=LcNxKdocrBtDx2UAaMmNIeKjKf0T2Oa7I8HVLtUdNt0MqqQlofLos/3f8vtIhv7qThgVD0 MqE3wkE30Zraj5x4d0Z666oDSTKLOuEDfTbQ3arzDw022mbOWmBBo5FxoiBUhLvDUq4Xer jl6XqfZMYRSdMNehNPERaw38jkyXOrA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AFGlLihX; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774203; a=rsa-sha256; cv=none; b=uJtf92AeGFJ8TZPmJLva0sl3mwH2Cfcf7Rw51qm7UQo5iMk6VOx0np7UNTdqZqBWUk8m8Y 4Vr42SAx4hFJQDmpipBCTeuStxfMAGBCkd378E/tvH9gzBu+NMF2vcuM8tNOwFoHjRmJO9 YdwAnpDm3YIDA2HXPnOQS7krg1M5ItY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rjJlYTxOmtT7wdKdf7S5Ju9QN7AtCPTA0VxjqyvYpeA=; b=AFGlLihX6O6GlDdSb+De7CjBMf9pvdmQZMzSiO1X4SGuZZTdUmXm/VoewXURCI6HLOR9/g 0+snB++L0ew+Xbw1d+KevFsbngYSPxMfjLCs0eKlcMP3cuEF5ZxKMhkDryUjiCN3hEDHaR EEusINXflIBBpyY34jVWbFjslZx2cfE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-3-tBtsn0JjNXKzzGvLcjsJcg-1; Thu, 06 Apr 2023 05:43:17 -0400 X-MC-Unique: tBtsn0JjNXKzzGvLcjsJcg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9072B2800486; Thu, 6 Apr 2023 09:43:16 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 33CEEC12901; Thu, 6 Apr 2023 09:43:14 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , David Ahern , bpf@vger.kernel.org Subject: [PATCH net-next v5 08/19] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Thu, 6 Apr 2023 10:42:34 +0100 Message-Id: <20230406094245.3633290-9-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 95D10100002 X-Stat-Signature: b1dxp548oytid1fe53inp66e6hzgcgqo X-HE-Tag: 1680774203-61238 X-HE-Meta: U2FsdGVkX19zPSdeGFMa+X2vVm3pQOOf8eeG3laKS6fQS1pI6KtVxmgqRCvgRV1VC+ARGtBzXYK627FoqI9ss0ZKPPnl/qeewy368gNSfCQz2CqC15qKLIiS3+N/eW8elQMQQnFw9+1XSr+Ah1iegJgxRYbr4r0QM6Ez2YGRJZzBJ35eCM/Tmb7sjgeXAJLTiVVAskKNe8rlS/sy1tqPwWkNxFJpdBsNlurk3601eno85QRqkzlVYO7clB0OC0Im64/v1cSToqM+MbhOSxXZfO6TOK/h+p7U8g2NWiWvzJ/spvz//HSGdqi0BMLrJ24pyIXUVtwUQ6a5omnFcam96gIkI6w7AOK7cUsuJ4pYxFsNkGgdP7vP7/kwAzT9u+0n3EafzVlKk1j7buVl/QPqyIntOQdiWwCmOIzLPNLObiwvBFnlCl7yyOz21DqyO3uCcFAn3Z5x32/Ke5l5lzTVO9EPE6/Lf/+Lptque6GDNvquxGiKmYhL68L924fNUhHPiQCcPTP54VMDayBxRvN3bIGBnY9VjzBDy65jyvXlQ8lMDsv+tDZr7KkVAAYtoboKHk6MtRRiMaCo0+KgFjw8ari4cTW9mL/TY0GAhnELR27P9Cjcp/RhiQObdez3i+b+XtVzTTJGTLvvu+amQqFJG8luVRMVaIqvdqkYum1TzVSRGfUB3Cn6M0760VBeoHb4YB4BlP2Qwp/9WTR5MZO5igRcLy6I2hUjjW5aAibmPvlRyuKbU7D26Gzr8l7h91fahzhCjqITw3rSjN0QkAiHiQ4Br7IDX9bDvC1ga072ctySEQBNRaO69bXbZjqxiU6T3JZonmdHUq8m7egLN84iPhZhz9Pbh7EQMFXHJmLb222UMj2aRmPlOluUB30db7D6QEVDdL/O8tvCMo8fLOfakNbgm9cVAgMpImMVLP3CtjCXOz7J8xOKyafLc6GuRDDc47VZKslU2CY+mQffNqu 6VasJIj9 FQV9Op4i3C/sQU8Lt1uz2lEgqkseXA2qKAq5+05Za1N1w1Ir3hPcCBRIsO/wxpIoAaSVqBhAfYPx63u0A7lleOcpPahcvcPvJ4TK1nQzavfpe3sjAYJW2oyu0+K2I5ywE26wpI5RRzPHDTaohT7eGkIf25PEB4lCJnrjPB/QPB1os0aRnoAG/ta1hCYbHGI/4dUDn3y5H1qGf+bToQ0vzdbNxMcCVdQbjuDRYiUX/zVvr/iLf7FaNDacqsX7m+LUsafDns4kXfLjo86t9+Ktw5SvNDuAuP9POjaEQCWJP/u2MrrrAwP6yAsbwFGxliMgNkhZu4y3+e4IS/EJjXr5+MD6y92Bnmz2z9D1I60eYi5ujRmkjVxvoQR1il9bUsAi15HlqBEXjU8qds2lxnXWAo5ED8DbMyJ8mczV8C3d7aOyM/QsDayHuLifnLGWJuUwQDLqrxpc4/ih0ybPTN4wSNGlBfI/A4mOwr+OzAAh4EwMgIeIeZ2Gf9H+2f+xp3TiFDWXA/gGUvyieKLDRbkrzPHEE6GO9o9ocTK8aVNrvGECPLtteHcc6w08Hbw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index ebf917511937..24bfb885777e 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Thu Apr 6 09:42:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82F21C76196 for ; Thu, 6 Apr 2023 09:43:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EABD6B007E; Thu, 6 Apr 2023 05:43:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 199A46B0087; Thu, 6 Apr 2023 05:43:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 017E66B0088; Thu, 6 Apr 2023 05:43:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EA2836B007E for ; Thu, 6 Apr 2023 05:43:27 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B31A0AB20E for ; Thu, 6 Apr 2023 09:43:27 +0000 (UTC) X-FDA: 80650478454.14.E24866E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 83BD840013 for ; Thu, 6 Apr 2023 09:43:24 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HLudmGwi; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=iRwawby9bmMsRKxGUlTfZs6PX+zTT51MpoJby/A5nEg9E8uxRncKjxAO8Eewb30/pQsg2T TGKoKxJf/TfD7aADlNiXKPTdr52ASKCKTAOfsd/YgQG8FGd4qIxj6G6H9I4EwC/t+bbiL0 mJnXnfxhwlz8EKOHaPYYjtDxjFAPH2c= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HLudmGwi; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774204; a=rsa-sha256; cv=none; b=Ce+cN58JcSZee5VzDa1x+AHs249EX0Tmayq4J3FImT31b6N4Ctt3U6OM0aR2YjyHGskxaD y9tmiv804SLU5jqvELC5ZEjE/zyvzw+FtbUQb7797pu7JDydoOMklEs3Z979Ik1GYV6p5Y H7kzzP9HZJlPEYLl7yQOIGexeWx7TvA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=HLudmGwizP9wTKVQyySHfQDH0UjwNOttKjHzOkaqqvJbDFCxXGmSpESKW8iR77ZzpVHeqe Je8vA03DwFaH4s4M8WvJt1TzxiqNP9LCowo1qkHCglTqgQHDnsNcNezlSsEqB3GSruPOsZ 8TaUy1ACRiJKyTYlRdAL9nfJdH+zCBs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-119-jk5GdTXgMCaItZ0dnoKllg-1; Thu, 06 Apr 2023 05:43:20 -0400 X-MC-Unique: jk5GdTXgMCaItZ0dnoKllg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 884003C0ED6C; Thu, 6 Apr 2023 09:43:19 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2FFAD2166B26; Thu, 6 Apr 2023 09:43:17 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu , David Ahern Subject: [PATCH net-next v5 09/19] espintcp: Inline do_tcp_sendpages() Date: Thu, 6 Apr 2023 10:42:35 +0100 Message-Id: <20230406094245.3633290-10-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 83BD840013 X-Stat-Signature: hqzbuwtbhmmogwgcbp8qcrd9z5xt7yp1 X-HE-Tag: 1680774204-979653 X-HE-Meta: U2FsdGVkX19wxyr1yUx6HteftzefHlylb3YPYP+LohUJmhgqJe8yQZSOoLxJcbY09wdnN+EqOwpPDyy1t25/bnF5Y2Fcz0r1CLa3UlmjVqon60OkEQmsTpMS/S/cvcLFaypG14sD/RrfY7SHXXee6ye57OksP7BVDTa3/ee7h2DtT6308xNPETcUH9tXCII2XYYbUoshG23hIehc7uJL62tzJuXtUgk5JSypB4cXej4nk7q2to1bCtIQ0jDM7UJdue/XcavUKFz3GHPgNqt6yCHy7q9a8pbS+CNe0deHLmGFbNSqRMifuaD1N6DAYh9d5Jiw3xrsvuLrYzphtQcludzMDShDXE6KnhQ2OsXtv5O8/btoC+EdNjZ4X3KioU8pdoDEB5tdQsjV9dI3kpaHCDArYEIdhFBDo56iyLS/jyllXYwseEgCJtuqdCIRm3T/yvDVYJ2RgsAqYFCZnhqXU5dsY/yveHhZbbhkKgbMcm4ambDhi+0NNbpfH8YOZWTArWHwjfMAC/PMmok8fdjEZ/F6zx6v9YpbjjJtDZ4dXqLS4DcDROCtt0ZJl4Fal3sA0Zzz+asDMlsUCqtyJbSZS/h5rp95JPdPEiu1whjb99ym2uftEIDOQRjB34bsWBReiqAho5tiCpMQUmDOY977Drt7CdjXVvz5vw9YoKjAU4f9Jk/J5JwohXqEcrhQ3d27PXRWg8XJTcLra7d9VQZ9tuo++SRpLC1DM8mnUOlCmlMLl/93BpqMp0rjCwNwkxddm+ZYFqb+kznqoQIdQ/FXVd9iaBnI8SP+UKclJWlnvA2JZHWUX0+46KM25fFbXvx0LAZDE9Q/VveAyKgdt3QdZfFv5SNhcAk7r5/ovkT9MasGQppuOxmfXh5QpyLx6NcF/NHubaUD3Sd+gpyhAgVT/XjpMh/EkxMbBqcruqzR6zIN+8HaIdSIMc0FGA7Ezmrax2Im2wQuOjJv0pB7KoY mkvGWAGz AW31WR8ge2xbHiqCia+zGJNgM2/cFKhKDLoflU6p7yxwuXIMMcHCdmlv/MgEeIr5bRNCy0g0Kw1M6taPH5WjRA2Q1k49XMr870syWlwgln5wJYhf3jzBUIYNIZQdnOQuhWQ04L9OJIcuorNfev+kpQQFQJja4BMgrp17FnyoKAV0O99vk1BSmogqcsGIlPbwIWaWHLJ8vUn6KCUcbpT8HSw6odexHL4tZOxU8PR5t2fHQrnS4p5iwStPcWLO5kw/zAH0gzfS8qEMlAFFXFhOa1L1AHUU9ZPPKz3J1I3pqtMVzd1f0tPwGyd5ybn0BEidw/j3ELjhq+hTQ2X3zSjsbZvJAAcGYK7SrNWEvJwRImjYC40fCAv5I5jMoPx/vvRNDrQKYKI/6C6rX0v9yO7FLjGDBLIhXlTkpJgCbMy9zCGXXvR0pDGSbKpfGanrgAYz3msLxftoVLKB6thZuMVRdJsDwGb2NHIesL1A40+8jd6t9l1gYl2Yq2Up4yRdP05N6sC4vkDNR7ZHSyX3vHK1hIRd2dg3HcnGVW7TiD5G4QqVsz9k3VCXIdHI6nm7xoqlywnCsZMIdawetOOnFjv4e15z2OrgjQAN9jSHmz8ODz/PTBqhsEVFPzbYGIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Thu Apr 6 09:42:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A905BC77B6C for ; Thu, 6 Apr 2023 09:43:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40A586B0088; Thu, 6 Apr 2023 05:43:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BB3C6B0089; Thu, 6 Apr 2023 05:43:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 236436B008A; Thu, 6 Apr 2023 05:43:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0A89F6B0088 for ; Thu, 6 Apr 2023 05:43:29 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D67D31C698E for ; Thu, 6 Apr 2023 09:43:28 +0000 (UTC) X-FDA: 80650478496.04.712AE7B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 31B78C000A for ; Thu, 6 Apr 2023 09:43:27 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cQgpHBma; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774207; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3FA5Yjp/rAglxaSde3E7QeP89aqCLnwKrUfyvWNxLew=; b=5k2G1UjfrKpXFoo/OP4ky1Hc4TzxQWpUvhnRYYBINH0kbnjObYk4+vmu/sNS+UFvycfExu rF+G8rJMO/OUVMLtsl4fIkt5hODRJQ1WION7krmKZ+Eva7oo1ybNi0JTAJ9i9Usi2FEMJ2 R1XIz/Ge6BRNFfIj+HfINCcFxuzpl6k= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cQgpHBma; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774207; a=rsa-sha256; cv=none; b=KLY6i3vU++l17cSrpmXAQZQbKhs+wqdwMJCZrCQXtNNTvw2ywHcMwaBM75+EJfykFmiqSg c5YCxi5IWG1DES5j7R/12dOSqdCBuz3GwAl7AzOVsFhqChND7ZLHJokzxeXoD18JbIwSVV Otd6JOv2sEaONFBO/QfBeg0aPIiShnE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3FA5Yjp/rAglxaSde3E7QeP89aqCLnwKrUfyvWNxLew=; b=cQgpHBma29RwI8QeQq5o/TmfCpwGcEqz2z/GzlYEFkLHeARI05F+pfIsdj0BhKvmf00Kg/ CpmlrIaFT4Qi50IMR65SFDGybKz+AVWZZnI1ZAi0ym4+XcSFKs0pbi1VJk8X6WyCSn+7nc 0taDkLyJamUrp3sqjGtP79lRmOUJ3wI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-635-Db4pioF3PMmf8aN9sn6_TA-1; Thu, 06 Apr 2023 05:43:23 -0400 X-MC-Unique: Db4pioF3PMmf8aN9sn6_TA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 633491C041BB; Thu, 6 Apr 2023 09:43:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BBE92166B29; Thu, 6 Apr 2023 09:43:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v5 10/19] tls: Inline do_tcp_sendpages() Date: Thu, 6 Apr 2023 10:42:36 +0100 Message-Id: <20230406094245.3633290-11-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 31B78C000A X-Stat-Signature: kkdibqdem3qqxsd6giyap73x43q5upnt X-Rspam-User: X-HE-Tag: 1680774207-957315 X-HE-Meta: U2FsdGVkX19uvKJ+p9Nhs9Uf3HHDWpBFdX/OyqjX+uztRbk7MZ3M4jBlKYAWXzFnp9HyK0jTT8WjnkmMRpMiA9qD+pk4n+76fYDQeDZknSnA/wAyIDY5TYm4ZlolWjSCVHHXSe1sgdXUe9u579jqt3NpfWHlx6kZhLQHjp2FjDe2WbR054PEd/gAoB765CZFwb6grmCZlEatHof1hV+LB3YS6SpegA0C4RpWbVIhvMt0/XnpEwiXiYvV7dGG00STStoGThmt/ymZN15tp+FV6fMtflkw1+jo74ytJqCkPWYhKPIJP83C7L7kXZKdFQM2I6PpLWAylFkvhB648RF++UmjNd61lFTho+avPz26/bzDxYNHV9F5nS74/U5rZwCYpMSicGkGjtS/zXjXnkzskZR7q9MyqzDLNnEDwHRIJRdXK6R48UReOl65EDuaFIpUF4w21b4DOpnzo/6APgXNIBtd6uxPdoD+/wrQ1EKt/ongNrYoiGBdosf5xp0lX5zE6BL+n6+Dk6Xsul0F4lGdedusNseKrI/HPJHMGOhEhHx3ZLHSnPCGdzUuKppBOg89WA/EE2/T5jtQGnkrO1+rHezJRzEqQY9hfpJ0fLoCpY5vjNKmT1Rubrq7ZIXNB4gNUcSiqqsROElglkWKkP3aNV9IXJ8jnmOstHJ/NPnHSTMGrTSQrDLNiu+zHVnjSbvc8QgxgpL+2FdfER5AmDcnWHG4xWk/rhaYsw6TzPrxr1P9enGUQ5nSQNKftN+5kfMo2fUZcGsbIqi+Ukg4RGuMqOG7TemtXbLOzuf9cd1+DWIBBDXQLDfnPlbvFG+J5RLoB1TUX6rClauGRpKA93kqP1IfVQD7Frz/nFjyk1BXeVXy6PDBU4TvvCOwSQNUUuynWRoYYEfRr09DAbdk6998qqzpmLkSeYmwd7B6rfrm75CkKL0Ew3KyWe9QNScTtWA3+d2zTp7Yx1GpbtNOY2S 6ZqvHKFt v4KqYBrN9BJkcPOvq7R2c+jFpYrOIfqON/pYmf0QuGBNfOXSG8JLG2Qgz5JY6AzLO+dHG/ul+6DUoqsu/nmwlOqQ8UrPjDawntL5dusj5IHiAkbn5RkhrgH8tyntWGPIOXJciXbaVC97wAFWaFS14rKroZAOQ2/ovuRzXN6iWryYiRpNM89SAyg7vHtWGdzW7bHzcXBat/aa9zm9nGW/e5jYolq9OAsSEo9GMK+st0AsxeIbaTtdMOYERjWkbK41WlvY1lWXgTn0m0MVIUaJ/7PzAvYv/EXTPe5d868UIgr8usf0vbOuO3JAyAh4xe/IaxYWhjV1c/l8PrG6/vOOXPgYlYXEixrC+Cf4zoRu1F/dmrwmd5m+E5f9KZ2o29petbvFTAJhKri8RX+jjpl5Wd8jIuH15iJU4I6Njb3+yMvTD776zO3b4GQXkmweOzG/auq5OK/n+QzD+a7uiRPy7kUa5jtmE0cyK/WWvOY/AVfLFSMJkEaTwYzbvrotVdSvuURGR+Vo7RbCoB8l/nRxBufz0GXXOnnW8k4ViRuYi8KXJwmSLUPQJH4lfwA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 154949c7b0c8..d31521c36a84 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -256,7 +256,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index b32c112984dd..1d0e318d7977 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -124,7 +124,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -133,16 +136,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -154,7 +160,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -168,7 +174,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -246,11 +252,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Thu Apr 6 09:42:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8EE2C76196 for ; Thu, 6 Apr 2023 09:43:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C5D06B0089; Thu, 6 Apr 2023 05:43:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 875D46B008A; Thu, 6 Apr 2023 05:43:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 717556B008C; Thu, 6 Apr 2023 05:43:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 635D26B0089 for ; Thu, 6 Apr 2023 05:43:33 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 44C64161254 for ; Thu, 6 Apr 2023 09:43:33 +0000 (UTC) X-FDA: 80650478706.24.910D629 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 91790160006 for ; Thu, 6 Apr 2023 09:43:30 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HocLaENs; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774210; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c83k4BnXfg8o4jtTDXWGTM6Kn2fXWMyaDkEU8sjy210=; b=o2qnaiZxoZVJ+On4N/Sm8p0DPE3N0vUsYSVvbxEHfgjHdYds8HT7pWFbSgulK4BZE4y6FW 3twjq3AgT3JpOTNr9xSJDBugIKWiQhPiXMKBm3uXoR8BLcoyTpmvf6zFJeYiiP8W5Q/6VU CNwnB2MawhzViyzqDZJAtns9AZB80P8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HocLaENs; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774210; a=rsa-sha256; cv=none; b=L4WDIiPXnRGkuJzjSWzuNWPtzwZWBqVHqDKTNvVsm1wACKNcBAZ8mBRT6czB6Md09ySXY6 22qJr3eTLf1UjSD+tS/98IY2hCK3x8ej0pXTde8ioJXeQOCjtNUR+mlDIcP420+NrYF6rQ eVCcY0BVDpW2Tf1Iz2R+IdNnmVqemXY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c83k4BnXfg8o4jtTDXWGTM6Kn2fXWMyaDkEU8sjy210=; b=HocLaENsXNx89RPtVJR7SYa5qFLvNq3/klnY1PTLmpFjjVHu0MS4W13v+JD/aFY8/0DMkQ Xf05egrb1fvSxQ2u08VLTGFnp5x5cCsMZPKNMkkxbL+kxAUNrDxCZvbOoLJSjXywv5WcRQ uQ6NpATSZUSjMF784zS2pZILHK+pgNE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-576-TYObJh0zMq67k0lTYOLEeg-1; Thu, 06 Apr 2023 05:43:26 -0400 X-MC-Unique: TYObJh0zMq67k0lTYOLEeg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 95B0E800B23; Thu, 6 Apr 2023 09:43:25 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id F305A440D8; Thu, 6 Apr 2023 09:43:22 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Jason Gunthorpe , Leon Romanovsky , Tom Talpey , linux-rdma@vger.kernel.org Subject: [PATCH net-next v5 11/19] siw: Inline do_tcp_sendpages() Date: Thu, 6 Apr 2023 10:42:37 +0100 Message-Id: <20230406094245.3633290-12-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 91790160006 X-Stat-Signature: jstukigcdr1pto7f9xj57he14bomyybf X-HE-Tag: 1680774210-384117 X-HE-Meta: U2FsdGVkX1/WZxhTHSHWwMd4iUhjS1zTiwex/YZD++1o5uTctxYA70mRKVXa9723Sw8YANiv6EkQTLEEfiXckmiqQ+ELRtgjI9e2pfckRoW1r+dSgoVKrWEuZOHAP9QUObQl1/sdgOOp66ayXvfVeHw7ERWKwT88mRw3KCJSgooj3gvsLBUufRSA3C45t04FRxgKI8Sb0H/7fHfcxrIT1xBsVvUEl9wvWADy9renm6TD/+1G2Fp/vNcoWwkfi4BH6As9H/o4Vhcw+p8kYPyOAaMgfCLCfXcGzY74fkrKc3MwI5PyDUKX6rGrEWG/8YTWIYM14XirnBVlK2sZnHpa4pjY9VNnOKy4vIgGuWt94J4kl51Duv49Za8shkUVU8u5zaTbOZxS+RwWCMxqAjpchqZ+Vg/0aLyuc2sOfDTWqHGySZFdUToM5OWtqbAr8niXTnsrdFaIJ7zh7EnkYAJQ8hDz05Fm6RmDW7iAeztBEchomUad9zDifhC8UN2feJewzwOMkPuCGDaF4fRpHk1x2nJTbQHaQ3kkUq3ciySrz+qKxtccPitCZFxxWviTN2JER1prt2NpyIBrzLCFiHJ4VqQqHACXxXkeGxK9YWdAfQT1nSVssh/O5XpuI46xspSk8YBoqO+f2xgho2ECcm+UyqoxtPFrfWYnDtqnsTBYfbsj+AhlQ31RbWAgqwKTufSiCiG9xqsBDDWI5i3BpCQDX/MdOi4UbT2LyhdLxgYgr99scZoumwWK0rfTXZZchyZHrbzJ/fhGBhEhwY1FZ2FxgHeJk8ccfQIfmHM5KNyQEbfP8oeeI8wh8/hQHTta4Kj9dNcmQEMkuJsMB7JU6nHPx/E18tC05R2Mk1tRHLZfo9Kmy39Sv5LUU8XDOSLBMPKoEIhpSGJK/wK+BgM7C64EACN6+TA1/EygA42Atq6tS6wgZK5pM0aPIh3Y3kRcm3997RaP3Dwu0+B2LU8qhej 5kb+lPf7 37LMcdReSMCAbTxXmqO7Sc27YpccBotbKgS5x8lD9qBVqKBzZr6hMIVLpm7kVr+DqWT4Bgxkrf/1JsWJ5A6dc2eis47nvFtrtT+hCVyo/Q1zfhdyOwxMO/FM88sIJ4bPkgs6CVX/D+IcAOSw8UvToPflGYz8VnLB8cEqohI8lbF0nK8lRmcBhEDrx9E4oQGZRI+6FyKpX7jGjSdP0WhVUS03lxaCXN1bwy4cFVKjXT3Eregw3mSathvAxCi9DeoEcz99+Bw1tQrykaolKg/tyrD4oAl1O730uZsnrN+lsbxuPmoqCQjwJMMTibHTlLyMv6bsuU0WbiOn6XygujbbFdKvfpD8MaaFSTpVGHemZyWLRAezmqs6e973pPUa+sYURzGDihUhivFuFkh+Q6hqYHOHXU/XjNnAR16bhWa1NkrQx5134tjGxO1dEdAu7kqayhtWRtGaGXqJuIqPGjXQNXFM899GCIdeOADdqXpev/HLkVAaM7qzr98GAstSPCKxLy9zpsERnncWDj7WlWT9XYFd+HeCVrOUz+tAfzu1KzAR8MIQMIHCawjxrSS8KNGZ4D9rVaf9aonQ2OEulY/4VB9V4weubKClk9/P0uY5Vv04xER9xswZ+pTCdq3Ayoerofx3chug75jMuOctCACvFqz0BAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Bernard Metzler cc: Jason Gunthorpe cc: Leon Romanovsky cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 05052b49107f..fa5de40d85d5 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -313,7 +313,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -324,20 +324,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags = MSG_MORE | MSG_DONTWAIT; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Thu Apr 6 09:42:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0352C7618D for ; Thu, 6 Apr 2023 09:43:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51C2F6B008C; Thu, 6 Apr 2023 05:43:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47EC26B0092; Thu, 6 Apr 2023 05:43:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36CC96B0093; Thu, 6 Apr 2023 05:43:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1A6516B008C for ; Thu, 6 Apr 2023 05:43:35 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DFFB31C5DFB for ; Thu, 6 Apr 2023 09:43:34 +0000 (UTC) X-FDA: 80650478748.10.C2F8F2B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 4D2331C0009 for ; Thu, 6 Apr 2023 09:43:33 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=anuwtPGl; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RHjrbnbFl1QPUSYdPKMdHAGAsNDTVNQC0aaPP3HcXyE=; b=Oa0aGbZCGrivyjNch+UPWAgyBhZpp+Kw0xwYkAvpqta1BrSTCz05ZF3MTjSzlq+7qZ96v4 R7dONDa2Vk6zHqFTgpB4oLpuX/E5szfGFVquCoxHQQ3e4xluZnhQm95+yRTpRBgFO+mUbG 6Pe5L3gOX7H7SZ2JqS0oD+OKQsFrA0w= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=anuwtPGl; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774213; a=rsa-sha256; cv=none; b=zQ9jQRfIMYFcaa+xAXZYlrmmGxOX2gH1xqBsEf/oHlObuGisH+t75EBlkTnHA2Ichc27qz ODw+PKLfXFxMx/LWJWlbGRNqPIfntLmgEwz3OFHFWoj47dM8MinUYSnTgZO+uaYXCXJEmf lMGWfO460/roCI9uL3B/ArU+s5N+Uz8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774212; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RHjrbnbFl1QPUSYdPKMdHAGAsNDTVNQC0aaPP3HcXyE=; b=anuwtPGl+SmWOPDRopO81v8ue9p9nJuvNhVXrXaqVKsGfyJNiUUZ4kTgVn54rIqwNJeHU2 zGV+xu/2vPMcergvBjdBnjUWQeymTX/vfBQSJZf5/S64ZKhpi7Wy+uy0HAEc+wOLjV+gV0 7nVQ1BR8t3OaUgMzMv29Cj/58IOd8FE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-262-3gYCWEu5OKKN3foGZGQGOA-1; Thu, 06 Apr 2023 05:43:29 -0400 X-MC-Unique: 3gYCWEu5OKKN3foGZGQGOA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5B8D8101A550; Thu, 6 Apr 2023 09:43:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4AEF240C20FA; Thu, 6 Apr 2023 09:43:26 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 12/19] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Thu, 6 Apr 2023 10:42:38 +0100 Message-Id: <20230406094245.3633290-13-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 4D2331C0009 X-Stat-Signature: 7xa3qf9hdxg34pxr1x7n8xdx158a8ygq X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680774213-163594 X-HE-Meta: U2FsdGVkX1//mzapmAg1ohppB0YYmaXb7Qw/pAklxWeWOioYy1qqcDa4/M68IPEY8ZhR4nE+h5P3dfnPXTK1lfXqiobsRjm5xHOI+t08REyTw9F6r0fajvF4rS8aHd+yWxD2dZOOZjWZ/WH9WPFpLiW7w4otXAYv895quKpXTvblh+mvgNgNReKUeVZYC6bC0IkraKdbj8Dx6lhZNrax/WIFSbH2h1oyo4W5TSXmyNcIx889TWfCjPRFu0eQ+OvepWoltpK2o6PT1er3zWm/OGKZi3UWIqp2JiL/xMO8yfUt0sJSkZ2ppAfvbNdREgF0ZD1HQHfEiwprBzhAvAzF7nO0nOZuxDfQxgTvk/YTE8kiSk2P0tED9wEv2qmVLStf2ooUp5tWpVz/KHz/stWENAhBMSGERXaSichvZUG+kSZe5RF71itL6f5sZGV4VzDsfMLGVYZ4zzaUCXfdKUQIizNpqmtlbFc5uWKBQfWY2CT2ejKdoDo01Ep5O9BMssSkTChBTWzLGkyjt5ILgnOVladwJgfLNnwl++6sCLD4t9+w2emiXQBCioEaD8oIBiZJ1KA6yuOpJRDbJe2fZlRidOvpU54p7bMKZNqDzs8/G42wsETHwZT6X50eI3uIs6LqkHIeEnL5btQrD/HuTMrVI3vtOxhpY/tHH8bfJuA3OhxopIm+eFOyqH9yHpZj9oj88i2o57rqYIwrIw2+cjijH1jQaZKMSK/YYrUKGKYomzwyTnP57bdRUAnST/JXgupgBAOa9gKzZp01gBfJoG+yuho8Ke568DaXA4V4vqn5uSM6Y5F+7zqZXRjZuMT0DOmgTkEMCQ+ew/LTowLaD6CmE8FtmG3bCfhILhv+eunJ4yHufS5YjfFTOjNb66OmRZJSuRfj6aAlZ+ptCsbHjukFlsXp+IA+GyIYzHsV7JEnWzcJunyqLFAGnOM+07yD+KjL98nvSWEDJlARyqetj1i IkWSRpuX LJ1TRV81M526yGRKYV//LFQMAPvqk+zWiTU0sLe+QR3HBTnKwYpKw2XyRGAtuSlD5j58fPFZNlI8GIb2cYtNOyRkdkPQnb5LpImgR4/zxVlCCFE8CW66iVE5jdLIP6f4IBs6Uk7Fz872uRifPaUpfT88mvjS9R5v9N5Jngz2Mp3atPF7Z7+oH6H17mkxVjHE06M5Y6iluO0LuvZQDVjR9apFEGcG4FWNc1gcYYzMdlPeSLouiqQ/KaNP/H/NDJmGUigFftxlkr1Od+hCDieehwjNTEZpPqIcEd3FZI8IPhtl7wZUpG81WqAG6OHVgokDv7W/a52cGU77wyq94iMHifpgG9y4VP8VWzL+3mrWhAoPE8LM50xSg2+BROFzrjxa6u9dEyEHlin1WjzTWvLiay9ejIrD8hYX7nxlD9dBLlUApS5V3dqHzUmARmf+kH9XhPbNECEeoiYORkNkdZPpNinSJsb2FwggQox9x8StU2TVuPl/Ies5z4TFPoG1/nMGowID6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index a0a91a988272..11c62d37f3d5 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index a8a4ace8b3da..c7240beedfed 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -972,12 +972,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -986,18 +991,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Thu Apr 6 09:42:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89A43C7618D for ; Thu, 6 Apr 2023 09:43:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AB4A6B0093; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25B5D6B0095; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FBE46B0096; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F07276B0093 for ; Thu, 6 Apr 2023 05:43:40 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C717940EED for ; Thu, 6 Apr 2023 09:43:40 +0000 (UTC) X-FDA: 80650479000.23.B853223 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 09A1310001C for ; Thu, 6 Apr 2023 09:43:38 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CG+o9E2w; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DWLW7zv5E95PcNMvzADLtZC4cFCYwscW+4auTp2qbuU=; b=vhJI4/9GBmyq5d9ig/k1LfWli3HeR3bVcX4gRIKrKR6sALRz2pZuXpFh55ErElW+9nLBqi qGeuzvDC4rqMA76j27xNEmwoxLWkLFd1JweAivpbcigs+2N2Pb1DIOitdNwzDLwnNn8iFl Fb+84zaEVKvlFulZyVMLsxIelJgnnzs= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CG+o9E2w; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774219; a=rsa-sha256; cv=none; b=3ssn1zOdChqUP7uMPbGR7sSiMW7OmC0lXNHDieoUzRMF+f6JcaIP62swxeqGYAV5jkE0vQ MC3vQjSl42JWONKkUlf7FcL2wZ06CtMkfeycZfgB035oReuDUl2S1LS3p4tQYSxxkqzD7z Hw3Sn8Y0CkUt9d1nm8BaDlkK86tLjro= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DWLW7zv5E95PcNMvzADLtZC4cFCYwscW+4auTp2qbuU=; b=CG+o9E2w7pGI6SvEHXiiNl7yNDxDm+mQhNEu9/RWPrzsmENiRoJrrMta+ZVrph6N0yWhtq 5BGdfdUpJogwThO0PPaB+GtC5WGGw/d1JHFTQ3VWC17SDE0jK3whv9zp9s3IYrlODSkozP RIndp7lbNnXl8XMOQ9aXkin27gdGWAY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-44-vEeTLutcMDW3WDa_hFgwtA-1; Thu, 06 Apr 2023 05:43:31 -0400 X-MC-Unique: vEeTLutcMDW3WDa_hFgwtA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0F9C2800B23; Thu, 6 Apr 2023 09:43:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id F03781121314; Thu, 6 Apr 2023 09:43:28 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 13/19] ip, udp: Support MSG_SPLICE_PAGES Date: Thu, 6 Apr 2023 10:42:39 +0100 Message-Id: <20230406094245.3633290-14-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 09A1310001C X-Stat-Signature: fhegwosqarnzwkonmz9o385z3h6dos9g X-Rspam-User: X-HE-Tag: 1680774218-465566 X-HE-Meta: U2FsdGVkX1+Uj8hQLi2BDaHv0/PuSaEVttTRkLYfO80B12sVIg90jEOiXsOjl2LT3Hn72JFi3QHsAbDMSnFqhtVvsXj8Mnbuo3VnogEKlf16FD+w1Exe3ZMdqNA67oHVQp85LmEbPDOvCcUYOf1HDz8njSdwVK7pNzjd0oU5W7lq77GUDxy8p1rnofepo4hhcrJq8eoCRWZIIyGOt8taW16XXYy+avwjNYR8kFLLb121JkZ7UBHkyYq5ekIqPlFfc+xGAg2h7oHhJAkZAdGz0cdeHOKMnVXk42GUiQJNH6ZJsm1nbdp30imZlRiQvCBwVt0Ck99IXNJFhxPwSuLqouIcT/b82gI9YTqsKXkEu+IHqSSiwPVY3bOCj2B52i6dCvHWIQqPUq8Zu7PXTSmccIqfT1L3k32T84z2uu0bwPlgroXIOe0sBdf4Q6v6A1qyKxPJyumSGCK9F7caKKTENLwulN46DHG0eCJMn2CN/zmIz3RVG0orfh3/D9jNy7OE8D33kWRUQZ5RLJ65VaNI6Wa9DUYiBQw5Ofed74XzmWLmQxUObEKHOprnCJ9ngA833JB8EY0emwJV9sIDNLDMQUu2R+51iM3VP2kTp2aSsWugE+0f/rokPp6yl2IMzd7HcKkBn8sWa8KrnFJiY6WZSK/Nh20VxumgK0qleHROWU70TmYvYPk4nxVVdUHldqesK3DcgCu4e0qgLaS6vGFS2aW4CDRHh+Ouu0Jzzt9S+MVZKhJoC7du5iFWP6XCZzwjHXv4pQ1AapOeB3qTCf9y1QrjI1jJk042+G5WO432JJmvYLmavX5bU+mrXkhiJmQ7PWc4l7J3f/9J6gSQUSMtNDLEGYABtazAUrAejrSC+FNtXkxKA62Fxsl4MD/PYR49+59y6Cr/aOJd46dskPCjCBTW/b4JAI162QwYiMBJMAv3bBkSdP9l2W6cXAe2AXuzkEntCg1DzVhxz2j5/HP 56yFKSd5 RLmlYQ8co1gwgqfw2EOIhEClP2DQkUeb2WFlk97cLRl/rPf+Zm+VcSBACbj6GZxTfv12HyaLqP+97bFjmlKBBF7iFlTKcGs591+Rilx59IjoDIc3xvg61G+j6uVfw7piieKEIaw2aUPJ5wZPazzVSFDcEqPDnch99IuASYiS5VVnLdfczmORo8sLpcrd/2z9sg788hNE8O++Smh3f8AkS0yq/SeOCvkUrAGUMh5z3J6q6eFJ1zkamE/MbSI6fDoPGcVSkgKsY0ajHL3BsTsBeHUo7eRrqxIT7LPuQp04Laanx1eLOge2q9znT8rObtG+Iu2JouhrPFH3sWd5nfLgIowmDZzHnCVRaGICcK6ESfq/apzIWMhy1pImcgF+mPFd5MpPL16IWgMdw13Bm++ayoiOk4ATWGjwhlPQBZrsb2/WsP0ivf/pL8K99jmzboLefR4he7l1/gXNnpwux/OtTPxO08tEVTZBq2vSAkm7+zJCsSXI+Kd+S86jJ3TEc6SYWp1v2gllwCgdo0krcjA5JaYshHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 22a90a9392eb..c6c318eb3d37 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -957,6 +957,41 @@ csum_page(struct page *page, int offset, int copy) return csum; } +/* + * Add (or copy) data pages for MSG_SPLICE_PAGES. + */ +static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, + void *from, int *pcopy) +{ + struct msghdr *msg = from; + struct page *page = NULL, **pages = &page; + ssize_t copy = *pcopy; + size_t off; + int err; + + copy = iov_iter_extract_pages(&msg->msg_iter, &pages, copy, 1, 0, &off); + if (copy <= 0) + return copy ?: -EIO; + + err = skb_append_pagefrags(skb, page, off, copy); + if (err < 0) { + iov_iter_revert(&msg->msg_iter, copy); + return err; + } + + if (skb->ip_summed == CHECKSUM_NONE) { + __wsum csum; + + csum = csum_page(page, off, copy); + skb->csum = csum_block_add(skb->csum, csum, skb->len); + } + + skb_len_add(skb, copy); + refcount_add(copy, &sk->sk_wmem_alloc); + *pcopy = copy; + return 0; +} + static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -1048,6 +1083,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1250,10 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + err = __ip_splice_pages(sk, skb, from, ©); + if (err < 0) + goto error; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu Apr 6 09:42:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AD0BC76196 for ; Thu, 6 Apr 2023 09:43:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82AFC6B0095; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 800BD6B0096; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6525D6B0098; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 580316B0095 for ; Thu, 6 Apr 2023 05:43:41 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2EF6F1411B8 for ; Thu, 6 Apr 2023 09:43:41 +0000 (UTC) X-FDA: 80650479042.21.CA1C111 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 750D740015 for ; Thu, 6 Apr 2023 09:43:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="APTj/8YK"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Lhpi7afnUSYBVmFcNV33/AfpfwJkESufDM5kupMXc5M=; b=l18bEJNK2QQNBwykmk/0ZwXSV9yAcVs6WPtZyPfTl9dUQVL9+UJbSt09uCUTPLUQIjOL8o V1QhuBUFR0R0XWPPzQ5X1RB6NPsa8w7rebdAYJbO5Dcm7JRSCFRKi3fe1SE1hVfXf6rWOp fm7lvZr0X+Iw3trQNnItxPmZtuSybLs= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="APTj/8YK"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774219; a=rsa-sha256; cv=none; b=wZfLQTkeZEWQCm6vOVck8fMHlGMBirPLnopp9TOojXG24zH7fsi8+5YV88lMffNMRlUEoJ eYj2Z+9ce3ogIvvCdsHcCSSXcRczK0x+du767LPkhYAuqKmAsHZ23XQIb3zhY+u25NVrZV Xyt1KPaRirlew/KvtaHkgBzfp7+fWls= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lhpi7afnUSYBVmFcNV33/AfpfwJkESufDM5kupMXc5M=; b=APTj/8YKiarlb4Q8hIrsVFbE+YiCU22qbJQASZqWpFRqCsnwavRncVQcO3MVF05fB8Tg/y rkhxT919FItO9I6JE07Q7u21ioishcdwZ/WDWqoTuLQoT5svzverlX0djz54MXDnloezSN oksxP4aJrZJp5sfl/3zF8As6xa7MD9k= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-387-ZCLBzN6RN_yR-HE_tGR4oA-1; Thu, 06 Apr 2023 05:43:34 -0400 X-MC-Unique: ZCLBzN6RN_yR-HE_tGR4oA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D9DE93C0ED40; Thu, 6 Apr 2023 09:43:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id C6DC0202701F; Thu, 6 Apr 2023 09:43:31 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 14/19] ip, udp: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Thu, 6 Apr 2023 10:42:40 +0100 Message-Id: <20230406094245.3633290-15-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 750D740015 X-Stat-Signature: 5ys3sfiew7xyx957x4mju8e3acc96bxd X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680774219-521304 X-HE-Meta: U2FsdGVkX18pEXp5+qKKFylhyM9gsv7AH0d2EmToKXzHHtJkclFEND008CbhEfOVKbO2BuLAsCUMa+lSgtVkwPVet4Eh6IoiMMzzX0PSO9gY0itgo2U9QcXtXXe8TvF1kbE84fdz5xlt5OPVOcMJUcKpjpojs27//42lR7mZGMa7Si9uUQMDXYdQcy41rVvQSIjMmodFGIr2QIn48pUsdV5JA0h4yyjlaOK4wx4sSkXgWkaBj6jQuCBIAXcmexnDsviy+3HPRYwCCGfnNyDIUkMPWniBvvvyGD8ZQJlMH5j/7uJU50LD6Pvsguaanw+AF8JY4aJyAyhrzgfWJx/wY3T0eQkg0cekl8N2UwI5TQyNNQRVb2b+8SewYk8u7tTDAycU5JFludHF37EyexpupQuR2t8m2dNG9cdXpM+RHbCaFPtkkFyn14AAOOKdiafeDRqUTnbJEFN3R2s+KJmACOVmjdXgne/MkpAXxf4tlXTVA4Xf+yRUCWRuk2zwAVVbh+pOpeL458dEPuSam0MA26aAKe+jsa7DDWkWVbzz5ZAHn6cFDLWfsnDXts9xqF0rVgzBzd8+skg0ZlLvmO0du/Fwupn//BLAXQ6m7j3m/WukERB13Lsv/AOufiA34CbZL1f8v9x6pOnEXfkgDMfSbLyTYyhyqpTuOrS1+XcmwtfpGBaGBxbZu7ht+uXLARrj9idS26aia3SI/dKTktrYM2NmPJwm9rHyZkvGQGJITUkbJva2aS4bKNK/7P8WyrhDOVDTGP1a1vdTMdtdkXOqtxmseEYswCY4vidT5KgcLtC5OQ43c1bfERVXR4vSVowLYGRQakiM5rykpfzWHGhV9SQE6RL5Dm8hz/fw3h2GV4n99gnIO2WZAj2luRXEmNqk8IG6HDj7fcZbS/SoHD/vs2IqYi+cQh5XYqGOVGRyw7A20sVa0MoeS8eLklbD+Re6NQTttHE6uFnPZ2dfTt5 3nOm0k5S twsZHnMpPAheGJ8PAF2g+qjYa7CyXFWt1On6GgHCKRqrXTXj54xqtIL6rSoYVejf+KPys6tTQBFf1bd+pZlK055P8+aEFhF0yLKWX85NAq8/hPkRR9yFAAVkm6zc04pY9m/FtTcb8opcLBASfnikj1Zm992T7609wDZQFrNR7mLUNWCVNHOQZXeQSoDBebJPsqMWohU/AQ57Z8YxW/GdTCVG9YP+fKkoszXUz2F2yAhldjD4xF23ito5TsN0FXDErh8eloXym0oajonTYHpSHLJyq4xZdhc/i+/yLr3xW8LeTvb9+fn/R0SPTGk0TcEHnINWx6VeC3OL+tF5ZCJRT4sBk7rLFJF8jYP5cIi7k1ZuriTn9EpyoSXqt5u+OigZX5Njoa5nrLI83mJvB/milrSVTH3y1H65fx7Rkh/TYOoZt6pqc3S7qr/0YJDAYz1VpS7znX5jETLUTzLKlpDFjD+i6G8HgFd8wF5VSbtPG3YG37MstWQ9inJQWoOfKJvFHrKaIyVhzw84hzAml4gRXkDp2WQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make __ip_append_data() copy it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index c6c318eb3d37..48db7bf475df 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -967,13 +967,32 @@ static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, struct page *page = NULL, **pages = &page; ssize_t copy = *pcopy; size_t off; + bool put = false; int err; copy = iov_iter_extract_pages(&msg->msg_iter, &pages, copy, 1, 0, &off); if (copy <= 0) return copy ?: -EIO; + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, copy, + sk->sk_allocation, ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(&msg->msg_iter, copy); + return -ENOMEM; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + err = skb_append_pagefrags(skb, page, off, copy); + if (put) + put_page(page); if (err < 0) { iov_iter_revert(&msg->msg_iter, copy); return err; From patchwork Thu Apr 6 09:42:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9335DC77B6C for ; Thu, 6 Apr 2023 09:43:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 348EE6B0099; Thu, 6 Apr 2023 05:43:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D1E56B009A; Thu, 6 Apr 2023 05:43:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14A276B009B; Thu, 6 Apr 2023 05:43:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 036CD6B0099 for ; Thu, 6 Apr 2023 05:43:44 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C93941A0F0D for ; Thu, 6 Apr 2023 09:43:43 +0000 (UTC) X-FDA: 80650479126.27.34BEF3D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 2F9B514000F for ; Thu, 6 Apr 2023 09:43:41 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dv5iGZgr; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DdOAoT9+lQC5BMkbFi+53knWSKBAKQi3HhFM4xBQ4Wc=; b=Ng4JAqYXDr66SHorL3TY48WUSNGkfdT7u2sF0QtRS4SvDWbHlD/hDskadwxeIaUG6uC+kp m/Uu4+ZrC+afpyWwfE/symS6IrwymzFSC7iOrsyaygsIDkvu7zbf72h6XOePR9cGiiNAfj 6/FPaAKeGYWeHA2dmFfEr8HASplb9Sk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dv5iGZgr; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774222; a=rsa-sha256; cv=none; b=HRGH4W00uxI5eGwBGHtC4MXfyvaUcv0CuhQkgC9wegEcqk33fhqfm0pBKD1F1O22oKHcv0 vpFiWEv/KYj7LWWN1e5eAv3eK7RbQIj5JWjMLPgpOzRH4ltfeM8oIWjjle3PCYzHc422kG 666iR6AbSvfm/PFS8xydcSwkIZlW4as= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774221; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DdOAoT9+lQC5BMkbFi+53knWSKBAKQi3HhFM4xBQ4Wc=; b=dv5iGZgrw5wCF7hFyQKCQ0Ana1KfaxruRxMp5CvhHybZuppL8Qb2egU0iOGQ2jpoW4CGct ToLeAySH80EyGnlImnJi5E2i/w9otnl8d0AmyFGNiqXvp7N0UJajLiQeEmlE7HLQ94wgr3 mhDWiIzgWZdEeiJ4JF6i21v1IsQPuwA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-245-8xTpgCDVMpmC5orYHYQeQw-1; Thu, 06 Apr 2023 05:43:37 -0400 X-MC-Unique: 8xTpgCDVMpmC5orYHYQeQw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 88C731C068E1; Thu, 6 Apr 2023 09:43:36 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 79E6B2166B26; Thu, 6 Apr 2023 09:43:34 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 15/19] ip6, udp6: Support MSG_SPLICE_PAGES Date: Thu, 6 Apr 2023 10:42:41 +0100 Message-Id: <20230406094245.3633290-16-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2F9B514000F X-Stat-Signature: xzphk9rhzygbk6w1bop4n315u1aj6ymq X-HE-Tag: 1680774221-115923 X-HE-Meta: U2FsdGVkX19aMH5BAkhgnIq7S1rNgLUsvgZomcFiU+HModKfa+0xRSCd1tz45fbdtKGcESMjZmKFuHS0aDwkqxafoOThmNJMnK1EqDoaq6BU6Pn93nTHAdH+G3y1He0XqzuKDiv0pF4r9evpsLPelTwD3+GAEBek8oxZW0j+36SPkD/tHftksfmeLgJoLOgTmjJJKCEw7HaG5eU0BmhvF8JAhA5mEJKdLu8ciZJfQ/EqWc3CZQ2SRlpvZ1+7/6unE782JShyd4K2rABzhArUzdY551V6ymY3i/qT71a0jYmSAEs6NentQBOSUr2GPNG6Y7OcBdYfslFaLbpVbsZ3zlzFIGKcijRTxuJEjrcSESCpKTmCHT4XibNEk5gLGkt+qe2CVwot8bmd8B5TRczb/2Kclrb/oyLuDP8EH142oDC1dQMBupdYlOGE7jr6O9sak8L2/+0lC/e+N0LGW3H5VZTRZPHpfrJjIdVDw15uxHvkD7a3tnAvbtiiOL4sIEPmSyj7GFZWOfyLFaj0kB7McLVGt/Jy05bukggeMCpJVrPvQlSeXrq5V+UO1gnrO4Zhn8dfikRDgMApCcW0QoXC2Eif/74N2xzhYSVHdhkmKZ49YbDRM/YKaKO5W5qRtM21LHwHz4iKUxoPTu3ezIIaWGh1kAN5keA5/RgPXrbM/c+P2DCcZ9juuHkVkZY4EbcAR0eyElO+3GCABjMoOSYR7/qLEZ5pIRAO4EzM2OBHm6Dg0fFmXaAEzIJk48u5NffBRRarZqtSQJtKj9gb/Hhio+cp66luL9KLRpfzV/50IEaS2ihnZRihRx4hK2T2Z8gReVBttp+QdCHUaN+E2bVeduWOYdAB/x/NVc0orFLeQVgPuj8yu6dwDbwxJpVgC8wuSbDlAvPvg7xHza6mK0v+DSepg51ITTxMpxFCda06twngTOvuF1sqCu0X9zQ7AumQhH8qsqeMvJgVIV8WqgX RD+relCG ttK2w4Zq7PZz1g2tFxJTkyDxY9QrsC5F7kuowV2lgujxi9hhkXlvb6VaIqZBXzRBlXoojJ/eccnmnebT9mJvIZ65+RhHjfqMNBihk3SOaJJmIKXCTAMu58hx4biodsAleNajoFKXCejQnM5g6UUA63CcO7TAWf28f6pAvAnoqxYLNBXAV4nNeN8+Lv9JK3Ms2OeFQACh6USxWg16Quq31STmiTUhcoUSu6Zn32jN3vW4sZ4A3EohBWQkzw168Y9FR5UX+4kF3WfbsdQvbgRENoJq0moT9BuOsl+G7AfKGqVHkNbslwIwArXKifFB8h9jQv6HpcWR/uk++UDjMTp6HgqvUttQmZ3IRfaLtTm/1XqEsXmORf7ubVrAtb8gjy6JN2KUOu2JXifAWCO1BAe/148+qqkLuDJn+EECD64KtTgWpY9UwJafdPSseyf0l9S/lz4OP1ce/CdIsly21aeYADjMakRM+ekvBP+s52Dj31uvRkLPHH3Nu2htxuykAXPS9OgZMpKuADTqaFCbnMevscXOeTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/ip.h | 1 + net/ipv4/ip_output.c | 4 ++-- net/ipv6/ip6_output.c | 12 ++++++++++++ 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..dcbedeffab60 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -211,6 +211,7 @@ int ip_local_out(struct net *net, struct sock *sk, struct sk_buff *skb); int __ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl, __u8 tos); void ip_init(void); +int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, void *from, int *pcopy); int ip_append_data(struct sock *sk, struct flowi4 *fl4, int getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb), diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 48db7bf475df..5b66c28c2e41 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -960,8 +960,7 @@ csum_page(struct page *page, int offset, int copy) /* * Add (or copy) data pages for MSG_SPLICE_PAGES. */ -static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, - void *from, int *pcopy) +int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, void *from, int *pcopy) { struct msghdr *msg = from; struct page *page = NULL, **pages = &page; @@ -1010,6 +1009,7 @@ static int __ip_splice_pages(struct sock *sk, struct sk_buff *skb, *pcopy = copy; return 0; } +EXPORT_SYMBOL_GPL(__ip_splice_pages); static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 0b6140f0179d..82846d18cf22 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,10 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + err = __ip_splice_pages(sk, skb, from, ©); + if (err < 0) + goto error; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu Apr 6 09:42:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B164C77B6E for ; Thu, 6 Apr 2023 09:43:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D1BF6B0088; Thu, 6 Apr 2023 05:43:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 281AC6B009C; Thu, 6 Apr 2023 05:43:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 122E36B009D; Thu, 6 Apr 2023 05:43:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 00B0B6B009C for ; Thu, 6 Apr 2023 05:43:48 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CB976811AD for ; Thu, 6 Apr 2023 09:43:48 +0000 (UTC) X-FDA: 80650479336.19.338823B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 1CB87A0006 for ; Thu, 6 Apr 2023 09:43:46 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T+M8ttna; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HO5c6fWLbjU5UW9pwObE4TQgRDIF6vfHUazrp9mJO98=; b=TysE7mlb0ha1wWpIhHkzYhdr78jBRlPFwbcL5JURqpuM0KuZFCraH+ojGpyF43fjbO8Ytj sy+tL79p27d326LZJtOVq1/PPMUgh9fuBuH36Jo2RI7k3cEY/D7VlXiOslUc16CKaYgJ5T 7uLubT8lU4mV/GcKB4e/gfXlry6dcDQ= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T+M8ttna; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774227; a=rsa-sha256; cv=none; b=tB+2HpCWN/EP0Rxk+u7l28J6XrivvV14Th9sjPtgrr42tsUwK+OcX3LlowuFSBNTjE0Yb+ yyKqn1IE5VHkzQx/njiCxsAOvHo8uoKsjhkr5qKyARDtdhEu/g2ruvMaDyQho/lvNt0TQ8 eQaGhrr0vlXqr1ZJ9LJJTMyqYV0wsuk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774226; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HO5c6fWLbjU5UW9pwObE4TQgRDIF6vfHUazrp9mJO98=; b=T+M8ttnay6jMPN8FBKAMSSCHStGoURSAZCVOdba9pl5CCrXFH3Woir5kWSC1Pg2AMXbULw j5trnGmOd8BwiN6VUh+CRVzkJaGk0Ii2AZUOXVezQhpkTFWpuL+kfBJJZQ2YwlS2vSozpy /qDcUEGuzYZiiTLD+XwXjHHZ1JMDJ34= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-122-QJgl81CHMCS3cOzCa4PB8g-1; Thu, 06 Apr 2023 05:43:40 -0400 X-MC-Unique: QJgl81CHMCS3cOzCa4PB8g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4BC24884340; Thu, 6 Apr 2023 09:43:39 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3C6221121314; Thu, 6 Apr 2023 09:43:37 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 16/19] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 6 Apr 2023 10:42:42 +0100 Message-Id: <20230406094245.3633290-17-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1CB87A0006 X-Stat-Signature: ya3854dihg1zy7fn4ziop55wzsmfugqj X-HE-Tag: 1680774226-793133 X-HE-Meta: U2FsdGVkX1+hFsD9TSK2L1aEnNxcqlEKC1YuMaUINfhz1JiRQ2X2sP35QvP8/LfYSkKtvFKaHmGr5XJKza+sti/YV7jmhB/XV2e+2C6O6XrQFgeOCD4TLnSSa50GExKX010vi/JNugW2Ec63CkVM739oe7ioY9oYTnGoOjrE1fXnUWZjTVVmz98tlFpAGNM5hzQOoV4JuMl+kKmuhrt8VjMyVT3kTPhkNtbVDLo5z7/+p0RWvz3ZSbJPSv2pLNs2BY7L9GH6N43SKczY2zcL82Wkn8cCCJS42LHQeb4kdmroXmNpp8hl87h1K6tE3dTmxDoxoOi1r9pOoIQJKDFqNhZkgGPdTE66P1VxLar/1TrYdFlSrK7evzYvIBPTYdacXlMI00Hy/dNbr4CLiJtMTeI2Ja3tPsnxWjCUL+HDKwUJ7ySu5AKabwuq1eKt4LLGL4SEIxPstu+8AlVNL74kdzPiK8zsVVMnnxcvloQHrD5WC/70x6vZ61mnts2C2R7SwCV+GX/+JIradSo7MhD3ebVLGb4Ja4KKamEdYFkqSywqg2eyRT9ef/lp/ihTjdhAvnVFFgDxMFEaDam0w/RpxP9FTIJ8w86ukxm7OYXu1gaoTK7zinHX8VbowbQp2sEIGF6ijL1HzcLiecBvzNzBoiUdAgSeBzoLkoU1Qs4be40SqUKWvvcklUc3UeDgmCT+Q3STWBJSTDP7E/PUPRNJO5JxjPX6+BWKMbxdSxzSfG/xiMRkVCnoT8Y2QaNBX9+KXUSvKrKerdJkkQcugPbNGajRfGSiw+FdhPo+XEwOnflO4lepszNQ9EoHQyVOqVOcksHdvky8qROaCBgUxRuG0fg3DR3tDSEHuWKCGg8RpuPYdhTSa83k+Pw3vkdULb5XMqLa1+1+AwoefNU1gVZks3nNNqkSG1KEfFc4Xe3LX2Jkcaqv3icxPMSOYVbe9T1H6VzELFcyE4JjarwgbMZ De1eQRie TF7mNyvEFIc0ROaesntwYTxiNoHXaS3wGvSPj+Cklw3p3SokAv/Tb5a2H7O00xoTsHwNydB5/tDjP6HlTigIaJFKcdop6/yPW5kt+WwQmE1WDDtJ/g3+uxtLdtC+YJ2+42/G4Y6IhJ4Gm5gv/mnCdCgxUqSd5naDab1SYSOvxj721+BY3Wh7TfEMxcMtObl6X0pCXmMQHFQc5y998EMO8/GVmxfG7SG65/ANc+6t8O/fu1cUIxSxrLmV+1G9DxI2/gPS9Dm3gnxyQbm6f7q5sAcUzkjSZpxaTXEfPGmT6xzZQv0IGz+EUYTJ0AMm0WhU8zEHbBNP+x0J8xeDE3POQXls05yws6gA86kPc+mHe0DVlydfh5P8pLS7gwJC+ZZpwC5xUQKfSNiJs4DIzDCFUtQPlCXa2xFncd7V8I7ISyVuw/1lD92p+eY3IhbMhvWaPnyIXrCZ0oDxENTrnaR039h1rejkh3q5Zii2ZvLZtwSY/T4RIyvGLY3brB+Q8bK4ra0VVK0PJWdYteji2ABzZCs4mQQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/udp.c | 50 +++++++++----------------------------------------- 1 file changed, 9 insertions(+), 41 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..0d3e78a65f51 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,52 +1332,20 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_MORE + }; int ret; - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; lock_sock(sk); - - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: + ret = udp_sendmsg(sk, &msg, size); release_sock(sk); return ret; } From patchwork Thu Apr 6 09:42:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E8DBC7618D for ; Thu, 6 Apr 2023 09:43:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4582F6B009E; Thu, 6 Apr 2023 05:43:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 407F56B009F; Thu, 6 Apr 2023 05:43:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AAE56B00A0; Thu, 6 Apr 2023 05:43:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1B3E46B009E for ; Thu, 6 Apr 2023 05:43:51 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D0C13120C85 for ; Thu, 6 Apr 2023 09:43:50 +0000 (UTC) X-FDA: 80650479420.02.D9E621F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 241ACC001E for ; Thu, 6 Apr 2023 09:43:48 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bnqHUXKF; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g8CQfKCAD+XBsRHySlPh9qrpCvVJgKV8Pnbt38JTMEY=; b=zFn6zhdK9mYhMMEluD1nUstiNPpK8T7RkPH7xOqUCJtMK/Wae9w77oQHqIT9IsyV+vBt0D YuXKFu7GNzFvqKhlaMch2MpQ8OZTadgdAwLUcWIuIPBfAnBu4Z+U/uG+fQCZfIitXTKJvQ eD3gLqlCFIwRSl4BLisWoejRP8+ZaEs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bnqHUXKF; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774229; a=rsa-sha256; cv=none; b=zfqOyvkxbc1yrpNcS9c++V9sbZ5E6Ok6jXY30NME7BkGeOBByr9wwjNOoy/Sd8ToyHa3JD QsbySsa5S41Eo81g67Ql2odELgBpGzYaUMvHGNyuKqThI9ha5XljncgWB5T2tgzlf8ILXL yN6Oa5VEBTLHTGP/wVFCRKqFif4C6cU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8CQfKCAD+XBsRHySlPh9qrpCvVJgKV8Pnbt38JTMEY=; b=bnqHUXKFWKfrXYuA6F/stfgidelSWH/1ztdtnfRuNz/UQpzs9hPnIeEbwFWBRTBImORlcB erRdaHIL2c9f4eZSRWgOhrgenqFhLz+IVi60FXn68FIPMuKiYJLkHyFjv2jAVGKw5zpouO kPej0wYSoMQB9Sw7Z7wbwbcrQmHNWjU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-74-HgBLwesAP063aPz_uGijkA-1; Thu, 06 Apr 2023 05:43:43 -0400 X-MC-Unique: HgBLwesAP063aPz_uGijkA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F0B048996E2; Thu, 6 Apr 2023 09:43:41 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD68D40C83AC; Thu, 6 Apr 2023 09:43:39 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Ahern Subject: [PATCH net-next v5 17/19] ip: Remove ip_append_page() Date: Thu, 6 Apr 2023 10:42:43 +0100 Message-Id: <20230406094245.3633290-18-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 241ACC001E X-Stat-Signature: je3yhrr3tx76ye6ohbxoysyxsbu1u55g X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680774228-346091 X-HE-Meta: U2FsdGVkX18QYx/nT8qM8VfLC3Ky+mMijOfXLXioaPwqkQBlbqykSRdwDGxXT9gfCyiBaR4emY8SYty3Q9SRewTRvR/sWiMLot85Egck8fgawLAVQD2I4/NYh+nggp+5jGu5rFiUaGonV1Vxuac+hkVnrqBzImuJSv5c5wokmUJT6q0aELRSIr5HyKC/NzA1FbuDwUo4TuoJ8WXhj0U9f7DYkfRaplKo/TlOLb8M4OPUjhqaPVZ69ACKnzcDlaSt2N7YDRZBy8sYCWCSklFY13PZO4EiU3KbhUbNA3BDWpCGDbxaJhjuFbS0KN0Q9FSmYYRtN6NAZoq1TRpYF18Rm+8xSqxZ1jfbuL/BmfFAsPFT6IKJUwRg0EILQ2DVAoE1ximjzVxanGX8ugN6Ibh1iKjNTV+DaL2VzaujDK3jaY5wz+VJH9/Uef+3HFzSL7wTsOS8ehC7AZmx+T2o7FqxxS28yBRsG4F1QFS9Ghql/UuyK/aPpN92RVTIRoYuVMGi7X+jzp0wWNUDxrIgHO6xsTD9YBIIQYMdkeXhTeIdsNzH5aJH75Yy+MJ4KpoZsRjWATJaNDCOQT3Ff6TSNEua1hfJBmL0n05bUFBV7CFOaukiBTB40GPsJ4Sk7i1RaW33/xoKOX+tASM/yyF2fthMBBKXK/tD12L8h9Yvnc9WGOH/XYi3BqU+3j4mFRvkzP3vqg4Z3WipwWoHMyOcRuVKSDCPOEXsjiSivCbbsmJcak/nfh5wyExW0b1+cOfISoliJpNdjCkNh5zTsdguBKgv5nGCqp2AqaP+0DYC8t2F2eaI7tUHDkRlRrT77cplkam4W/HvpM28WFBdk1GCLb0UH6/lV3o5d+hKVBu0RzibGsGFlS04j10VI6CWtO7pItnNRTeL0928l1T3TLBqpiaMGY8vEs35U+rlB8CawTMty59Brw95zFKMLI5YuZXVUwC7zTcHIt4/BPBkxIUsVzP tZW/tpok MZE+quskNUB4udMa5YhpXyymmi+hY8lYovMzRMMGmZLfTd4J7C4hKkII1VYqJfNjlqSlxrbdxhmTiqT91sq40OwxG0RCZmKYgVS3QgVp178UnbeRmzY92FbTgn5zN7FritXBs2r/IHhzMn7f4EyeYqu9mLEhzN9kBlRjx+mEmgE1vDdnB+b3yYYUQJ1ZqTkB2pLQSTIuedINaWd32h+1DV5zhDqd3woCCWJtIhLroH+HJTI5jrG4wEhAFiGE5m++61CVagdrh3MqvWnV6ADaE3aqhIEFl3mu66D0WDTFcbwhE1aweEkbaMM39IhjR6lHLiG4Xhiff6BuvjjcJJ7Y8DLNet6O2vHG/t4EfY8EQHax1OtpbuvnBccHWeU7TGB9xYQBpN0O4SHkPKre0KFL6lD0kurRqXX1vImAsauW8U7tTtC4cbw+V+QupuKAvz6zb2d32iaazmdTyorkHyo5oeyleOOUsJXpMIs/vlBTUqmxxURTxA4DDzh7v0Q8JqAecyQ8vS8UojnLg0OIumaq78cwpUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/ip.h | 2 - net/ipv4/ip_output.c | 136 ++----------------------------------------- 2 files changed, 4 insertions(+), 134 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index dcbedeffab60..8a50341007bf 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -221,8 +221,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 5b66c28c2e41..241a78d82766 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1376,10 +1376,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1412,134 +1412,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Thu Apr 6 09:42:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01710C77B6E for ; Thu, 6 Apr 2023 09:43:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 718516B0075; Thu, 6 Apr 2023 05:43:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C8816B00A0; Thu, 6 Apr 2023 05:43:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A53E6B00A1; Thu, 6 Apr 2023 05:43:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 30A376B00A0 for ; Thu, 6 Apr 2023 05:43:52 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0887E1C6C3F for ; Thu, 6 Apr 2023 09:43:51 +0000 (UTC) X-FDA: 80650479504.25.2A9A33B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 5B54F14000C for ; Thu, 6 Apr 2023 09:43:50 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P8dYSaBn; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774230; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5yIMxPSUSHRsbpXFBHBFC5Cox+6AF/paYR4bJAbS7VI=; b=J+fdwOU+vplUb6SPGbL37omjDklKWYE2O6oH3+LstwpPtzkRw/pQzz59sfh49Ol+4PmAVU /Ns6ykJS64uoLxrpT6nksZ4orMjEXryxGTKd5ETVfl95/HsG2zx0AcuVmEIXypFIzUidOE c7A5G50hDAqlTmvW6BoiAjbTEO5hmLs= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P8dYSaBn; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774230; a=rsa-sha256; cv=none; b=iMQxcBZT/kHlU4z8L5GswTjXMbKbEFKm+TXqj/f3YUyhhog0YjKPL4IdPJs09CGboIXwZk SybCWm5WRJmY9H83fDsdnNJkligqmNCLUoi1u1g1BYnja4kkKnZygoe8owFVgC7PWdZ70P xLL7Y44m2C5QMnt+1p71laLawHQtL1s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774229; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5yIMxPSUSHRsbpXFBHBFC5Cox+6AF/paYR4bJAbS7VI=; b=P8dYSaBnH8s7FCi/zFM3DR+Bs4tuiRwo8R3whldUb6w9eJg9k3hxSYNwSqmIXYfDrGf5mw 6/Y7/Oc6BI0AYbzVy9PhWNbuin75ItXbr2mzkKkc/Vz3axcexz6HjKYzqX5CxEefvyRmEJ 7+MPOq/X/UtAZewZq0L8t7VL4WNmbxA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-319-vnx3onwwNa-Pcv1t3H8O3A-1; Thu, 06 Apr 2023 05:43:45 -0400 X-MC-Unique: vnx3onwwNa-Pcv1t3H8O3A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B2F60811E7C; Thu, 6 Apr 2023 09:43:44 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A25F92166B26; Thu, 6 Apr 2023 09:43:42 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v5 18/19] af_unix: Support MSG_SPLICE_PAGES Date: Thu, 6 Apr 2023 10:42:44 +0100 Message-Id: <20230406094245.3633290-19-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5B54F14000C X-Rspam-User: X-Stat-Signature: 8xyt7ipkbkk5y6t39d136hn5fwxkwmjo X-HE-Tag: 1680774230-188473 X-HE-Meta: U2FsdGVkX18vxCst9H1mbw3Q0r2h/Mtix5hDQHUU7NgWK38ZKL+jxxunmoVoPUQsSbxtSRgx1LDHe6Kf8gSshh5YAGL+EWld+dXzeUM7+95HspYcMVdFXE9YKPpaynEqm9TjHFN7qCvor+t1VdTHeQopKur43R+Bg29SJSd7gh+ae0Q7KI6WctvrNzbdpzoNH9Vy1d/AOD4J30mK5bV7WMUEK8RlwlD6+ah3KBuV6w+FUPPVXGJM/Ac27zpjzyst6mWx5WZ/x30QsymzjD+iLX/UuoyHGud/ZqQ2utTmu7hHhsrEzPzNipAqAtpJMb0risi+Me2Ykm4HCxKiYH5uFBYpq7K8VlFDw56wsU8njP7eZykdqfPn5qx/FD273lzfjTRr7hRR+qeMvgSBYxktKPqqYpMAef/bgvx5tmWDeKiIvIyCwAJ4sB1C3V7PguO+xqnvAN9hmUh3XG/rN1JVwecyZesJd+EotOxpCz/SOgblGZOufrxollVtIO0KKjCKmiBsv/5UIjE73sfCjLtWw5K5QkeiGOWPBMHZrlemiUofcufvL7NArsCVcBKTHeD7w+mkaPGSOySisA0iva7RgrzAkZC5UlCKDLzBjWkhUCNrnYeHz2fzyRTI1Zl2szcACNhDA7RfgeviLdhjFbogTpdPMA4LI/uMg+A5MSsM3EkYrUxxaVZ46spxyBT+CSB+wfmSmh0CVZK/B30rofdpf6LLsR1OfvYD2OgNaRp/rGI6WqBWj0TqLKVIISu15QQm1/krj4E73H23JBUR0rFlOajVeLaIcuipm5cKo5AjvPCyzAh/82QNzAY5iQiGl8+SmDujRi4ifMYdpWvP2O9Zct162UP98QAXICMB4Gq4YSeZLcnuyCgR/uZGo1WxkHv30xqxPzecSBnbuLAvrPg/Mh5qT02eMO6sTA2hmEy0l7sEUXZBHI1SEUkgk66D0i8Woh+FI4IBNZHAYSw0kqT enDftJ/h iO73e8Ujem7vZYKkEBeUNe1ZtXogXphVNm4d5OSi7Uun/liXRAK37baqjE2A2wgUO7U2fr8lKXAtAkCc7E08DlQjshuYjabyt2jpNax1a9TiPrTCoUqd3q2J5BV/zVS2R9SY20j6eI46zAwt3gVXl1fIkUHlcfVSY+qxorvXyyNQReEuQrDykeebsmFz24NkGEO/jSs4t3LmixdNiCZYKNFcNZC6MmbxaAEmsAzIiueTg/MRZWujxQzBcg063LTppaGKnBr5At2MinEnYdtPV6mDV/xvunvFIP4LF+kOE+TUnmcEfwuod97Qz4zNIJajAkeqcclWkh/XkGs+0xKB/R28oRHIQg3NyV7mXbTVhhr7igk5RrFse7YaRus5uGNqnG2zZpXK4blMndqFJiBBDNM79e5rcQH9r9rADAOpNRppZzYAvg/rI5+XR/dp/bP9UtpCZbP4mJw+zXAdRT4EwN1+b+QlSmFwp1C6jF9596pZmuvYzhvKiz7mO63pfuSqHAi+42+pv86em5rfC1l3qxyretQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 93 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index fb31e8a4409e..fee431a089d3 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2157,6 +2157,53 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other } #endif +/* + * Extract pages from an iterator and add them to the socket buffer. + */ +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, + struct iov_iter *iter, ssize_t maxsize) +{ + struct page *pages[8], **ppages = pages; + unsigned int i, nr; + ssize_t ret = 0; + + while (iter->count > 0) { + size_t off, len; + + nr = min_t(size_t, MAX_SKB_FRAGS - skb_shinfo(skb)->nr_frags, + ARRAY_SIZE(pages)); + if (nr == 0) + break; + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + if (!ret) + ret = len ?: -EIO; + break; + } + + i = 0; + do { + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + if (skb_append_pagefrags(skb, pages[i++], off, part) < 0) { + if (!ret) + ret = -EMSGSIZE; + goto out; + } + off = 0; + ret += part; + maxsize -= part; + len -= part; + } while (len > 0); + if (maxsize <= 0) + break; + } + +out: + return ret; +} + static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) { @@ -2200,19 +2247,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2277,21 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + skb->data_len += size; + skb->len += size; + skb->truesize += size; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Thu Apr 6 09:42:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13203066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F37E0C7618D for ; Thu, 6 Apr 2023 09:43:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C08B6B00A1; Thu, 6 Apr 2023 05:43:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 123976B00A2; Thu, 6 Apr 2023 05:43:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB8CB6B00A3; Thu, 6 Apr 2023 05:43:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CFB0B6B00A1 for ; Thu, 6 Apr 2023 05:43:53 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AA2F11A0F23 for ; Thu, 6 Apr 2023 09:43:53 +0000 (UTC) X-FDA: 80650479546.14.309989C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 062D7C0011 for ; Thu, 6 Apr 2023 09:43:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hJllBV7S; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680774232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ryfphnd6F+ChdmyXxRDqpuzs5jttBzLaOe0KjWoTLBo=; b=oBbr+tnOd8MVZGn+n2RZir4XUCCVupK2YhuLAAZvDFUyXmXzTwqmpTC6Y1U7r/g7PcWUpU O+SpwLvp8tFJ7GzH+Khvj+No05k3Ha8lKuX97PTumYA/im/G/0STRhm7uAr1RwQkyuoNJ+ ktORObLsf37hAFQBWgk7JcGuVRF40Vo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hJllBV7S; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680774232; a=rsa-sha256; cv=none; b=0N2/Fu4/6mZq7WqQgjkTUMabk1ylA9lKRwwZZUdSsX+NziHa5D9VWv2yzzZbVEVk5b0bA0 TVpMyvs828fiImhWXHiZhiMMUFQYSIpeGVDpHldrPfh1hac4avbtg3Dft1yQvLqiLX0Rg2 pkM2fw1KzqophtEHfbx0FIW0Tiomb9A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680774231; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ryfphnd6F+ChdmyXxRDqpuzs5jttBzLaOe0KjWoTLBo=; b=hJllBV7SyRXufrI4TvTs3MJ78xemK9j1bW48/7saYqMK1n5z5jbwh2QEq25UQ6h7pYwneX X62YA3BpVKWgnKBaErf97ml80PbghR1wDnmH4uAafAmBSqmv0pZqup3RN4FZDeJIMplsKY wIp7CoVDtn4TOXtHNuh3WHLmaAsmToI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-316-BqhN1L0dM6OPW16vDF8pVA-1; Thu, 06 Apr 2023 05:43:48 -0400 X-MC-Unique: BqhN1L0dM6OPW16vDF8pVA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 61D36185A78F; Thu, 6 Apr 2023 09:43:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4E84618EC6; Thu, 6 Apr 2023 09:43:45 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v5 19/19] af_unix: Make sendmsg(MSG_SPLICE_PAGES) copy unspliceable data Date: Thu, 6 Apr 2023 10:42:45 +0100 Message-Id: <20230406094245.3633290-20-dhowells@redhat.com> In-Reply-To: <20230406094245.3633290-1-dhowells@redhat.com> References: <20230406094245.3633290-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: nt9g6yo151wjzwfs17e15goghxhizc11 X-Rspamd-Queue-Id: 062D7C0011 X-HE-Tag: 1680774231-26584 X-HE-Meta: U2FsdGVkX1+Rl81pUIXcUk9BAOjNnHXO2HEy9PeHWGRB1RLoCoX3wkX0ORotBUs8lzwosdpmJV2AvrCzYtAP5U5C5MOzRXDu41QAqlev1YA3OF2+n2NX5DWVI/Te9G2AZLlecG1lvNlsK31rymEFr9A2/Yl0sWfdp8uy6Di/l6Px45yjD8GZU6oTcYx6vKbTj2cR9B6b4qP/q3dXdg2meiu5wDtBCdR6VfMKChh9lY+AFNKwkMC/4FFivWbCQaWRLRbn/U8rw6JLnFFLBX4qYtWighXlcbLc6HtD9iVcY8GDNR8C8CIWEgH/OPwRdHXsQllg5dwGOEc8Yp40LyF6VNsepbP5JNZWu8T9VO3v0+PnGs0eDpb2to8I6CePSUSe/uOYl4H2vvvmYtrTD/radWvvU+jw6DjgYPtlPCG+zbkUR3GB8xixiS5sHthAic3cTewWNqc6FnRahBN9I4n++gTFXasBKX4zjTKhAXFP7crca+wAlCk39tLM+1P6lycV+ZwT1+BCQAXtsIrcU8oh+XUT7A90hkRQkcwfI9micO3wQVJFYIktQRyTkl7bgCDR8dTX1Mau/5icXGrGrfrGJQDTyxvYUfz0ROdUL84jRdqNiW6JwU+/nQ/n3NQBmnfE9QZeOWULmKGTZd1blV6HW5JkfJdcQhlT+98+xVVgmWMSw9nZgAT2ljGa3yRmRK/mH8vpH5CuchuoU5Eg0LopAdLHFR9lhearnOoek/QszhPXQLgPr9COdbjHZXjtg2iOBQEcBNU8sFL0suFNIwtQ0yYlJxWddE0FVxPGmaBlUukPswLnrdpm+hh0Np5MVOAtN1HqhPv+VEYjn8Mx/xyW1m+wHeYV9pxHbQbzcGtvEtz4g5zDDLfJk6rHi52+CBCpTsaRtOmyXiT41TK4rhX5E5xH2YhDg2l1iK+UhY8uRwkTS4P+pic6C4f8MXcNelagucj7h+im3RFD4YQAlR/ 4gY3cRfI I4MVq1qleIh3vnuTnBMNC7zCcHu+UFp35bn1+I01qFBLwKu4bF4jppO3QfvCY8kbpK3XYmeGNuMpHyoBcg3J1N8IfSMUIDiLiDIdCns8gk4DBWy0JkRdTws0XqcHRzaqSIEMw9/jrdKvRitwl8ypGK8QSdBVEr2moYJ/tRTFk0w4Pse8zSYQSrMJ7eFq7m3OyYFzSQ9o7LgEIDfA144zzKzDbCLqZsgKjhGwFofD8LArCC5pWbgh+8suhld1IzVNK2NCnP7MfvgjBsYdVisLVkP6nrpi7kLziGpqt1BX1R2Tk03bIs+eZdGQcCiVvlWzEzONceYVSg61fylj8lqhEABXmgrOC1KveuC/9Re63NSnTjdXPFGxleH8tXQ5n0jWAvqXBNFx3cXVMVll7ZEgcc9GXSwwSxY7eXjZQxtwxw0n3OFDjpiAw/SQK1FLUoHN7n/05i6HHcz0LHXxZ9r37vV96sPMWwfiWVm2UQxTRZOBt052Dpipv39GGBm4yQGl0ljDU3EGmhUxL3qJHeDxCz3RMPf6JoHBIVMBJzpoletf3VMZpEY1ch8Vp1ZYiP4CFB8JLV9OHjJwwr9I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If sendmsg() with MSG_SPLICE_PAGES encounters a page that shouldn't be spliced - a slab page, for instance, or one with a zero count - make unix_extract_bvec_to_skb() copy it. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 44 +++++++++++++++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index fee431a089d3..6941be8dae7e 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2160,12 +2160,12 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other /* * Extract pages from an iterator and add them to the socket buffer. */ -static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, - struct iov_iter *iter, ssize_t maxsize) +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) { struct page *pages[8], **ppages = pages; unsigned int i, nr; - ssize_t ret = 0; + ssize_t spliced = 0, ret = 0; while (iter->count > 0) { size_t off, len; @@ -2177,31 +2177,52 @@ static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); if (len <= 0) { - if (!ret) - ret = len ?: -EIO; + ret = len ?: -EIO; break; } i = 0; do { + struct page *page = pages[i++]; size_t part = min_t(size_t, PAGE_SIZE - off, len); + bool put = false; + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, gfp, + ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(iter, len); + ret = -ENOMEM; + goto out; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } - if (skb_append_pagefrags(skb, pages[i++], off, part) < 0) { - if (!ret) - ret = -EMSGSIZE; + ret = skb_append_pagefrags(skb, page, off, part); + if (put) + put_page(page); + if (ret < 0) { + iov_iter_revert(iter, len); goto out; } off = 0; - ret += part; + spliced += part; maxsize -= part; len -= part; } while (len > 0); + if (maxsize <= 0) break; } out: - return ret; + return spliced ?: ret; } static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, @@ -2278,7 +2299,8 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, fds_sent = true; if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { - size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size, + sk->sk_allocation); skb->data_len += size; skb->len += size; skb->truesize += size;