From patchwork Wed Jun 7 14:05:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13270796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76558C7EE25 for ; Wed, 7 Jun 2023 14:38:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 092988E0005; Wed, 7 Jun 2023 10:38:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 042D88E0001; Wed, 7 Jun 2023 10:38:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E260C8E0005; Wed, 7 Jun 2023 10:38:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D28198E0001 for ; Wed, 7 Jun 2023 10:38:11 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A8034160262 for ; Wed, 7 Jun 2023 14:38:11 +0000 (UTC) X-FDA: 80876206782.17.BCF1ED2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 9FC504002E for ; Wed, 7 Jun 2023 14:38:09 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KmtRkVTg; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686148689; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LDTGeEfLo2GvEbWeYwCYunQ9JPRlxJinWTpGXOSqvLY=; b=I0ZOecNxmZhefAc0vp8Rvf3W+zKhlnoQOsusmnz7KqSdp/SruUs9zJrvb43KPcZ3iQ5K1B let8HRxrn4O7GMlw5BV3GfqGhzfndBVduCZ2XReH/1lGcaYVqpV5z6aEerObdyJ/DPnOLe PGHun+aHAid5sRve7wpGcYmKL5rM0h0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KmtRkVTg; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686148689; a=rsa-sha256; cv=none; b=fKSbkubuSwS4T0l/YXcAQgPJLVUPJZHAG+YK9Bvq/gHP4OX/oKXv+Of8pr7ZCLEEZhsC1d bfbZIyNOa5cqZp682BjnbiLXUpueI58utFyLI6Kn3+ps8irbiQwHN88rWvZhhd8xbUmqAC WipyD1R5E/xamq27+7aeW5TH08Sak34= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686148688; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LDTGeEfLo2GvEbWeYwCYunQ9JPRlxJinWTpGXOSqvLY=; b=KmtRkVTgxunsBi7PVJu8HJVILYyI8E7eWwsCjPKtsq5zdkgm+oXZHlbbxnncvRXRA5JsV8 5sy0n+LVOFcpkHGC7fZdnjmy7XK9MCrPmoKC8osxOAkrNAOUHgJLkOap2lN1Pxb+zw3HnW xA3uwZ0JajyMeuHOt10bJwUqKo1SFXE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-331-X5b66WfYOJqYx_MNR-dG8A-1; Wed, 07 Jun 2023 10:06:55 -0400 X-MC-Unique: X5b66WfYOJqYx_MNR-dG8A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C08E2185A78B; Wed, 7 Jun 2023 14:06:54 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id AA26E2166B25; Wed, 7 Jun 2023 14:06:51 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH net-next v5 12/14] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 7 Jun 2023 15:05:57 +0100 Message-ID: <20230607140559.2263470-13-dhowells@redhat.com> In-Reply-To: <20230607140559.2263470-1-dhowells@redhat.com> References: <20230607140559.2263470-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9FC504002E X-Stat-Signature: k8o6imw1u6ghf73cd4ih5zitwngb6ozd X-Rspam-User: X-HE-Tag: 1686148689-112244 X-HE-Meta: U2FsdGVkX1/TPte5vnfrw/PkJZbFDPFhFWj8M+2D193354dbsp5loUrb5CsBB7fyJ3nyjc5AeCRQ6h8IWxLPGMnRX0pxIn7kjl/VliaHdXnASiw/xUd6vkp7MnYJHOBW/sQUDYaWoBM1N3aXI6XLW1GDQUjAVASEo3Ow0EYSNrEe3J7k/tfQHbAwdm38NvO4zjBQW3ABC+bP4x4Fzcxln6VIEtf1XDcGJZHz7JdEucFXlcVSMy1mqEadpH3rVmz46f+Zj5OpJ5maLBJnPROa0TzHuVOoWW2v7fDqBQLn03VtYIqBk+MoOHaWQHhYI4TA+TGTYd6T4427KVf828r8ix0EXmcTDaLPhUsEY1wMOAKeTScankzZSUbLTYmqKwl7uaLcghCdyQ1qE90LOb7QJ2mmwov0EpsrP7R0g3T6vYX1rtQnR6lRs6whbcKxMnl64sesR20dKstT8ZTxye47kjwAE7OVvOwacVQ3GTK+ymHojv0x45K9pCZIm3xTxJnMIOW9FEQcKP5eR++KjesVf8+rT28vqUfgm2qUhMzeoUguhDrV+FkUkNE0DtWDnnrwVviF95MUmvkMHzz9U3alZS6JvXwDMIwZ7jt4XtPp8udGCGlnJV0ElJv6YfvIjelcJqlw4G6OeC3fQeaT1LDvaTqYYkbBrZT2SsRSD+ggfwuMmvSdLMfyA7qnhyPUl4iOL3BtsMMGjtKUWcJy2wAUniPK2yD0hEu/FqyeOIyB0914i0wxhdssYpVnsoGCwB8BwmDGiLN5fl1FqD4NwW2TKEdalgOrYRx9pwyCgiXpuDHT7vlxMfxmC0R9BAKzL3T0npOz5XRn1uSEyyBJj3ehsKLiwiSqIY9ZZGPXK5m65wpZU1G9a5pa2LWp5Q4YPDIKKZdKoqujoILwHckH65lglWw0myMDlArpTJNtX04EPCB3DcQm+kknql6V4t7kbYOL+U5KPOLoM9HXFmpmPyw 1ePG0PjS xqQe3lFgv32jWjVM2qtAGMLwd2CmO91bUut8UoRPUjwRDPk2R34lng4JN35394XEYpWcp+XGRw34pW39kRwCB00BgnxfbY4DRGj1P3+5HuNHJH3oo8dP5NWkOMz8pvh3nGSsZoq0tIkd0arToYXnJlRdSf0OwEHrHLbJBLMmVXDAcBqkE+UY25eEcjF+nxBeRPpxAkWSUOUyaoEFaWFIyuwU8ESGoUWkEzlNxyfAs3yP+G9nBNNCU1WlvIWPsa25XBmnX9yesveXAfhya3fNrjHQFq9EICTIjkRX5ag+K30/er+QglP3ONtzw2ByN79Yux268wn6h6nxK3800ro8brwGT4AFrK73f2SFyUFwf74JCG7onBDqmyV/L9lUuC2C43pm6i1nIIjJLQZCK37TGtz46v4WZQ+LWboTDL23iUTx3YZyTNnhKf9ZOR0SJMTjH/frt85I/rsYmJmMQ1lnNw+6X+j6cY7dDaNc4BiquJg59/fTRPYv2pzIAwW0FT/dLP6oJ/tzta6IxGduck8XP6Yr7kHb3xCLLaKWcfJYJnL+KnDOK+BjRI8zYQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org Reviewed-by: Jakub Kicinski --- net/tls/tls_sw.c | 173 ++++++++++------------------------------------- 1 file changed, 35 insertions(+), 138 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index fcbaf594c300..aafa02bb5723 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -963,7 +963,8 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, + size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -986,15 +987,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT | MSG_SPLICE_PAGES)) - return -EOPNOTSUPP; - - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1195,10 +1187,27 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); + return copied > 0 ? copied : ret; +} +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +{ + struct tls_context *tls_ctx = tls_get_ctx(sk); + int ret; + + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; + + ret = mutex_lock_interruptible(&tls_ctx->tx_lock); + if (ret) + return ret; + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); release_sock(sk); mutex_unlock(&tls_ctx->tx_lock); - return copied > 0 ? copied : ret; + return ret; } /* @@ -1275,151 +1284,39 @@ void tls_sw_splice_eof(struct socket *sock) mutex_unlock(&tls_ctx->tx_lock); } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) -{ - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } - - if (ctx->open_rec) - goto alloc_payload; - } - - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; -} - int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int