From patchwork Fri Mar 31 16:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC74FC76196 for ; Fri, 31 Mar 2023 16:10:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83A1B6B00A4; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E98D6B00A5; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68BC96B00A6; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 56EC66B00A4 for ; Fri, 31 Mar 2023 12:10:52 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3A8A01404A7 for ; Fri, 31 Mar 2023 16:10:52 +0000 (UTC) X-FDA: 80629681944.11.5219F57 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 6D5FB4000C for ; Fri, 31 Mar 2023 16:10:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MhzwnM/S"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279050; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qgrsaUecWqqjS48xBO3DK0ifte08BTz2ZXAzew8KrV0=; b=zP6Mol8mOQPttkHhZ4du6VugwCWuIqqV6FIWnN0jE84phJCo9gJ0oUHRv1xl7u5YSUh57U DzvsXDOdd6Ev4w8LZHR4ro6OYiNRatuGuqoFyUrV9uE5aXHSDSTikZOV+W6u0g5sTZr5cW qGenxox9bUNqCOs43kSK8I50F+ULixQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MhzwnM/S"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279050; a=rsa-sha256; cv=none; b=0YDDSSHeROKmhJ85Vv6rrcgzVAG5KIRXXVOJpXISUmYuAsHRQSIfL2ci2N0SMyGrPoTe2X GxcP9enRiDngsyzi2ugd31vtpcLFYIyV6AUPRAbAqRGsPhSVeb1706gJibBhojK8qiwccP C9y3VPcIKSVyGaEZLUzcdUM4Zi7peWM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279049; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qgrsaUecWqqjS48xBO3DK0ifte08BTz2ZXAzew8KrV0=; b=MhzwnM/Sf8/ug2PrOb4iRm104bG3aBUo0vHl704bsw85/uI7/upQfwioLQDxEEC+wxclCy f3fGW0lQN8XKpD/wZAG1axFHK21/QO6p175qmYhQdviYZ25b7SrSSz/FCiNyXSbaebs313 ZdK275BwMMNs6KT6vwZjE5QRwykRA2E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-146-u0t01DzwNZa0rrcf5yQq9A-1; Fri, 31 Mar 2023 12:10:46 -0400 X-MC-Unique: u0t01DzwNZa0rrcf5yQq9A-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 875E28028AD; Fri, 31 Mar 2023 16:10:44 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 745581415139; Fri, 31 Mar 2023 16:10:42 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH v3 30/55] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Fri, 31 Mar 2023 17:08:49 +0100 Message-Id: <20230331160914.1608208-31-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6D5FB4000C X-Stat-Signature: q8ho8hdwfwsjnpsowchcx7r85xbw6m6w X-HE-Tag: 1680279050-169094 X-HE-Meta: U2FsdGVkX1+EHLkpLGkPARURcBemLxmBSa+Y9nN0XYMUHi4C0jLAVB7cM8blq/i5zvmidUR01Qz/dH4Yztg0GHN1Lg7kcIBemX2XOQEPqKSND1X+hvhTDu4nPyIBXsxxH4RPRbFTTbwVstIMwgTX3DK2lN7AiGXgDSHXExuu3esjyHzjqXjpizDwu6IUU0d1XnqN+Vz+Msv2qgwp4OcJaX2QO3716QUBq39WO/eIO6MiH48EVLH/FdYfHqSd9XQlcEUTQsaPmCdj3cel201Mu06RhrjxBcgXoW23OlCtMvJSB96w8e8Mr2ZvketmI/lpBxsq4dAIUgmy/0Uz3BvF9k0ivYoQ+7FdQm3a+8JKOdpf8Sau2EQd5bPmlknuEHBaQcVun1xoCekOx0VMEq0pYMn/k/WEDqSI5AN3eLlXnqaNAkpOBbieHQEoKo578wBPffK5xdVNKzDQFouiARXbgcD70HuW21STkhmT5Lc/HkWvvSjK5Bsu02fiTKF3o+/F9+782/rFCun8YJe3xLYhueOhdzjUJLmKR8AFNr2vJEBapbgT0UKkXcvjRZIdF81y7PgGNXuTSuKMTIDg6XINYL+4zqq317OQQPB5pN/35wx9m6TvFPKllVmZDXDuXqDWAiJYPaRlWz4mcgrorNeT+7hy1UJkcvI6rxuRAxMtNDhgRErFnDhPfbebm9V/3t9DULzh6QJSprBplr2+DVzzEmB69v5jzkdmUm2/j4iO3hWIsSHk32GIAXBwdDzT5gSV4LHSTs9czFHMFlaCtKXA16pXYolqP6/Uh3e0sxx3k0xAG2h6Ldoag9WVJPpCsG3CdRAHiFDqmGLqVQGVTzHIZPmsNRYPNLGt8gKiDAQiI7L/ApyAEHQixOS0XPhstfHy6g+efFTtBt1yb7H7/BvPNKr+bh7Xwm3yszILVM9P933yeKc6sf6DQ7zfcNe7dcOpVRn38n9ez08qN46rafB pfQ+aiIX y5UksR+9V34EBgD4kiI0Of3wr2Ns2FlkT/coX+k2lTNbQpej/0oeDb9aV4OzbjSnA3MgBQuvvuCGtbKg3t4One0tP2cLVlXO9ptwJ9/FuFfaGqCkUV0Ma/fE+ml93jqKF3T1gNkXYtWbhDY+HvF/Up3xBoyDtn0rdMzDrFMJhtsK+mUhE/dsUSF0/pR1n7VrGIyrsKKXc7laBTAI04w4bs1D/k0JJB9iBv2vkQs59uqsyLnJoO5uc2wjWZ7SLiruq8vYSnjNpBaBuaiY0ndxG2n4ymfAhEAzIU14uVDPxbWlG1puzsPoULaOrP0Y3w3Asf02ecQ1rkavtI/6csC4uFDm8p+IB8f7oN/WOJlj61YI8joliGZfFdqj5uqZI/kVEJfwR5N4nswZI/07Z9+dxWC6sjzVJiH4ZVeTBIwyyK8xh/U3MwnXWBocTsh8lqtsocHqyrJ8hb2IbkDd1+PcKBh/eYcmBjaE6VwPrCrrj90MTW3AEsj6SVNvcykq8lycZ0S4pHN+UaB/OSOvgfcppFVv89gnQMhUtfiGmt3tuIz5IeA1HILujIjGPwctz8LlhgEZL1RUxfXCL1T8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_sw.c | 158 +++++++++-------------------------------------- 1 file changed, 28 insertions(+), 130 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index ce0c289e68ca..256824fca651 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -972,7 +972,7 @@ static int rls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -995,13 +995,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) - return -EOPNOTSUPP; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1202,155 +1195,60 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); - - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); return copied > 0 ? copied : ret; } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } + int ret; - if (ctx->open_rec) - goto alloc_payload; - } + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; + mutex_lock(&tls_ctx->tx_lock); + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); + return ret; } int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, offset, size); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int