From patchwork Wed Mar 29 14:13:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFC3CC6FD18 for ; Wed, 29 Mar 2023 14:15:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EFFD900004; Wed, 29 Mar 2023 10:15:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A0CB900002; Wed, 29 Mar 2023 10:15:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE4ED900004; Wed, 29 Mar 2023 10:15:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DBDFE900002 for ; Wed, 29 Mar 2023 10:15:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9D0751C60C3 for ; Wed, 29 Mar 2023 14:15:52 +0000 (UTC) X-FDA: 80622134544.22.A0262AB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 7A319C002A for ; Wed, 29 Mar 2023 14:15:49 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V86LIBwS; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099349; a=rsa-sha256; cv=none; b=qHgnccjhZ5LDwob6nvQnlxrSp0HKl46irOB/GRgwkTsLX79ayORcy2TAjGQYMEmWHKjO94 xNKkQjejHuA2xdJpY12JkuNDYMJMlpoLRLdnskqXWoYGbgiY38gK+n4lhOjSEJPd0R9eQ+ LKYA7gGwPYrLU5IdGAbIaZy6WmQgYBI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V86LIBwS; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=U+eygar7rqS4PdLVXXM5nMIIcTG+n39OJc0zt46Qdt9uDbTl3cK81X4Il9+Bp34aV3vBEa rlEmLfhr5MQZGrR/aRVHVKvaVqEqtnQ/oAWeQRtI1K+RIkhwOPiHrOrH0Ft5dfmxJCzvsn uj7FUsIbxVl16/brWhT4oUopDr624zM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=V86LIBwSGpkib+VJnc4A7sLtmU4N1fe3eRmG35IlpueGTtNSkHJosMJc/Pm3cCiaE6gf2S L2BFbpKsaWUHY2LrgQLaMfcHec0LHbQZgpM09+Ew77tZsxynpTXMPetnFjmGARiFeh7cU5 8zirau/ZGS/QfRE7EctJaMw78GGxw3Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-494-exPjG2AiPg65VJXx__yQjQ-1; Wed, 29 Mar 2023 10:15:45 -0400 X-MC-Unique: exPjG2AiPg65VJXx__yQjQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 99A6F185A7A4; Wed, 29 Mar 2023 14:15:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6BD31202701E; Wed, 29 Mar 2023 14:15:41 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [RFC PATCH v2 38/48] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Wed, 29 Mar 2023 15:13:44 +0100 Message-Id: <20230329141354.516864-39-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Queue-Id: 7A319C002A X-Rspamd-Server: rspam01 X-Stat-Signature: i9dzoj839rkp7s9tpmmp97o7j84d3t1u X-HE-Tag: 1680099349-590121 X-HE-Meta: U2FsdGVkX18tk5BChFRTw9Rdb0UfVC1qIqyQmV7385eXff8oMwRUtMUv8Q/ncv/I6UEBAI/L5+T0B5Sy6E6eb3zDUfy+8aPFdZma1S5xyL/uulukDa17qMx4eA742mIlzKFC4THQsM5jWv+DjdFqr5mWfykfaDp7wOss3XyBSXjKrmkhKpOBRzjnNwRv4afT8zNsaafGbYaEAJH8f1Ylzy9tjz/mOXE+DDwQ9Nj/44SBa4/9Hn8Bum9fCreHqw5SHEySDhYeGkfLu66yWZRnuGJZTPrHAM89A8UXOtzOnrsteGCIDAmw8mmVQMYdaytfFMOmnTXHE5BrdvuFmHYzIenp2/EBGOvCl6lq1jOzAl5LTj8Kq4F2DDPgt6b04R35ouK+OvS9JlgF+KMVKn/yjyyenpgroTu9Izj2ttcVzM+ICEfXFX+YFXuTpdrzdz8iSX/tdd+ISyomuU90boJIGFdUN3wE4C99Gk2ED7Vf+qjJPirZ08EtGNoo/Ugl3D4Liub5U6FyePfCKfSWa9R99piLYaTB6oMNDWZLMG1ZjL0qaN8wRIcjxu3oAlKM4V3RCaOKic91GEPBQGth7jSji38hzEG4OhR78pzOE4ysp+NTZZpg6U140SwoRAC3Z6GMidATY9kB/yzSG8ONNzFBwR27lE3oJNyyxDcHvbsZ6/9gFHoCS2TsIjS62KY0xxt8BZwvuVZPeK3A9wLp3efNG4p+6c7LoAgme70VfpWK69t6ymV1N47GjTK6azc/KTHYynQOXRwk5bEen8uO+0aWBmd4chxt8ANelEGsfGyL/tWnoj5rT4wOusv3sy15Ug7h7Li97nxZH3Ixm+Ln54RdpjFOQpi0f8DdfRE/WFAZXiStizZ5qZ5Mcg/gkuSROnFLzFSLC5oEk695D3zHCTJgdQvI9zyXzMfAw25h+J3MtyUV965deN4NSTDol6CzVP0o3fQwJX7FQ4HR71vY7Nf EHjPQrfC 0g1L69m4+FFyGGWs9SkiOQSErrgVsZho9hp7M8JaDw30U5v7lThCOQFO5HT1ipdNAT6NlsAu1RSdn9/Za2EdflBlojCE1IvwUDoloGHWjFIitWGBZthEDeRfj8fqm3oTztH5G5WO/T2nVpdxE7o9mRGH9sJLYBlsdiAAhEdLhNzllLIvo0TvY1nLslHaLOs4VJtcg2I5h9MpcBrT+wwhmKnnhpvbil6QvSjBC2HUoUb9TopUnOavykY7Kw+3EGo/Mey9dYfDMZEVi+qlky+49e8ydw6DbWS9H6DFe86PnAFKacqei7a+ELWedxxKgZ72oOnKzYyzUYVT7FbvU4cxxk4GeIKftId5PHD7Hgg59utfBTB70ptNXoMtPlBgfrOvZkrOBbZVPXbRGIYXWeUAMojJhk6Y84x1ouq+geM0YswH3oWeMMij2XKDQusaHEJPAmwqn6hHTqh4CJ5eeAxs/lkC+cg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 86 +++++++++++++++++++++------------------------- 1 file changed, 40 insertions(+), 46 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..660d9f203d99 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -52,29 +52,24 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; + bool free_hdr = false; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -99,43 +94,37 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, if (hdr_off < sizeof(struct rds_header)) { /* see rds_tcp_write_space() */ + void *p; + set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); - if (ret < 0) - goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) + ret = -ENOMEM; + p = page_frag_memdup(NULL, + (void *)&rm->m_inc.i_hdr + hdr_off, + sizeof(rm->m_inc.i_hdr) - hdr_off, + GFP_KERNEL, ULONG_MAX); + if (!p) goto out; + bvec_set_virt(&bvec[ix], p, sizeof(rm->m_inc.i_hdr) - hdr_off); + free_hdr = true; + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +147,11 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + if (bvec) { + if (free_hdr) + put_page(bvec[0].bv_page); + kfree(bvec); + } return done; }