From patchwork Fri Mar 31 16:09:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13196207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6D44C761A6 for ; Fri, 31 Mar 2023 16:11:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 468756B00AA; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F09C6B00B4; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B8BC6B00B5; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1CF3E6B00AA for ; Fri, 31 Mar 2023 12:11:37 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A6DBDA0DD8 for ; Fri, 31 Mar 2023 16:11:36 +0000 (UTC) X-FDA: 80629683792.07.D2F6749 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id A63B2180006 for ; Fri, 31 Mar 2023 16:11:34 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8bwMiLm; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680279094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=HrfWnZkuf2JeSItqSqsRwr4MlCt5DEqQHMEqjShm+3nqvX3HfX0fm7EOI9Ofm63+Wx15md wDbk1WcKOKJmqasFxHsc9Nk4V3U54a3q7MTtZhv/kyyPSM40Ybhq4Yvz7x4zW4A8RZFszP ud87oITSHI+sewB7kLWDjcmgSyqj0vI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h8bwMiLm; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680279094; a=rsa-sha256; cv=none; b=8eKGeO0ZO34eWIjZjtfNkpFbzEKOWvGheNcwnjl5pmeKZeLO0/e4UL+72N1b1WOyDAVZur vngoL6o2rQCn17u12JsNzMJmnFYYIGHQEzY/yzs3ryIeILawWxPAQ8iisiT5uGHr8UcmbS DViimQYYbiV+yHQzumizOGs/ehwvcrM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680279093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pMIF11EWm/8cej/KXQhM8DA1wUu2yCmqcuxOX1F+AiY=; b=h8bwMiLmhys0FZ010tAMaYe063cgK0+OrbCwYI3qQGgOMC9ErPZhyMxhf33w/ygMoc9jm5 I9RlSLMx+PlZMMpKSCOLG3Y8WpXgezvW6XqZF7qJKcV+lejIK6BcDyzgoBJI9pIk5nSito xCAWdrQtb5vpNF7DlxY7u6MVGPvj2Sk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237-7dchcmswOjSwjHDFElj0Zg-1; Fri, 31 Mar 2023 12:11:30 -0400 X-MC-Unique: 7dchcmswOjSwjHDFElj0Zg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9D36988904B; Fri, 31 Mar 2023 16:11:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 724AC4020C82; Fri, 31 Mar 2023 16:11:26 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [PATCH v3 46/55] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Fri, 31 Mar 2023 17:09:05 +0100 Message-Id: <20230331160914.1608208-47-dhowells@redhat.com> In-Reply-To: <20230331160914.1608208-1-dhowells@redhat.com> References: <20230331160914.1608208-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A63B2180006 X-Rspam-User: X-Stat-Signature: bxunzxy9idnsw1uhfhfqxg1e579keu6e X-HE-Tag: 1680279094-195512 X-HE-Meta: U2FsdGVkX19RuYxy/ZoF2tUMv8SZ4HgB0XCU/8eFVx0MLKCJolPl6SqXiBBp3MoTFeQ/Px6pg2/U0EwH4lnWQ1PZfR3NTZbHt7VMxVEbr3UvLWNck5D9464OsSDbihbXIYz5uFDNNKAfgMKoPK4tK3AbkAwDS71tguD1ZaOeQZ4ejfU9f1cDYynVTtpMKCIH/IL94JAtG819Z3U2hdytk4u2nmIWcPVf2zQKfEvM9hho3Mn67pLjP9l+Pc9aWUwYR3z369VYM8qaSO3h5n4gYPfOkNd9b0K+9sgrc+mxSVTZ92Wgu4ocmDOWUdzS1afuayD2ABKW4pNgv4sKq4yHziWU2OQSywSMpcSsqndJcO+r0Lta08ZsIpqs/Fj6CYp2U3qVWOm66HtePbiCiwqhKlRdPJZRcZJrV3LbANSznp798cojilo9ZvW/p3fmNyfJbVYi0PY4i1mFGBQBJH00WWqVtsGBF8NLngT9qJEBDwXFzzMMpuNb6lJ1WCsF4nCVqWiwMp4FliI7h+Peuqx/9yIciarCnTaKvLqsTSABdBHbCZZ7NYBZvdzmC4xTM1ZIhzryw6a56pECJxq8mRbaNaSu8KZMLXMCsdJrCmxS/vn9QBntmO5qU7NXUpxANs5sj382hg9pYaOM3D57pQjg6EONG2rHBa0BFG8apbOsVxnmmyiGl5HCoKjr0r+BTxMRX4BHMuSZyA/ezrlxG59iElqLsujWG1Rx3995S+m2f/yiwHMXJytv8KWnwnN/iHZzfkB5K1R+7z6euEf470Hp/6Al0QfWsTLe/DVxumFZ3eW/nh+5TlhHtecEJ7AndQOsYrZVO6a+N6vd3A/4ilutckNHvuq3frzyBnUw17iGKqDHfv+caYK9MGs9Z4by/runmNXNn4PjVt4YUJOI5uQX/HzIJL6RAvCO5alsCMQmkNaaftTh28Gj/vWH+q62l/4PDG0WUfo4kYSyVQF5fv7 ID5q2yRx 7yOaSDGFiUhbNUbZJ8AQoFVGdirq+2SuDX4R2frBsbxr7yjI6Ugkpg1i1hiW0yUvl9NllhX2cVaWtcMb8ie2tkJvawOVHR3mAWJEw7B0s24IoLA76Z9/sGeMoIfb/5siSs3oG4a+RPEu+yex+eoO7ARd/aRB/0KuWa4nckVzo3XxGvW2MeF6u9JiozF5I6xKKGZVA3XctptR3RIZVdTFH927ApwoDIDpXfx9bH2TbduGljOTz+IPQkCxTkv+9OySufI7foyKkbixTXmpEBMnMJ2uXypCxVrlxJmPjwTO0pF2V3qziAiINe0eZHY0fuA6ca0pBBOKnITaRqf8Ub7X9t5rOywKlC4g3zXpM113vX+6Sl1Dk84dkDd7TSMMxvqNB+0BoiBDtYMWPxp2UY2PKBBgO/xkkUMyMciCr5kF5QxZOdeW/45UR9aPF/d5AStZYQY0LezZ8Sd/kll1+RQVztAs7CA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 86 +++++++++++++++++++++------------------------- 1 file changed, 40 insertions(+), 46 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..660d9f203d99 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -52,29 +52,24 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; + bool free_hdr = false; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -99,43 +94,37 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, if (hdr_off < sizeof(struct rds_header)) { /* see rds_tcp_write_space() */ + void *p; + set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); - if (ret < 0) - goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) + ret = -ENOMEM; + p = page_frag_memdup(NULL, + (void *)&rm->m_inc.i_hdr + hdr_off, + sizeof(rm->m_inc.i_hdr) - hdr_off, + GFP_KERNEL, ULONG_MAX); + if (!p) goto out; + bvec_set_virt(&bvec[ix], p, sizeof(rm->m_inc.i_hdr) - hdr_off); + free_hdr = true; + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +147,11 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + if (bvec) { + if (free_hdr) + put_page(bvec[0].bv_page); + kfree(bvec); + } return done; }