From patchwork Wed Mar 29 14:13:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13192549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ECCAC74A5B for ; Wed, 29 Mar 2023 14:16:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEE3E280002; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9EDA280001; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F193280002; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7D800280001 for ; Wed, 29 Mar 2023 10:16:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4263980D64 for ; Wed, 29 Mar 2023 14:16:03 +0000 (UTC) X-FDA: 80622135006.03.95FCFC8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 11049140025 for ; Wed, 29 Mar 2023 14:16:00 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LzZhK2yW; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680099361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=soF5gavmzneSvIIvHwhPEmiQq18BggR/9B/uQZFrN8w=; b=uSrCVy1/y86QAhYXMKO9aQYdMHIFKncdrKiqV0TOBG+dekWTpSfCQsPIE+AzMLo/+UHyPw SuA6lopRJQWLfpK7RMyisdLeE4cjJhapgTF2dfqxKTJeUIIpcMebraz/TuGdUu261fjN5H +Ei9w/d6Owyd0fRQdgsAaZceqKnIiyA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LzZhK2yW; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680099361; a=rsa-sha256; cv=none; b=Yy3lWFMTaQu3wSyDype7E7dDmBYAsnYnAOt46Vq7oSM9e1FXGaEIoF24o3wQm/mu+guaQu zdj1TNP6IQM96pY5ig7YW2wZX1R+pRsXxL+SkJy4MxBauuljTEo3NNMPb47OrmexoyiGCz Be3YBmGjGIGC0PYG/YGbBMqX7ssagyc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680099360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=soF5gavmzneSvIIvHwhPEmiQq18BggR/9B/uQZFrN8w=; b=LzZhK2yWfs3lJh32dj1p8CrZrcpoQnBtF9CzxGfvS2BUFOOYWaKraZcbqH7FA9BurO1ED1 tggVJyEJhAPhKCcM5rhn5H37Xl8LR4X/4sobQMsvEOpwwzMSoikd9H7EHnXj2622vV26jY OlzF+VRpDgIgxPvYWP/v2uluR9xZhBA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-56-ruOkGzEjPIqn8iT3b-ztNw-1; Wed, 29 Mar 2023 10:15:54 -0400 X-MC-Unique: ruOkGzEjPIqn8iT3b-ztNw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 99B85855300; Wed, 29 Mar 2023 14:15:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6CE9C2027040; Wed, 29 Mar 2023 14:15:47 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org Subject: [RFC PATCH v2 40/48] sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Wed, 29 Mar 2023 15:13:46 +0100 Message-Id: <20230329141354.516864-41-dhowells@redhat.com> In-Reply-To: <20230329141354.516864-1-dhowells@redhat.com> References: <20230329141354.516864-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 11049140025 X-Rspam-User: X-Stat-Signature: a4cf1su3afyfgd5m3ndd36jqwgxiqj1a X-HE-Tag: 1680099360-120205 X-HE-Meta: U2FsdGVkX181oTBmekla6Spf+6gJ2wPkElrCEBuwZLwIOfNcTQuP1QbvlUbVcunCkFkDbe0uw579KhLxHYddcJse8lRULeBJNC5CC00VOMojcJ09JdLBmA7bbSpRntgC0QLHgYDSu/582Kuvs29a+xFBRn73w0yIVlyYqBQO7X2WdpkUj4WcAUyDm0XIr7y8G2OwIGr40XKQ1uI+yWWMx5S5X7270pbR0MX+rbNwkW/vRTDJfs8cXMJkr1L9Z5S5SOcD/xtESXlsR+am90PrQgYuqLbiBWxfYTVc9jtMBQbp3IpKdH7LrJcnT64UwEZGg4bzm4lCf37uNQ4OOkah5C54OkL6X7Nvy1lEAi1ze/vqeZrh1WfG5ITmqMKgEg8tcK7rDjVIbLIbizhCu3OR1LLcpnIWsT7qFNgR3eEoENIqPqsDtzrbkcPJluAuJB/NaoMcE9jbNY2UaA45CWxnrd8EkKLXMgIxxt2btdi7R/OUmNGkyrPzPPM6NVOQzxlAlT6XjYzNTH+EZacnWwEKnbieRtLYs5Cg8F/DdcjY4GmTGdT+BOZWWw/4oT+lKJ4Q9cBkV7z6883wzISfaI+INJAlm5FLij76s6tznmSQDPJfMmKqj8DPh51sPpz+qZytq7Jhg5//GUBCgl6WTN8mu03sq/7Me3V2f456wJPk8/cWx09eAnlxvgsNZsmZdcA1cd2NzJusTw9tiUQx52E68itb0S3PxLZn0FrccU/F2bfjCrpdmh6ZMWHCU6AnC5Grt4/n2rRxVBLgAt9e6kPUc5S5EZm8ZiVRbsC2uyubvoPd5+nsvQfy/6KfuR3R9jJaII8xRq3eYuZ3WtvP+F3syKwi/9snvNH6zz0vMU2HnKECPWKR1mEacw0y4mpT8Cu2vCTREzaMDup9/3ffLGhs+7mWSq3geKh5lZKvv92FIfxrp0u4nf+w/YcER61KSKeUoyrnadbTCfeCkgmZalC Aq+PziCr Znzg3ZEgOVt6prsoK4VOw/ojQxTbURuNMJFah7d6e5bPY/ekyhJhNkzscZEW6/JVNd2Ygpm3ddZpqdcu+CIqLDUjF2FhFIT/gsGTKCOi1tuOVCpZR6Uyspm6oCn2WDc8N0Y+SEiO/vkIpG29k9Y7UJg7S3NYEuA90Vs/T8dHNykzNMtZDKRQB5AVpa3aeuvHfbqdWkTZdnEgHU573OJ5CtBtHhk2AvR4KLSy4UHfVoNHWC9OFuLVLKyqXVIIP6J9zMhMwh9rse93FSgrL2OPpuFDXOT0gpJfe6FMqQQWO36SzAQ3N5UAdJ1DfwrcsS6m6hid4IB+OzEjQcHDusBlTDRnzCNhS3YURSwzZIKkaszx0qw17sX0IS/zz4K1lXj8GfVL5bAkTV8+ZP7zDRV+4dlaK3lMB7+r9oGd6sp8zOUjSWxGKbA4X3h5X3aNOuN85OUMHtSTBzgAQpanIM2vjLGOujoPwTxPpqq2GClnJCM9r+KMqMM24Z4rTpCmeHwG9PX24p+LJc6KgsISzYnfhOTPhTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer are copied into page fragments so that they can be freed with put_page and attached to iterators of their own. An iterator-of-iterators is then created to bridge all three iterators (headers, data, trailer) and that is passed to sendmsg to pass the entire message in a single call. Signed-off-by: David Howells cc: Trond Myklebust cc: Anna Schumaker cc: Chuck Lever cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nfs@vger.kernel.org cc: netdev@vger.kernel.org Signed-off-by: David Howells Signed-off-by: David Howells Signed-off-by: David Howells Acked-by: Chuck Lever Tested-by: Daire Byrne --- include/linux/sunrpc/svc.h | 11 +++-- net/sunrpc/svcsock.c | 89 +++++++++++++++----------------------- 2 files changed, 40 insertions(+), 60 deletions(-) diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 877891536c2f..456ae554aa11 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -161,16 +161,15 @@ static inline bool svc_put_not_last(struct svc_serv *serv) extern u32 svc_max_payload(const struct svc_rqst *rqstp); /* - * RPC Requsts and replies are stored in one or more pages. + * RPC Requests and replies are stored in one or more pages. * We maintain an array of pages for each server thread. * Requests are copied into these pages as they arrive. Remaining * pages are available to write the reply into. * - * Pages are sent using ->sendpage so each server thread needs to - * allocate more to replace those used in sending. To help keep track - * of these pages we have a receive list where all pages initialy live, - * and a send list where pages are moved to when there are to be part - * of a reply. + * Pages are sent using ->sendmsg with MSG_SPLICE_PAGES so each server thread + * needs to allocate more to replace those used in sending. To help keep track + * of these pages we have a receive list where all pages initialy live, and a + * send list where pages are moved to when there are to be part of a reply. * * We use xdr_buf for holding responses as it fits well with NFS * read responses (that have a header, and some data pages, and possibly diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 03a4f5615086..f1cc53aad6e0 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -1060,16 +1060,8 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) return 0; /* record not complete */ } -static int svc_tcp_send_kvec(struct socket *sock, const struct kvec *vec, - int flags) -{ - return kernel_sendpage(sock, virt_to_page(vec->iov_base), - offset_in_page(vec->iov_base), - vec->iov_len, flags); -} - /* - * kernel_sendpage() is used exclusively to reduce the number of + * MSG_SPLICE_PAGES is used exclusively to reduce the number of * copy operations in this path. Therefore the caller must ensure * that the pages backing @xdr are unchanging. * @@ -1081,65 +1073,54 @@ static int svc_tcp_sendmsg(struct socket *sock, struct xdr_buf *xdr, { const struct kvec *head = xdr->head; const struct kvec *tail = xdr->tail; - struct kvec rm = { - .iov_base = &marker, - .iov_len = sizeof(marker), - }; + struct iov_iter iters[3]; + struct bio_vec head_bv, tail_bv; struct msghdr msg = { - .msg_flags = 0, + .msg_flags = MSG_SPLICE_PAGES, }; - int ret; + void *m, *t; + int ret, n = 2, size; *sentp = 0; ret = xdr_alloc_bvec(xdr, GFP_KERNEL); if (ret < 0) return ret; - ret = kernel_sendmsg(sock, &msg, &rm, 1, rm.iov_len); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != rm.iov_len) - return -EAGAIN; + m = page_frag_alloc(NULL, sizeof(marker) + head->iov_len + tail->iov_len, + GFP_KERNEL); + if (!m) + return -ENOMEM; - ret = svc_tcp_send_kvec(sock, head, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != head->iov_len) - goto out; + memcpy(m, &marker, sizeof(marker)); + if (head->iov_len) + memcpy(m + sizeof(marker), head->iov_base, head->iov_len); + bvec_set_virt(&head_bv, m, sizeof(marker) + head->iov_len); + iov_iter_bvec(&iters[0], ITER_SOURCE, &head_bv, 1, + sizeof(marker) + head->iov_len); - if (xdr->page_len) { - unsigned int offset, len, remaining; - struct bio_vec *bvec; - - bvec = xdr->bvec + (xdr->page_base >> PAGE_SHIFT); - offset = offset_in_page(xdr->page_base); - remaining = xdr->page_len; - while (remaining > 0) { - len = min(remaining, bvec->bv_len - offset); - ret = kernel_sendpage(sock, bvec->bv_page, - bvec->bv_offset + offset, - len, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != len) - goto out; - remaining -= len; - offset = 0; - bvec++; - } - } + iov_iter_bvec(&iters[1], ITER_SOURCE, xdr->bvec, + xdr_buf_pagecount(xdr), xdr->page_len); if (tail->iov_len) { - ret = svc_tcp_send_kvec(sock, tail, 0); - if (ret < 0) - return ret; - *sentp += ret; + t = page_frag_alloc(NULL, tail->iov_len, GFP_KERNEL); + if (!t) + return -ENOMEM; + memcpy(t, tail->iov_base, tail->iov_len); + bvec_set_virt(&tail_bv, t, tail->iov_len); + iov_iter_bvec(&iters[2], ITER_SOURCE, &tail_bv, 1, tail->iov_len); + n++; } -out: + size = sizeof(marker) + head->iov_len + xdr->page_len + tail->iov_len; + iov_iter_iterlist(&msg.msg_iter, ITER_SOURCE, iters, n, size); + + ret = sock_sendmsg(sock, &msg); + if (ret < 0) + return ret; + if (ret > 0) + *sentp = ret; + if (ret != size) + return -EAGAIN; return 0; }