From patchwork Tue Aug 23 17:52:44 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 9296029 X-Patchwork-Delegate: Trond.Myklebust@netapp.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1FC36607D0 for ; Tue, 23 Aug 2016 17:52:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1432028A4D for ; Tue, 23 Aug 2016 17:52:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 08E7A28B2C; Tue, 23 Aug 2016 17:52:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 856BA28A4D for ; Tue, 23 Aug 2016 17:52:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754197AbcHWRws (ORCPT ); Tue, 23 Aug 2016 13:52:48 -0400 Received: from mail-it0-f67.google.com ([209.85.214.67]:34816 "EHLO mail-it0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753046AbcHWRwr (ORCPT ); Tue, 23 Aug 2016 13:52:47 -0400 Received: by mail-it0-f67.google.com with SMTP id f6so8769303ith.2; Tue, 23 Aug 2016 10:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=EfmLSk+4izLfwCBcDyqU9/qUVYxZn6xIkbP5q8K5bSQ=; b=RPbzntqkRQnIXUlLKJ/XyD5MEY8d3LPbuFmz78Lk5EjB7YtG210j/1GuN3r+p1yTk3 7Ol6iDQcGwyctOe65G1P52Hyvb8uSRgSgIN6YqezQjPdBe33C+Gh45/Ixa/+uI4FK/ge u8gyhLYTsiMLCav89640z+PAYcRWX7e6WXmYR/U/w+nNPmWOH5CIOTQgDxVfyT0eS+8r LH+wIEyhM7EfM7NhoyMLPRUFjjUbP0mf+N+5ZC5LbJnWE6BuwwFq9bbmilALmSxOjmC6 GfeGx5hz2FF1Rfn5jU+O4Bqevc3VoSPHUVxYR3fZH80IwjBAQPrJdsuc22iZjvZsq+hW OhIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=EfmLSk+4izLfwCBcDyqU9/qUVYxZn6xIkbP5q8K5bSQ=; b=C8Vz7sGDKz44hZferlagY3+IA0dQHryNsif+NUmYDE+Y3HN2e7DmPyEGZiBNPRo/KX Tziug/9iUZO51/jxKeWqBHVLECBPCPsRUjYxy8YbhfZWUvKQanL/17oNq8FWgTQo/5iA I2JwcQhMATY3S9ep0Yd4GsRTipICZ5HcUFHrkcpEtiPqHRFj7uqYt6ziKgnR9yF/t2va h+yghSl2bzjwWmrEnKLxm/u+R0ZJQh6lgNxfw4olI6P0B+587hW/kTp43qNWsiRrBjv0 dZC0Vg9WpAMznMGq+yLpo5wTHnhnD5PdhYid9fvnt+WSKqppCJwzB3xW2u7d9yYUL6qW V5yw== X-Gm-Message-State: AEkoouuSOvuNU8gUqlev0omISvRdPwatJYjgB2I2fXMThnUGy6ChCYLLOpQg7ijQR3iN9A== X-Received: by 10.36.55.133 with SMTP id r127mr27421591itr.15.1471974766461; Tue, 23 Aug 2016 10:52:46 -0700 (PDT) Received: from manet.1015granger.net ([2604:8800:100:81fc:ec4:7aff:fe6c:1dce]) by smtp.gmail.com with ESMTPSA id w12sm1763550iow.29.2016.08.23.10.52.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Aug 2016 10:52:46 -0700 (PDT) Subject: [PATCH v2 05/22] SUNRPC: Separate buffer pointers for RPC Call and Reply messages From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 23 Aug 2016 13:52:44 -0400 Message-ID: <20160823175244.13038.39619.stgit@manet.1015granger.net> In-Reply-To: <20160823174402.13038.84561.stgit@manet.1015granger.net> References: <20160823174402.13038.84561.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For xprtrdma, the RPC Call and Reply buffers are involved in real I/O operations. To start with, the DMA direction of the I/O for a Call is opposite that of a Reply. In the current arrangement, the Reply buffer address is on a four-byte alignment just past the call buffer. Would be friendlier on some platforms if that was at a DMA cache alignment instead. Because the current arrangement allocates a single memory region which contains both buffers, the RPC Reply buffer often contains a page boundary in it when the Call buffer is large enough (which is frequent). It would be a little nicer for setting up DMA operations (and possible registration of the Reply buffer) if the two buffers were separated, well-aligned, and contained as few page boundaries as possible. Now, I could just pad out the single memory region used for the pair of buffers. But frequently that would mean a lot of unused space to ensure the Reply buffer did not have a page boundary. Add a separate pointer to rpc_rqst that points right to the RPC Reply buffer. This makes no difference to xprtsock, but it will help xprtrdma in subsequent patches. Signed-off-by: Chuck Lever --- include/linux/sunrpc/xprt.h | 5 +++-- net/sunrpc/clnt.c | 2 +- net/sunrpc/sched.c | 1 + net/sunrpc/xprtrdma/transport.c | 1 + 4 files changed, 6 insertions(+), 3 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h index 72c2aeb..46f069e 100644 --- a/include/linux/sunrpc/xprt.h +++ b/include/linux/sunrpc/xprt.h @@ -84,8 +84,9 @@ struct rpc_rqst { struct list_head rq_list; void *rq_buffer; /* Call XDR encode buffer */ - size_t rq_callsize, - rq_rcvsize; + size_t rq_callsize; + void *rq_rbuffer; /* Reply XDR decode buffer */ + size_t rq_rcvsize; size_t rq_xmit_bytes_sent; /* total bytes sent */ size_t rq_reply_bytes_recvd; /* total reply bytes */ /* received */ diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index ab467c0..fd389c0 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -1768,7 +1768,7 @@ rpc_xdr_encode(struct rpc_task *task) req->rq_buffer, req->rq_callsize); xdr_buf_init(&req->rq_rcv_buf, - (char *)req->rq_buffer + req->rq_callsize, + req->rq_rbuffer, req->rq_rcvsize); p = rpc_encode_header(task); diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c index 6690ebc..5db68b3 100644 --- a/net/sunrpc/sched.c +++ b/net/sunrpc/sched.c @@ -891,6 +891,7 @@ int rpc_malloc(struct rpc_task *task) dprintk("RPC: %5u allocated buffer of size %zu at %p\n", task->tk_pid, size, buf); rqst->rq_buffer = buf->data; + rqst->rq_rbuffer = (char *)rqst->rq_buffer + rqst->rq_callsize; return 0; } EXPORT_SYMBOL_GPL(rpc_malloc); diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index ebf14ba..136caf3 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -524,6 +524,7 @@ out: dprintk("RPC: %s: size %zd, request 0x%p\n", __func__, size, req); req->rl_connect_cookie = 0; /* our reserved value */ rqst->rq_buffer = req->rl_sendbuf->rg_base; + rqst->rq_rbuffer = (char *)rqst->rq_buffer + rqst->rq_rcvsize; return 0; out_rdmabuf: