From patchwork Wed Aug 2 20:10:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Long Li X-Patchwork-Id: 9877619 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 82A4460360 for ; Wed, 2 Aug 2017 20:21:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 748A8286BB for ; Wed, 2 Aug 2017 20:21:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 69BF3287A8; Wed, 2 Aug 2017 20:21:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0C344287F8 for ; Wed, 2 Aug 2017 20:21:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753421AbdHBUSE (ORCPT ); Wed, 2 Aug 2017 16:18:04 -0400 Received: from a2nlsmtp01-04.prod.iad2.secureserver.net ([198.71.225.38]:36940 "EHLO a2nlsmtp01-04.prod.iad2.secureserver.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752975AbdHBUMZ (ORCPT ); Wed, 2 Aug 2017 16:12:25 -0400 Received: from linuxonhyperv.com ([107.180.71.197]) by : HOSTING RELAY : with SMTP id czzHd52yB3vhKczzHdxf19; Wed, 02 Aug 2017 13:11:23 -0700 x-originating-ip: 107.180.71.197 Received: from longli by linuxonhyperv.com with local (Exim 4.89) (envelope-from ) id 1dczzH-0005H9-P6; Wed, 02 Aug 2017 13:11:23 -0700 From: Long Li To: Steve French , linux-cifs@vger.kernel.org, samba-technical@lists.samba.org, linux-kernel@vger.kernel.org Cc: Long Li Subject: [[PATCH v1] 15/37] [CIFS] SMBD: Post a SMBD data transfer message with data payload Date: Wed, 2 Aug 2017 13:10:26 -0700 Message-Id: <1501704648-20159-16-git-send-email-longli@exchange.microsoft.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1501704648-20159-1-git-send-email-longli@exchange.microsoft.com> References: <1501704648-20159-1-git-send-email-longli@exchange.microsoft.com> X-CMAE-Envelope: MS4wfMFnhETofGkfwt3aGqkIkuFX6+ugSsd9BgXh6JevBQRUlop58lDaMj6p1Sz7IV8fOrFAyJI4+RuRcjrdbsibFeoVRiLbg8u4zi3+nldHD7wSE2xYkyJV C6ACn3GLEdqZRkTaONvXQxqn6kuYT8c36n9/IPK2a25VeZHofYbiVg/eXnIx0ETM43QrWGqeFhEXwtQeON5wBFFo6Q1O1dz14Jrs9VgCHZ2OKJOrEC74E8DX QOtVu4GtkvzBv93+vRI4W00X6UFd9rMnpbj6XZyagnOyZKk7XETyZnebitJYkIAdIS6cS+3fJZnkPcRY+jtpyBaHhPB4hSEAD4/1KK/UgqI= Sender: linux-cifs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-cifs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Long Li Similar to sending transfer message with page payload, this function creates a SMBD data packet and send it over to RDMA, from iov passed from upper layer. Signed-off-by: Long Li --- fs/cifs/cifsrdma.c | 119 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/fs/cifs/cifsrdma.c b/fs/cifs/cifsrdma.c index b3ec109..989cad8 100644 --- a/fs/cifs/cifsrdma.c +++ b/fs/cifs/cifsrdma.c @@ -66,6 +66,9 @@ static int cifs_rdma_post_recv( struct cifs_rdma_info *info, struct cifs_rdma_response *response); +static int cifs_rdma_post_send_data( + struct cifs_rdma_info *info, + struct kvec *iov, int n_vec, int remaining_data_length); static int cifs_rdma_post_send_page(struct cifs_rdma_info *info, struct page *page, unsigned long offset, size_t size, int remaining_data_length); @@ -671,6 +674,122 @@ static int cifs_rdma_post_send_page(struct cifs_rdma_info *info, struct page *pa } /* + * Send a data buffer + * iov: the iov array describing the data buffers + * n_vec: number of iov array + * remaining_data_length: remaining data to send in this payload + */ +static int cifs_rdma_post_send_data( + struct cifs_rdma_info *info, struct kvec *iov, int n_vec, + int remaining_data_length) +{ + struct cifs_rdma_request *request; + struct smbd_data_transfer *packet; + struct ib_send_wr send_wr, *send_wr_fail; + int rc = -ENOMEM, i; + u32 data_length; + + request = mempool_alloc(info->request_mempool, GFP_KERNEL); + if (!request) + return rc; + + request->info = info; + + wait_event(info->wait_send_queue, atomic_read(&info->send_credits) > 0); + atomic_dec(&info->send_credits); + + packet = (struct smbd_data_transfer *) request->packet; + packet->credits_requested = cpu_to_le16(info->send_credit_target); + packet->flags = cpu_to_le16(0); + packet->reserved = cpu_to_le16(0); + + packet->data_offset = cpu_to_le32(24); + + data_length = 0; + for (i=0; idata_length = cpu_to_le32(data_length); + + packet->remaining_data_length = cpu_to_le32(remaining_data_length); + packet->padding = cpu_to_le32(0); + + log_rdma_send("credits_requested=%d credits_granted=%d data_offset=%d " + "data_length=%d remaining_data_length=%d\n", + le16_to_cpu(packet->credits_requested), + le16_to_cpu(packet->credits_granted), + le32_to_cpu(packet->data_offset), + le32_to_cpu(packet->data_length), + le32_to_cpu(packet->remaining_data_length)); + + request->sge = kzalloc(sizeof(struct ib_sge)*(n_vec+1), GFP_KERNEL); + if (!request->sge) + goto allocate_sge_failed; + + request->num_sge = n_vec+1; + + request->sge[0].addr = ib_dma_map_single( + info->id->device, (void *)packet, + sizeof(*packet), DMA_BIDIRECTIONAL); + if(ib_dma_mapping_error(info->id->device, request->sge[0].addr)) { + rc = -EIO; + goto dma_mapping_failure; + } + request->sge[0].length = sizeof(*packet); + request->sge[0].lkey = info->pd->local_dma_lkey; + ib_dma_sync_single_for_device(info->id->device, request->sge[0].addr, + request->sge[0].length, DMA_TO_DEVICE); + + for (i=0; isge[i+1].addr = ib_dma_map_single(info->id->device, iov[i].iov_base, + iov[i].iov_len, DMA_BIDIRECTIONAL); + if(ib_dma_mapping_error(info->id->device, request->sge[i+1].addr)) { + rc = -EIO; + goto dma_mapping_failure; + } + request->sge[i+1].length = iov[i].iov_len; + request->sge[i+1].lkey = info->pd->local_dma_lkey; + ib_dma_sync_single_for_device(info->id->device, request->sge[i+i].addr, + request->sge[i+i].length, DMA_TO_DEVICE); + } + + log_rdma_send("rdma_request sge[0] addr=%llu legnth=%u lkey=%u\n", + request->sge[0].addr, request->sge[0].length, request->sge[0].lkey); + for (i=0; isge[i+1].addr, + request->sge[i+1].length, request->sge[i+1].lkey); + + request->cqe.done = send_done; + + send_wr.next = NULL; + send_wr.wr_cqe = &request->cqe; + send_wr.sg_list = request->sge; + send_wr.num_sge = request->num_sge; + send_wr.opcode = IB_WR_SEND; + send_wr.send_flags = IB_SEND_SIGNALED; + + rc = ib_post_send(info->id->qp, &send_wr, &send_wr_fail); + if (!rc) + return 0; + + // post send failed + log_rdma_send("ib_post_send failed rc=%d\n", rc); + +dma_mapping_failure: + for (i=0; isge[i].addr) + ib_dma_unmap_single(info->id->device, + request->sge[i].addr, + request->sge[i].length, + DMA_TO_DEVICE); + kfree(request->sge); + +allocate_sge_failed: + mempool_free(request, info->request_mempool); + return rc; +} + +/* * Post a receive request to the transport * The remote peer can only send data when a receive is posted * The interaction is controlled by send/recieve credit system