From patchwork Wed Mar 22 15:07:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9639185 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4D87C602CB for ; Wed, 22 Mar 2017 15:07:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E5D02845C for ; Wed, 22 Mar 2017 15:07:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 322B02846F; Wed, 22 Mar 2017 15:07:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95F3B2845C for ; Wed, 22 Mar 2017 15:07:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935290AbdCVPHk (ORCPT ); Wed, 22 Mar 2017 11:07:40 -0400 Received: from bombadil.infradead.org ([65.50.211.133]:42450 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935279AbdCVPHj (ORCPT ); Wed, 22 Mar 2017 11:07:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Rw7tSXk6W3YC+WFgYyy8KZtDSGwtG7TnDNnqzSQVul0=; b=H0B5foHbQhLsgoqnXt9U2dDV1 87h+cZIb+N10ZHKGQLb2XTwh8p6Dmjv36LuuBnrmOIryGAoVriyhTkIcbmykMbJF9qRGJVrWjqwgA g6nGml/G8/8aLpEz7DDNnO93Fn3BURCL9QTkxSsYciH28lnYcCWkmo0HoffibDTWiePIvby59Mxxx czPhW9UOkPOBdTIQX9XY4pTUhAvlv6+11EnSjI2kC/uEth43fSlpSh9lm8pP9d7FE3RWC35SrnVXQ QiwOl8ZsgNcpbvs77EbgdD9V3xWedSbfI51uSi1ItC7wfB3L/+CaxvmbUWqj/9FbqmN16aU5BQSum +sw9cF4tA==; Received: from [69.46.234.238] (helo=sagi-Latitude-E7470.event.rightround.com) by bombadil.infradead.org with esmtpsa (Exim 4.87 #1 (Red Hat Linux)) id 1cqhrI-0004EB-3N; Wed, 22 Mar 2017 15:07:32 +0000 From: Sagi Grimberg To: "Nicholas A. Bellinger" , target-devel@vger.kernel.org Cc: Potnuri Bharat Teja Subject: [PATCH] iser-target: avoid posting a recv buffer twice Date: Wed, 22 Mar 2017 17:07:30 +0200 Message-Id: <1490195250-9816-1-git-send-email-sagi@grimberg.me> X-Mailer: git-send-email 2.7.4 Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We pre-allocate our send-queues and might overflow them in case we have multi work-request operations which tend to occur for large RDMA transfers over devices with limited allowed sg elements. When we get to a queue-full condition we might retry again later, so track our receive buffers so we don't repost them for a retry case. Reported-by: Potnuri Bharat Teja Tested-by: Potnuri Bharat Teja Signed-off-by: Sagi Grimberg --- Nic, this goes on top of your tri-patch series that takes care of queue full handling. it pairs with patch 3/3 "iser-target: Fix queue-full response handling" Thanks! drivers/infiniband/ulp/isert/ib_isert.c | 12 ++++++++++++ drivers/infiniband/ulp/isert/ib_isert.h | 3 ++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 9b33c0c97468..fcbed35e95a8 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -817,6 +817,7 @@ isert_post_recvm(struct isert_conn *isert_conn, u32 count) rx_wr->sg_list = &rx_desc->rx_sg; rx_wr->num_sge = 1; rx_wr->next = rx_wr + 1; + rx_desc->in_use = false; } rx_wr--; rx_wr->next = NULL; /* mark end of work requests list */ @@ -835,6 +836,15 @@ isert_post_recv(struct isert_conn *isert_conn, struct iser_rx_desc *rx_desc) struct ib_recv_wr *rx_wr_failed, rx_wr; int ret; + if (!rx_desc->in_use) { + /* + * if the descriptor is not in-use we already reposted it + * for recv, so just silently return + */ + return 0; + } + + rx_desc->in_use = false; rx_wr.wr_cqe = &rx_desc->rx_cqe; rx_wr.sg_list = &rx_desc->rx_sg; rx_wr.num_sge = 1; @@ -1397,6 +1407,8 @@ isert_recv_done(struct ib_cq *cq, struct ib_wc *wc) return; } + rx_desc->in_use = true; + ib_dma_sync_single_for_cpu(ib_dev, rx_desc->dma_addr, ISER_RX_PAYLOAD_SIZE, DMA_FROM_DEVICE); diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index c02ada57d7f5..87d994de8c91 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -60,7 +60,7 @@ #define ISER_RX_PAD_SIZE (ISCSI_DEF_MAX_RECV_SEG_LEN + 4096 - \ (ISER_RX_PAYLOAD_SIZE + sizeof(u64) + sizeof(struct ib_sge) + \ - sizeof(struct ib_cqe))) + sizeof(struct ib_cqe) + sizeof(bool))) #define ISCSI_ISER_SG_TABLESIZE 256 @@ -85,6 +85,7 @@ struct iser_rx_desc { u64 dma_addr; struct ib_sge rx_sg; struct ib_cqe rx_cqe; + bool in_use; char pad[ISER_RX_PAD_SIZE]; } __packed;