From patchwork Tue Sep 5 14:36:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Amrani, Ram" X-Patchwork-Id: 9939137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A88AE6038C for ; Tue, 5 Sep 2017 14:37:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99D8D288EE for ; Tue, 5 Sep 2017 14:37:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8E6FC28991; Tue, 5 Sep 2017 14:37:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C437289A1 for ; Tue, 5 Sep 2017 14:37:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751149AbdIEOhs (ORCPT ); Tue, 5 Sep 2017 10:37:48 -0400 Received: from mail-dm3nam03on0040.outbound.protection.outlook.com ([104.47.41.40]:15377 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750932AbdIEOhr (ORCPT ); Tue, 5 Sep 2017 10:37:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=CWDUdY0ekhvqKYr8Ll3u7rJgDbrP6lcCRyKieBdPMIU=; b=Oc0NLmXQx3pzLxwsukdAnrSD7vGoI5n6wpCJEJAoiMZxLzx7jI/GC98RUP3/jbLbbxcSHJrwy2Wk0dyOUD76w9O8j6FR/jKfAUCne3BHxWr9yX5qTtwIHij111h9RR8q6Ys99fjNPJgHwLTwA9g8YG9kiEbfXgp0iLvEFTqO9Z8= Received: from BY2PR07CA0088.namprd07.prod.outlook.com (10.166.107.41) by SN1PR0701MB2094.namprd07.prod.outlook.com (10.163.132.29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.13.10; Tue, 5 Sep 2017 14:37:45 +0000 Received: from BY2FFO11FD037.protection.gbl (2a01:111:f400:7c0c::191) by BY2PR07CA0088.outlook.office365.com (2a01:111:e400:7bff::41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.13.10 via Frontend Transport; Tue, 5 Sep 2017 14:37:44 +0000 Authentication-Results: spf=none (sender IP is 50.232.66.26) smtp.mailfrom=cavium.com; obsidianresearch.com; dkim=none (message not signed) header.d=none;obsidianresearch.com; dmarc=none action=none header.from=cavium.com; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) Received: from CAEXCH02.caveonetworks.com (50.232.66.26) by BY2FFO11FD037.mail.protection.outlook.com (10.1.14.222) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.1385.11 via Frontend Transport; Tue, 5 Sep 2017 14:37:44 +0000 Received: from lb-tlvb-ramrani.il.qlogic.org (10.185.6.119) by CAEXCH02.caveonetworks.com (10.17.4.29) with Microsoft SMTP Server id 14.2.347.0; Tue, 5 Sep 2017 07:37:42 -0700 From: Ram Amrani To: , , CC: , , Subject: [PATCH rdma-core RESEND] libqedr: Add iWARP support for user-space lib Date: Tue, 5 Sep 2017 17:36:54 +0300 Message-ID: <1504622214-9730-1-git-send-email-Ram.Amrani@cavium.com> X-Mailer: git-send-email 2.1.0 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:50.232.66.26; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(428002)(199003)(189002)(69596002)(478600001)(77096006)(2201001)(104016004)(50466002)(305945005)(48376002)(54906002)(47776003)(105586002)(106466001)(50986999)(5660300001)(8936002)(5003940100001)(626005)(101416001)(86362001)(189998001)(2906002)(356003)(8676002)(36756003)(50226002)(81166006)(81156014)(6666003)(72206003)(4326008); DIR:OUT; SFP:1101; SCL:1; SRVR:SN1PR0701MB2094; H:CAEXCH02.caveonetworks.com; FPR:; SPF:None; PTR:50-232-66-26-static.hfc.comcastbusiness.net; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD037; 1:U50tSKAq2upZOM0//OUDTK5vL/91HbD2rR0BLfGzQBUdzTU9qUeDhMfQoDsTNst9jH+xzTmu/DKxBQ+bbPKcX64d1QSvYPQ1gmRrskYldAUfVba67gMGORF8hTxN2gR+ X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d40ffa24-9ea8-44dd-7b92-08d4f46babe6 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:SN1PR0701MB2094; X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB2094; 3:fL+6VVN5eN9EpZaVrcAwAstWgnhtWV8Z8RdCimPxfQ6YB8judeDC/aSa87xSzA9dvCqmG8ZtNM8KHH0EjLB8F77t5HNGNd1HbckiLhQx2zmRd1D8KYk7YBguBtPg5oB1Q+qu298Q0rO13X+t+lQTss/ODfJ2VAKlNVdhrL9UvuJMNTswVX1q/eOKbPsbR77p4RnJxqmzhJFMb13ENzpLqhbmAvK02PlFN6dx5fe7XToXZzJvbphyVmJ2exz9Usai2VnfhFzeqHLIZRo2WFDxEXmCSuZlbsXqNh+6PCb68R8XYKHSBINqNXh7/EPbRS+3Q5VvsrNkDiBrWs/JvvTdHKrQMstohm067B4Nau5atmE=; 25:OPQOuUk8Yn+YxsCISlwMwuNaztBGl1d98eQn/JLQTrCBJVQLgYsaEZqJB+GMoV8wrK1++g2977e7no+M88ZmiqmEq3rvGqeG+gLAwSjH7iVDEiKiOKKk4gL+n/mhqBwPdb87qjc/CE9ttPSU6PrMBpOx/ZMGINHpBGxaDuKAIqLrcEVtRS32DwtmeWUb74bxzJEQp0KzSpLOIRUiGUxAnRXmdTPInS8p70zQ1Y2h+vZFawZ4b6CmA/KLqTMplpjFn1UXY18mf7+Z9dzv2QWBU29NcTKrGii9VbFbt7Lw5I6WIO+Jdz5OUakUJ3RrGyz76GEMkdbj318Itru5Y5hrWw== X-MS-TrafficTypeDiagnostic: SN1PR0701MB2094: X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB2094; 31:taCUvg3SXJ25YX0MQgST2o34KnU+vhYny2gcufjdhRJx8/JOvvSTHQyybFb2hh4kr9AcULUubPetvczHKGhpDovjcaHxGcfVrgAKxUfdg+BRlMnznWjiOrqD+hA2iAZiSV/SIHuOiAH5qgs+QuWEvlQUAttnXPYRTejANGt9ZgtBG7tDtKbNhVVeg/BRdu0Ilci3LzAoyJ+AyPUtV/xxkGr3GHPyChfR/Sc8oNQZBFo=; 20:YIgGzHkV051A2lKicd3r0869xvrjU+JDDBWRp74Aue/Efn2EzT6Br0K1qb5XjgxAUqy1h3CZmsyFMgUFK5ptF9ZcVeFCcX1sBewJxtl9pGWJv9qWOnrqec8TSSi02cstrsl4Tamj/3QxNIgeuCuzknb5jKzE4EUTjRPaN66TjvWmx9KHq+7S07tZznDpKnpz0Pfd3Dd687fkkyriS0qF+4MPR+4Q8wsUhZ0SvcYuev+qEjKOuRf5zk0ehd/J6uDNZuxPDhbcAEEGR+kg647mQ6zvHjcFcMpuzXFUOu+WUIssGs5DgOkTtCtZjIOJ79N3DJjSAkNTKOuT+hfckt8cev+IJFc2DaynDMKdRWgtHHV3OEIhBezmSoGPIGje9QYjzqHNIwboD71laDhpyXdFADMV8X5aoH75c8kRUGhI1JrqG1v1NiedgrU5J/QA2vFxhf4MWmFMBq5b6SY802h4DmpM8EsdjCN/WQTLgOgdyCWSYYDIq7ybvrOyO2lS8++Z X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(100000703101)(100105400095)(6041248)(20161123555025)(20161123560025)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:SN1PR0701MB2094; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:SN1PR0701MB2094; X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB2094; 4:hxarHle5IB1PZaYtI4H2BvrchB7siwKms3g6ZTQKj22SsrjkZ70HN/RVk4HZ6I6ib6QiZfTmy93C5iu2smI81mAFq2mrCfC4XYxV/1+uatKqGHLbOWnp+WxEJKBX1Fa8hjB9c0ZcIX/8guOp63NKHlSIe5dTHKyMx1FKswqXXbI3XLdaDsOEXnYKYDummopfwldne9eqUHjeXcespoJWuONMWe9Xhj7zF9xgrKWxU4aXGUzt08tt1bHh6S7THGTA X-Forefront-PRVS: 0421BF7135 X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB2094; 23:4B1hDdlmEbA9MSOho93SlCQ8z2mODo6H3uAy9Y4ZPLV9rKM1QJdEjNnVmHS997Ng97EXpt+IgQGcWieV2bFTScuI70x71PGVqSAIGdPIuHjIQn8I10e0F8ZXdfx+t3f0Vt67fxAO29vyHeY/k2Ihn5ONO9c6CaFBpT98L87SL+Ozxbyf39GT7NSSe6nUYzBWDui4HnUxEHMnIoG9d5aPTbt5JRUZzc+FFlq9R/+0p51PwwqekYQGwUvbKOOI2w2Y0GPXCQJWmun6brob+vJltvwJLuXme80uJJWihR8h4Y13n30J9FJqen5o2q76T8ZPVoQHDHbg9gLn9Hx+Ka2uubMBwOw6PW5zD3GNoOKS/kzchPzAVcbJ8Muvf5G57HQMNatwjAjqtRzF22XinwfxOGWfC2jg8wQKaMtsRTp+ZUNMQf37j4d2tj0fr73GHJXYn7eH2PxdIzsrScGleLuQLH1bN2yE0RLIMq7STLr/fg4YVhwpDSUiLlGjxRCtCka+GLjJLcPPCnL1xSohWFtDX9/kwVVFHQLe/YYfztwaCtfxPDda5jPsScTmSrkYIMiR1D9Mi4Pvd7aQLl87PZNkPhzs7oBezo5yR7cBElN9HiCRRraaAXFwmiTfnRvu3mB5txwxxlQU9lTk0VdFobamGnZbvDbf+68YMI61T6wfbBQASR0Q4MhOc4JClvtEGv3lvxHpuJPe5J3BbhBlQVXnnTQKmWoT5afeZFwFIgM0b6b6774iXR0hGuNAD/uzAF8Q63MZTZv6/Hf/768nv+rARUO506psHuuWC7o6rViTqZdSr5j1Onb9qjbFCdJSug9bL0liAx4CcK1KSPoBVxVeuFgTiYomMrcFSXcOr/niM7Y= X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB2094; 6:/1R+VFRgtBI+CGLfZ7O4KjPHrTMs4nVlRwQa+6c3XT2EHC2Uo7dXGdDo9jaTyXh22Lv90u6OdazYP+npt/UBnIMT6l/Rm3mvaGYjJKI+hN2/ManF94ssQoH1sG2KcBOEMcHZiFTo8E3NUyqftufvCqIiOW8grAVVBWbw74A8DGxhbBuQ/E/PxnGeoEuuKwlr/3qMj+kdYsZJg0iLZSiU33wFlloLPqSXccMbUkL3dzH6C73TXYyc63zI/XUly9MnJMSvMz6yIhp/7MD2SLo/nJJ1YLyyEE8G5t2Mt42gIDx0rM9g5dhOES0/HWZWD0PJcejp3OmFurXTFcdMz5h5mw==; 5:rAn+f1ozufywRCkOhkqBTC/4aptHF09PhltWQqMn7ky84nA9ATfTJ3UV0SwTNNlV9kwl/jZ86umB2XknHQkjzTcaP9/01jRLnKoFSOphR4g795D7bgj6Uthq4XDNotWbPBjoqNIeIlr0ga+oz3i8J9YE1fxUC3Hr+YJ3Iwer/sE=; 24:KTyygq7CpDUJPBjCVQypuOwBUvBfxdcfTChqjUb87BFegsjwOgwWh+fxpEM6MLZAGacuPMwS4DzHHq9657eMEgeugTxaE8+cfwSzqzSUO/I=; 7:CpfgPAJdnjPoQDWeokNiPg7uud0uNZiTollwn9gFFdIRIJ7xX48i7E2B0JMttEQ3egdQJd+oftnnauORFeD/6AwHozjEtd043/JPudoFa7z3LDRhVPGkN7t2DAxGH5JdQ8TI+AfuBEjnCz551KqBLgq2L57w/Y6QKLLGzjb+7om9Ob13H2SVL+Al8eyhEZmjoeWNSQF9S6M5C625wLl/fYFOJ+5ecxUFYTCD9ap/j+U= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2017 14:37:44.6741 (UTC) X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=711e4ccf-2e9b-4bcf-a551-4094005b6194; Ip=[50.232.66.26]; Helo=[CAEXCH02.caveonetworks.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR0701MB2094 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Michal Kalderon Introduce IS_IWARP / IS_ROCE macros for libqedr as there are no global macros for this in rdma-core. There is a second doorbell required for RQ. QP states in user lib are not updated, as they are updated implicitly by the driver and not driven by modify_qp as they are in ib / roce. For this reason, decisions based on states are encapsulated in IS_ROCE. Signed-off-by: Michal Kalderon --- providers/qedr/qelr.h | 2 ++ providers/qedr/qelr_verbs.c | 34 +++++++++++++++++++++++++++++----- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/providers/qedr/qelr.h b/providers/qedr/qelr.h index 0dd6835..3c663e7 100644 --- a/providers/qedr/qelr.h +++ b/providers/qedr/qelr.h @@ -192,6 +192,8 @@ struct qelr_qp_hwq_info { void *db; /* Doorbell address */ void *edpm_db; union db_prod32 db_data; /* Doorbell data */ + void *iwarp_db2; /* iWARP RQ Doorbell address */ + union db_prod32 iwarp_db2_data; /* iWARP RQ Doorbell data */ uint16_t icid; }; diff --git a/providers/qedr/qelr_verbs.c b/providers/qedr/qelr_verbs.c index 676b18c..a087499 100644 --- a/providers/qedr/qelr_verbs.c +++ b/providers/qedr/qelr_verbs.c @@ -59,6 +59,9 @@ #define QELR_RQE_ELEMENT_SIZE (sizeof(struct rdma_rq_sge)) #define QELR_CQE_SIZE (sizeof(union rdma_cqe)) +#define IS_IWARP(_dev) (_dev->node_type == IBV_NODE_RNIC) +#define IS_ROCE(_dev) (_dev->node_type == IBV_NODE_CA) + static void qelr_inc_sw_cons_u16(struct qelr_qp_hwq_info *info) { info->cons = (info->cons + 1) % info->max_wr; @@ -434,6 +437,9 @@ static inline int qelr_configure_qp_rq(struct qelr_devctx *cxt, qp->rq.icid = resp->rq_icid; qp->rq.db_data.data.icid = htole16(resp->rq_icid); qp->rq.db = cxt->db_addr + resp->rq_db_offset; + qp->rq.iwarp_db2 = cxt->db_addr + resp->rq_db2_offset; + qp->rq.iwarp_db2_data.data.icid = qp->rq.icid; + qp->rq.iwarp_db2_data.data.value = DQ_TCM_IWARP_POST_RQ_CF_CMD; qp->rq.prod = 0; /* shadow RQ */ @@ -648,6 +654,12 @@ static int qelr_update_qp_state(struct qelr_qp *qp, int status = 0; enum qelr_qp_state new_state; + /* iWARP states are updated implicitely by driver and don't have a + * real purpose in user-lib. + */ + if (IS_IWARP(qp->ibv_qp.context->device)) + return 0; + new_state = get_qelr_qp_state(new_ib_state); pthread_spin_lock(&qp->q_lock); @@ -677,9 +689,11 @@ static int qelr_update_qp_state(struct qelr_qp *qp, /* Update doorbell (in case post_recv was done before * move to RTR) */ - mmio_wc_start(); - writel(qp->rq.db_data.raw, qp->rq.db); - mmio_flush_writes(); + if (IS_ROCE(qp->ibv_qp.context->device)) { + mmio_wc_start(); + writel(qp->rq.db_data.raw, qp->rq.db); + mmio_flush_writes(); + } break; case QELR_QPS_ERR: break; @@ -870,6 +884,10 @@ static inline void qelr_init_edpm_info(struct qelr_devctx *cxt, { edpm->is_edpm = 0; + /* Currently dpm is not supported for iWARP */ + if (IS_IWARP(cxt->ibv_ctx.device)) + return; + if (qelr_chain_is_full(&qp->sq.chain) && wr->send_flags & IBV_SEND_INLINE && !qp->edpm_disabled) { memset(edpm, 0, sizeof(*edpm)); @@ -1405,7 +1423,8 @@ int qelr_post_send(struct ibv_qp *ib_qp, struct ibv_send_wr *wr, pthread_spin_lock(&qp->q_lock); - if ((qp->state != QELR_QPS_RTS && qp->state != QELR_QPS_ERR && + if (IS_ROCE(ib_qp->context->device) && + (qp->state != QELR_QPS_RTS && qp->state != QELR_QPS_ERR && qp->state != QELR_QPS_SQD)) { pthread_spin_unlock(&qp->q_lock); *bad_wr = wr; @@ -1445,10 +1464,11 @@ int qelr_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, struct qelr_qp *qp = get_qelr_qp(ibqp); struct qelr_devctx *cxt = get_qelr_ctx(ibqp->context); uint16_t db_val; + uint8_t iwarp = IS_IWARP(ibqp->context->device); pthread_spin_lock(&qp->q_lock); - if (qp->state == QELR_QPS_RST) { + if (!iwarp && qp->state == QELR_QPS_RST) { pthread_spin_unlock(&qp->q_lock); *bad_wr = wr; return -EINVAL; @@ -1517,6 +1537,10 @@ int qelr_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, writel(qp->rq.db_data.raw, qp->rq.db); mmio_flush_writes(); + if (iwarp) { + writel(qp->rq.iwarp_db2_data.raw, qp->rq.iwarp_db2); + mmio_flush_writes(); + } wr = wr->next; }