From patchwork Wed Sep 6 09:25:59 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Amrani, Ram" X-Patchwork-Id: 9940205 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A3EAE602CC for ; Wed, 6 Sep 2017 09:26:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96A5228B61 for ; Wed, 6 Sep 2017 09:26:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B95D28B63; Wed, 6 Sep 2017 09:26:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D2D428B61 for ; Wed, 6 Sep 2017 09:26:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752350AbdIFJ0O (ORCPT ); Wed, 6 Sep 2017 05:26:14 -0400 Received: from mail-co1nam03on0073.outbound.protection.outlook.com ([104.47.40.73]:25952 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752226AbdIFJ0N (ORCPT ); Wed, 6 Sep 2017 05:26:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=XY0CID3UJHSrr0Fp1S1Q3psvglt6cK415BXg7DQnm0s=; b=LJ95xN/y4lsd9Oe5iJFbndle4FVoCpMNQVfSCIGaZSYyhXF47sSkWbYzGUAYXGvCjmNNywHnX3fxZ/nn6sm7tRDU7qs0YMuQnGfPAo0BOucUpM+KsRpbx3W795VB0WHWdxzLQdo255FqZHI7jYMZlVmkqM+NQRGFrowIxtmQf9o= Received: from CY1PR07CA0017.namprd07.prod.outlook.com (2a01:111:e400:c60a::27) by DM2PR07MB494.namprd07.prod.outlook.com (2a01:111:e400:2446::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.13.10; Wed, 6 Sep 2017 09:26:10 +0000 Received: from BY2FFO11FD024.protection.gbl (2a01:111:f400:7c0c::196) by CY1PR07CA0017.outlook.office365.com (2a01:111:e400:c60a::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.35.12 via Frontend Transport; Wed, 6 Sep 2017 09:26:09 +0000 Authentication-Results: spf=none (sender IP is 50.232.66.26) smtp.mailfrom=cavium.com; obsidianresearch.com; dkim=none (message not signed) header.d=none;obsidianresearch.com; dmarc=none action=none header.from=cavium.com; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) Received: from CAEXCH02.caveonetworks.com (50.232.66.26) by BY2FFO11FD024.mail.protection.outlook.com (10.1.15.213) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.1385.11 via Frontend Transport; Wed, 6 Sep 2017 09:26:07 +0000 Received: from lb-tlvb-ramrani.il.qlogic.org (10.185.6.119) by CAEXCH02.caveonetworks.com (10.17.4.29) with Microsoft SMTP Server id 14.2.347.0; Wed, 6 Sep 2017 02:26:03 -0700 From: Ram Amrani To: , , CC: , , , Ram Amrani Subject: [PATCH rdma-core v2] libqedr: Add iWARP support for user-space lib Date: Wed, 6 Sep 2017 12:25:59 +0300 Message-ID: <1504689959-32494-1-git-send-email-Ram.Amrani@cavium.com> X-Mailer: git-send-email 2.1.0 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:50.232.66.26; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(428002)(199003)(189002)(8936002)(47776003)(54906002)(189998001)(2201001)(72206003)(77096006)(50466002)(478600001)(6666003)(107886003)(50986999)(5660300001)(4326008)(50226002)(101416001)(5003940100001)(48376002)(104016004)(36756003)(2906002)(8676002)(106466001)(86362001)(626005)(69596002)(305945005)(105586002)(81166006)(356003)(81156014); DIR:OUT; SFP:1101; SCL:1; SRVR:DM2PR07MB494; H:CAEXCH02.caveonetworks.com; FPR:; SPF:None; PTR:50-232-66-26-static.hfc.comcastbusiness.net; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD024; 1:Pe8y41aiburFJIX+bT5CYBzSulFOX1WXr5ZvckYfL1uKA8qwMs7A2ZBgMMxVIxUJU+tOMddVh4YsRxH1Mc2UBjdkk3Cp5api2s7YdH0Nasl/6mDPei9k5Dnnatd+pafp X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7246db86-4de5-4ba8-8f5d-08d4f5094eb1 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:DM2PR07MB494; X-Microsoft-Exchange-Diagnostics: 1; DM2PR07MB494; 3:XcIKw/vkKa2HmEu3Ef59nCivVHBmUlrwQ2OnMLk5IkuKK7rXUDEXS4zVaROXpniiZOb7NlvEoCwtly+dBvkKzrtXSafTdK9OWabxyHdPrb5zyPwqy1eWk577f+hDk75ZdGrjrfPtxkXPm9vRdbiwO1jyzoAPcIZ5jmJnYsdZfr2XLAp8gcjl408Po+XzzX0tSmh3tYVOqy8hDXNM1/SH0EQgqbF9kgLr7VwNCrkQMkLfULxHoK0Y4SSy8FW1p+IGDZmC5RDX0/HoBSz6lkP0DUzI6i3dfqse7LxMZTXgquBo51aq2YnwWkYwte07Anzf1s0T85JG0+48MKMOomektA==; 25:A2TnlkQM1ZpWDlyHtYlUEcW3N6r1nGT0+2araj6IVEnexThU6Ex2vnwaDOookL8fGkt+Q2Kkw2tsVBLBJoWQncszzSYkeqxmEkKLNKwo3sHNgmWYx4kskvt/pmE7H/Sv1Qg5uTimRJSCVu1qIDMzfyEQKhMWxc9rHWVvhHBTrwJjj8QBj0wZFIYXoZPMgUPZbEFfX7zU9/L6mkek2x5E6YPmoF67OnWDqxT8ncgbC4KLGcx/vBecZGbvXljNCSCEC9xTtLSsdKV59XAwdRwB4IZ9p8Ujyap99rGAmMlRckDz/mSZBo+CTowJ+4jyeYqr7zFLqkHH8f1PCJck55Eb9A==; 31:ec2REiBNTjQ7KOXYmqeUV+YEAyztCtvtr/9P+ajOCR2WcNL6os1s6p7L2Yljxyf/v4ni+X6EBYERiS644cSOOz/6E9G8IqK+x74NvKQ4pk1V0YuFv13AGKDoMh9DpV3P7D+pi1y4Sm6/x11A0z2W2r9elPu6W1SmwQGRMJTwzH4+MwFP/ZMymdYDyfQzhIKtfi1viO2rd/jm5fgrcCBjdcHFTWgDV0dFE3Si4Br1l/g= X-MS-TrafficTypeDiagnostic: DM2PR07MB494: X-Microsoft-Exchange-Diagnostics: 1; DM2PR07MB494; 20:voj5oIuJli87LlZ4fF1NuNOcAzMm3i3Qn0utSbwI+8dnTjtBErfJTwQFtf6ls27WA2vc6kXdKeLs9kUdlSUw+FGkMwQAySXLl2wVePseYAlISerVEJY19c+QzN2X+u3VdqFXebq+6yGPXe7yh9dsIeT0LDYb9sXCTnTZLyf+RBNKOlMWM58cs/Y4erpotkR6zAzE9FhAOzACBPpgKihJ2I+yhiGjPmhcWdf7gISQHnbj4/+VkXiPhAiyxW28SB4avI5T98FXjUwM6YvQt2I72256QKLlnC3qiSdVVmYptVIJl0gLcjIA0HTeJSgOqWpYISAu3k5yFz5+ZBiQ/Hnbjrc9mkPCRMEpGgHZVqRc+rmVvohgqES2vNeNufprVSIFym7thCrZuVH0L5XDkyKEPI82EyMUftexDPhtQYeFGH5VHxmKpd0cfoOdz8mIKPR0AAMWo3/+U0McTtfZBDHonJaQPbbL+FPPOEtifb8ljVO9T4Gn0n3BOXGNcpq7D2W4; 4:edkMosLveDtmuMtFqucfnYBJ4uTmnAyKeHcX2J04jd4Bdm19W1bYXrzsU/WT/uX/zkeSvpWFMg1XXJPwBZmomor25Ev4PTnLXsBFECqGjsc7yVl9xWg9TKURTYm7YhspWUFHXLh7CGJwnMx6zwwcsewp+F2AWEn+LhjzwdfC0YSyNur7w+7PxV9II+g/we54zIsqHk55LmjdMBcrbEe3sUjpmD0N3DElifuPZOlDYowZCC8qbAq9sfH4+qGA9/FU X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(3002001)(10201501046)(93006095)(93001095)(6041248)(20161123560025)(20161123564025)(20161123562025)(20161123555025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM2PR07MB494; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM2PR07MB494; X-Forefront-PRVS: 0422860ED4 X-Microsoft-Exchange-Diagnostics: 1; DM2PR07MB494; 23:eWesKRWPuBGG5qa+3Hu3T0w9dRA09lP5Ufb/TqU01/20NtO/mORoc6hupLWGd4/gTHho9FgevpIlthse7YbDwHNxOODnJunapb7ZiO3wjFiioGsnb95aqTAvqONzdRNFv7Yvvy197ifXqx21ogq+L9ulPGmQfMHSuUz+i8lkNrja7gd0PWbqGTX7hNOGux4LnuNg9IhqZHH/Zp9e09Vf3yEBVnhgQ7nbjOpJVLglhdJu7Yhhcgx3f1E+zS4iGCKOHXkoC3FBn81BeyfFIVQreU4lbWBScYzXybBBvbM2/mtgVG1zaWsPQEq/F73RBIH74Q82MKVbVciOxly+fyQE+x9p04ie5pCGO5ohK2Spt+y3rEKltOfLK/rCjInViqczRV6Ayaak4CcWDBxG3fY0Wgt8BKfHDXMse9Helxv6Q6mCZFzMLg2jeOxtXCyAjGV3ucviARr6fEO50P1t9XWbJLqPWci2G2yQsFVXj4q89c4HKnJmfx6ilQzO6sB7rA4D63HhZmrf5Nc5ScF+kJ9bNQu5mbSBIH9QUBgj933nMriSBwLmvcLLl1Sib378802S26a8MFmnOGFUY+DCaQG6HfgchmrcOViS77TBPSffthXJ7QDMGF6pHgkg5tw3hfk9EZ+aNivTpGuUhwbZ85EgsWCMiEg/EGtRYVjn8fXvb691nSuHKby9meMkUO42EgWhHvF1ZDC/aMjKb+xAnM0J9NJaZVTCS+sq4xxZzyZLYYzMztL/j+UtiKeMNiNR1zbopDitWE+3bSuuLWv1+vofOK/IluqtmyLvd0lgk9LbI0IlWyQO832lLdtl0PlhjP0l5m3aVFTk18JWfxdNvC57AOEJrosfwFZgI2O0r3LCptRJPPvMSV4KncCFMJ03hQwK X-Microsoft-Exchange-Diagnostics: 1; DM2PR07MB494; 6:VB0a0buXqlp76UNwzKr14/gyFwhQrVMEqIpiD73ONqMBPskKxT0R8BcB6WeiBqAyFlvu2u4M/t4qAAfzmCcEubJccxuaN1eQCjYcXINh/lpQsDHASjdlo9RWbLomYBBOQwA29cIUCVZHcEPDttbpHB5OFzQAadMpdYUnmafHZyenk9rpwjNREL96CbjztOvQo4sucsy0F+HBFUDKfxdkdwncZUNCePBc7FiVvlnQokRyLXgRRZM68ipScyE+H2phLyCsqmd0+OlNDj49+IbZMWj9C33A7tAxzaBVIVgkDwbJDT3waN2cJBudBApdD2/dgcowqCngFLHHhHQ9521Mag==; 5:HkF9l1h1kqtapORIAnfjiMIbrAYXhet7793W24UIAWhu5nz4Jb+jvekO1dinnBDhgOCj5VYe8MQZIbkyMoF/5xu7nHRVbxJrcusWJ8vi0T+7FDS/4yPE7I8IvcOixdlnR1YpctTs/DWlwv+Wl4jWfQ==; 24:v5jrEyDP3ELtBQl8G5DDU/vAA9XhjdSJzRCDGTSeLeJMgx+bHpT0fD2U/bO06CJGqpvSyozCF5CPk7PDqxBJevFJq39NuEqHNS2dQSF2Fl0=; 7:sQ1WJL1GIwT1jnNMkJnekR/0Wbnqd6G16ZzUG6Z+NH7vw5xcjmRrRW4w6DgdstGq8ofh2IskdS3LAA9ALDjYahVYi1yJ/zmnT7hFeP2E//pgvDAF94jT8cxAQmPnhA3Oknl1WJ1Rw0eYW2WJ0TdV7IzCEVaWH/PP/pojkwCKgWgi2ODtHN1+3hzRU20iCMGEb+g6GoAs+Natic6sbx3d3rKFuyBINvELxrruIbO7tEk= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Sep 2017 09:26:07.5344 (UTC) X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=711e4ccf-2e9b-4bcf-a551-4094005b6194; Ip=[50.232.66.26]; Helo=[CAEXCH02.caveonetworks.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR07MB494 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Michal Kalderon Introduce IS_IWARP / IS_ROCE macros for libqedr as there are no global macros for this in rdma-core. There is a second doorbell required for RQ. QP states in user lib are not updated, as they are updated implicitly by the driver and not driven by modify_qp as they are in ib / roce. For this reason, decisions based on states are encapsulated in IS_ROCE. Signed-off-by: Michal Kalderon Signed-off-by: Ram Amrani --- providers/qedr/qelr.h | 2 ++ providers/qedr/qelr_verbs.c | 34 +++++++++++++++++++++++++++++----- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/providers/qedr/qelr.h b/providers/qedr/qelr.h index 05b1640..1a5920c 100644 --- a/providers/qedr/qelr.h +++ b/providers/qedr/qelr.h @@ -192,6 +192,8 @@ struct qelr_qp_hwq_info { void *db; /* Doorbell address */ void *edpm_db; union db_prod32 db_data; /* Doorbell data */ + void *iwarp_db2; + union db_prod32 iwarp_db2_data; uint16_t icid; }; diff --git a/providers/qedr/qelr_verbs.c b/providers/qedr/qelr_verbs.c index fb52f9d..4a74c50 100644 --- a/providers/qedr/qelr_verbs.c +++ b/providers/qedr/qelr_verbs.c @@ -60,6 +60,9 @@ #define QELR_RQE_ELEMENT_SIZE (sizeof(struct rdma_rq_sge)) #define QELR_CQE_SIZE (sizeof(union rdma_cqe)) +#define IS_IWARP(_dev) (_dev->node_type == IBV_NODE_RNIC) +#define IS_ROCE(_dev) (_dev->node_type == IBV_NODE_CA) + static void qelr_inc_sw_cons_u16(struct qelr_qp_hwq_info *info) { info->cons = (info->cons + 1) % info->max_wr; @@ -435,6 +438,9 @@ static inline int qelr_configure_qp_rq(struct qelr_devctx *cxt, qp->rq.icid = resp->rq_icid; qp->rq.db_data.data.icid = htole16(resp->rq_icid); qp->rq.db = cxt->db_addr + resp->rq_db_offset; + qp->rq.iwarp_db2 = cxt->db_addr + resp->rq_db2_offset; + qp->rq.iwarp_db2_data.data.icid = htole16(qp->rq.icid); + qp->rq.iwarp_db2_data.data.value = htole16(DQ_TCM_IWARP_POST_RQ_CF_CMD); qp->rq.prod = 0; /* shadow RQ */ @@ -649,6 +655,12 @@ static int qelr_update_qp_state(struct qelr_qp *qp, int status = 0; enum qelr_qp_state new_state; + /* iWARP states are updated implicitely by driver and don't have a + * real purpose in user-lib. + */ + if (IS_IWARP(qp->ibv_qp.context->device)) + return 0; + new_state = get_qelr_qp_state(new_ib_state); pthread_spin_lock(&qp->q_lock); @@ -678,9 +690,11 @@ static int qelr_update_qp_state(struct qelr_qp *qp, /* Update doorbell (in case post_recv was done before * move to RTR) */ - mmio_wc_start(); - writel(qp->rq.db_data.raw, qp->rq.db); - mmio_flush_writes(); + if (IS_ROCE(qp->ibv_qp.context->device)) { + mmio_wc_start(); + writel(qp->rq.db_data.raw, qp->rq.db); + mmio_flush_writes(); + } break; case QELR_QPS_ERR: break; @@ -872,6 +886,10 @@ static inline void qelr_init_dpm_info(struct qelr_devctx *cxt, { dpm->is_edpm = 0; + /* Currently dpm is not supported for iWARP */ + if (IS_IWARP(cxt->ibv_ctx.device)) + return; + if (qelr_chain_is_full(&qp->sq.chain) && wr->send_flags & IBV_SEND_INLINE && !qp->edpm_disabled) { memset(dpm, 0, sizeof(*dpm)); @@ -1434,7 +1452,8 @@ int qelr_post_send(struct ibv_qp *ib_qp, struct ibv_send_wr *wr, pthread_spin_lock(&qp->q_lock); - if ((qp->state != QELR_QPS_RTS && qp->state != QELR_QPS_ERR && + if (IS_ROCE(ib_qp->context->device) && + (qp->state != QELR_QPS_RTS && qp->state != QELR_QPS_ERR && qp->state != QELR_QPS_SQD)) { pthread_spin_unlock(&qp->q_lock); *bad_wr = wr; @@ -1474,10 +1493,11 @@ int qelr_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, struct qelr_qp *qp = get_qelr_qp(ibqp); struct qelr_devctx *cxt = get_qelr_ctx(ibqp->context); uint16_t db_val; + uint8_t iwarp = IS_IWARP(ibqp->context->device); pthread_spin_lock(&qp->q_lock); - if (qp->state == QELR_QPS_RST) { + if (!iwarp && qp->state == QELR_QPS_RST) { pthread_spin_unlock(&qp->q_lock); *bad_wr = wr; return -EINVAL; @@ -1546,6 +1566,10 @@ int qelr_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, writel(qp->rq.db_data.raw, qp->rq.db); mmio_flush_writes(); + if (iwarp) { + writel(qp->rq.iwarp_db2_data.raw, qp->rq.iwarp_db2); + mmio_flush_writes(); + } wr = wr->next; }