From patchwork Thu Jun 20 16:21:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bernard Metzler X-Patchwork-Id: 11007373 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 730841580 for ; Thu, 20 Jun 2019 16:21:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 64BBD1FF82 for ; Thu, 20 Jun 2019 16:21:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 595D6287DC; Thu, 20 Jun 2019 16:21:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6AE46287B8 for ; Thu, 20 Jun 2019 16:21:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732051AbfFTQVv (ORCPT ); Thu, 20 Jun 2019 12:21:51 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:43896 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726562AbfFTQVv (ORCPT ); Thu, 20 Jun 2019 12:21:51 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5KGJm0H126956 for ; Thu, 20 Jun 2019 12:21:50 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2t8d9ygtwg-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jun 2019 12:21:50 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 20 Jun 2019 17:21:48 +0100 Received: from b06avi18626390.portsmouth.uk.ibm.com (9.149.26.192) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 20 Jun 2019 17:21:46 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5KGLaHc32637254 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 20 Jun 2019 16:21:36 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C022E11C054; Thu, 20 Jun 2019 16:21:44 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9C21911C04C; Thu, 20 Jun 2019 16:21:44 +0000 (GMT) Received: from spoke.zurich.ibm.com (unknown [9.4.69.152]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 20 Jun 2019 16:21:44 +0000 (GMT) From: Bernard Metzler To: linux-rdma@vger.kernel.org Cc: Bernard Metzler Subject: [PATCH v3 00/11] SIW: Software iWarp RDMA (siw) driver Date: Thu, 20 Jun 2019 18:21:22 +0200 X-Mailer: git-send-email 2.17.2 X-TM-AS-GCONF: 00 x-cbid: 19062016-0028-0000-0000-0000037C17EB X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19062016-0029-0000-0000-0000243C2B30 Message-Id: <20190620162133.13074-1-bmt@zurich.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-20_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906200119 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch set contributes the SoftiWarp driver rebased for latest rdma-next. SoftiWarp (siw) implements the iWarp RDMA protocol over kernel TCP sockets. The driver integrates with the linux-rdma framework. A matching userlevel driver is available as PR at https://github.com/linux-rdma/rdma-core/pull/536 Many thanks for reviewing and testing the driver, especially to Leon, Jason, Steve, Doug, Olga, Dennis, Gal. You all helped to significantly improve the driver over the last year. Please find below a list of changes and comments, compared to older versions of the siw driver. Many thanks! Bernard. CHANGES: ======== v3 (this version) ----------------- - Rebased to rdma-next - Removed unneccessary initialization of enums in siw-abi.h - Added comment on sizing of all work queues to power of two. v2 ----------------- - Changed recieve path CRC calculation to compute CRC32c not on target buffer after placement, but on original skbuf. This change severely hurts performance, if CRC is switched on, since skb must now be walked twice. It is planned to work on an extension to skb_copy_bits() to fold in CRC computation. - Moved debugging to using ibdev_dbg(). - Dropped detailed packet debug printing. - Removed siw_debug.[ch] files. - Removed resource tracking, code now relies on restrack of RDMA midlayer. Only object counting to enforce reported device limits is left in place. - Removed all nested switch-case statements. - Cleaned up header file #include's - Moved CQ create/destroy to new semantics, where midlayer creates/destroys containing object. - Set siw's ABI version to 1 (was 0 before) - Removed all enum initialization where not needed. - Fixed MAINTANERS entry for siw driver - This version stays with the current siw specific management of user memory (siw_umem_get() vs. ib_umem_get(), etc.). This, since the current ib_umem implementation is less efficient for user page lookup on the fast path, where effciency is important for a SW RDMA driver. It is planned to contribute enhancements to the ib_umem framework, wich makes it suitable for SW drivers as well. v1 (first version after v9 of siw RFC) -------------------------------------- - Rebased to 5.2-rc1 - All IDR code got removed. - Both MR and QP deallocation verbs now synchronously free the resources referenced by the RDMA mid-layer. - IPv6 support was added. - For compatibility with Chelsio iWarp hardware, the RX path was slightly reworked. It now allows packet intersection between tagged and untagged RDMAP operations. While not a defined behavior as of IETF RFC 5040/5041, some RDMA hardware may intersect an ongoing outbound (large) tagged message, such as an multisegment RDMA Read Response with sending an untagged message, such as an RDMA Send frame. This behavior was only detected in an NVMeF setup, where siw was used at target side, and RDMA hardware at client side (during file write). siw now implements two input paths for tagged and untagged messages each, and allows the intersected placement of both messages. - The siw kernel abi file got renamed from siw_user.h to siw-abi.h. Bernard Metzler (11): iWarp wire packet format SIW main include file SIW network and RDMA core interface SIW connection management SIW application interface SIW application buffer management SIW queue pair methods SIW transmit path SIW receive path SIW completion queue methods SIW addition to kernel build environment MAINTAINERS | 7 + drivers/infiniband/Kconfig | 1 + drivers/infiniband/sw/Makefile | 1 + drivers/infiniband/sw/siw/Kconfig | 17 + drivers/infiniband/sw/siw/Makefile | 11 + drivers/infiniband/sw/siw/iwarp.h | 380 ++++ drivers/infiniband/sw/siw/siw.h | 745 ++++++++ drivers/infiniband/sw/siw/siw_cm.c | 2072 ++++++++++++++++++++++ drivers/infiniband/sw/siw/siw_cm.h | 133 ++ drivers/infiniband/sw/siw/siw_cq.c | 101 ++ drivers/infiniband/sw/siw/siw_main.c | 687 +++++++ drivers/infiniband/sw/siw/siw_mem.c | 460 +++++ drivers/infiniband/sw/siw/siw_mem.h | 74 + drivers/infiniband/sw/siw/siw_qp.c | 1322 ++++++++++++++ drivers/infiniband/sw/siw/siw_qp_rx.c | 1455 +++++++++++++++ drivers/infiniband/sw/siw/siw_qp_tx.c | 1268 +++++++++++++ drivers/infiniband/sw/siw/siw_verbs.c | 1760 ++++++++++++++++++ drivers/infiniband/sw/siw/siw_verbs.h | 91 + include/uapi/rdma/rdma_user_ioctl_cmds.h | 1 + include/uapi/rdma/siw-abi.h | 185 ++ 20 files changed, 10771 insertions(+) create mode 100644 drivers/infiniband/sw/siw/Kconfig create mode 100644 drivers/infiniband/sw/siw/Makefile create mode 100644 drivers/infiniband/sw/siw/iwarp.h create mode 100644 drivers/infiniband/sw/siw/siw.h create mode 100644 drivers/infiniband/sw/siw/siw_cm.c create mode 100644 drivers/infiniband/sw/siw/siw_cm.h create mode 100644 drivers/infiniband/sw/siw/siw_cq.c create mode 100644 drivers/infiniband/sw/siw/siw_main.c create mode 100644 drivers/infiniband/sw/siw/siw_mem.c create mode 100644 drivers/infiniband/sw/siw/siw_mem.h create mode 100644 drivers/infiniband/sw/siw/siw_qp.c create mode 100644 drivers/infiniband/sw/siw/siw_qp_rx.c create mode 100644 drivers/infiniband/sw/siw/siw_qp_tx.c create mode 100644 drivers/infiniband/sw/siw/siw_verbs.c create mode 100644 drivers/infiniband/sw/siw/siw_verbs.h create mode 100644 include/uapi/rdma/siw-abi.h