From patchwork Tue Feb 26 09:18:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gal Pressman X-Patchwork-Id: 10829743 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ACB871805 for ; Tue, 26 Feb 2019 09:19:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95BF72AF0B for ; Tue, 26 Feb 2019 09:19:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 86B7C2AF15; Tue, 26 Feb 2019 09:19:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 971212AF0B for ; Tue, 26 Feb 2019 09:19:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727493AbfBZJT3 (ORCPT ); Tue, 26 Feb 2019 04:19:29 -0500 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:14996 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725941AbfBZJT2 (ORCPT ); Tue, 26 Feb 2019 04:19:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1551172765; x=1582708765; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=ZGF2Qxtwvp9I4ZicKLWLD0FTtc3J1oXBUXFSM2ISuV0=; b=Jwldzrsk+Fu8HP3wdCQNzF6PtK87TGkKRBnUj7opwFBj5XHvzUnsRNwC 7jB4WzLu/0IgRyJXDMXAgEduVgJfMXIknMR0saBgfoajFBNCh716lNgyx bDiWBjlbT1CwQBO1+Uu1lhUJRSWa092PsZso6faag7+n68lPGghqwlpGR E=; X-IronPort-AV: E=Sophos;i="5.58,415,1544486400"; d="scan'208";a="760654497" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 26 Feb 2019 09:19:23 +0000 Received: from EX13MTAUEA001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan2.pdx.amazon.com [10.236.137.194]) by email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id x1Q9JIgh011984 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Tue, 26 Feb 2019 09:19:22 GMT Received: from EX13D03EUA001.ant.amazon.com (10.43.165.33) by EX13MTAUEA001.ant.amazon.com (10.43.61.243) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 26 Feb 2019 09:19:21 +0000 Received: from EX13MTAUEA001.ant.amazon.com (10.43.61.82) by EX13D03EUA001.ant.amazon.com (10.43.165.33) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 26 Feb 2019 09:19:20 +0000 Received: from galpress-VirtualBox.hfa16.amazon.com (10.218.62.29) by mail-relay.amazon.com (10.43.61.243) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 26 Feb 2019 09:19:16 +0000 From: Gal Pressman To: Jason Gunthorpe , Doug Ledford CC: Yossi Leybovich , Alexander Matushevsky , Leah Shalev , Dave Goodell , Brian Barrett , , Sean Hefty , "Dennis Dalessandro" , Leon Romanovsky , Christoph Hellwig , Parav Pandit , Sagi Grimberg , Gal Pressman Subject: [PATCH rdma-core 2/3] efa: Elastic Fabric Adapter (EFA) direct verbs Date: Tue, 26 Feb 2019 11:18:58 +0200 Message-ID: <1551172739-399-3-git-send-email-galpress@amazon.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551172739-399-1-git-send-email-galpress@amazon.com> References: <1551172739-399-1-git-send-email-galpress@amazon.com> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In addition to verbs implementation, efa provider implements direct verb for QP creation which allows users to create Scalable Reliable Datagram (SRD) QPs. Signed-off-by: Gal Pressman --- debian/ibverbs-providers.install | 1 + debian/ibverbs-providers.lintian-overrides | 4 +- debian/ibverbs-providers.symbols | 4 ++ debian/libibverbs-dev.install | 4 ++ providers/efa/CMakeLists.txt | 9 ++++- providers/efa/efadv.h | 59 ++++++++++++++++++++++++++++++ providers/efa/libefa.map | 7 ++++ providers/efa/verbs.c | 11 ++++++ redhat/rdma-core.spec | 1 + suse/rdma-core.spec | 18 +++++++++ 10 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 providers/efa/efadv.h create mode 100644 providers/efa/libefa.map diff --git a/debian/ibverbs-providers.install b/debian/ibverbs-providers.install index 43a8727f30e3..025a0e50f50c 100644 --- a/debian/ibverbs-providers.install +++ b/debian/ibverbs-providers.install @@ -2,3 +2,4 @@ etc/libibverbs.d/ usr/lib/*/libibverbs/lib*-rdmav*.so usr/lib/*/libmlx4.so.* usr/lib/*/libmlx5.so.* +usr/lib/*/libefa.so.* diff --git a/debian/ibverbs-providers.lintian-overrides b/debian/ibverbs-providers.lintian-overrides index a33f879e68ea..07b7acc8ae03 100644 --- a/debian/ibverbs-providers.lintian-overrides +++ b/debian/ibverbs-providers.lintian-overrides @@ -1,2 +1,2 @@ -# libmlx4 and libmlx5 are ibverbs provider that provides more functions. -ibverbs-providers: package-name-doesnt-match-sonames libmlx4-1 libmlx5-1 +# libmlx4, libmlx5 and libefa are ibverbs provider that provides more functions. +ibverbs-providers: package-name-doesnt-match-sonames libmlx4-1 libmlx5-1 libefa-1 diff --git a/debian/ibverbs-providers.symbols b/debian/ibverbs-providers.symbols index 9be0a9483dce..85bd5326f588 100644 --- a/debian/ibverbs-providers.symbols +++ b/debian/ibverbs-providers.symbols @@ -57,3 +57,7 @@ libmlx5.so.1 ibverbs-providers #MINVER# mlx5dv_devx_destroy_cmd_comp@MLX5_1.9 23 mlx5dv_devx_get_async_cmd_comp@MLX5_1.9 23 mlx5dv_devx_obj_query_async@MLX5_1.9 23 +libefa.so.1 ibverbs-providers #MINVER# +* Build-Depends-Package: libibverbs-dev + EFA_1.0@EFA_1.0 23 + efadv_create_driver_qp@EFA_1.0 23 diff --git a/debian/libibverbs-dev.install b/debian/libibverbs-dev.install index 19588068c43d..b0c77825fd6c 100644 --- a/debian/libibverbs-dev.install +++ b/debian/libibverbs-dev.install @@ -4,6 +4,7 @@ usr/include/infiniband/mlx4dv.h usr/include/infiniband/mlx5_api.h usr/include/infiniband/mlx5_user_ioctl_verbs.h usr/include/infiniband/mlx5dv.h +usr/include/infiniband/efadv.h usr/include/infiniband/opcode.h usr/include/infiniband/sa-kern-abi.h usr/include/infiniband/sa.h @@ -17,9 +18,12 @@ usr/lib/*/libmlx4.a usr/lib/*/libmlx4.so usr/lib/*/libmlx5.a usr/lib/*/libmlx5.so +usr/lib/*/libefa.a +usr/lib/*/libefa.so usr/lib/*/pkgconfig/libibverbs.pc usr/lib/*/pkgconfig/libmlx4.pc usr/lib/*/pkgconfig/libmlx5.pc +usr/lib/*/pkgconfig/libefa.pc usr/share/man/man3/ibv_* usr/share/man/man3/mbps_to_ibv_rate.3 usr/share/man/man3/mlx4dv_*.3 diff --git a/providers/efa/CMakeLists.txt b/providers/efa/CMakeLists.txt index 1fb6acaa4ecb..45385d032250 100644 --- a/providers/efa/CMakeLists.txt +++ b/providers/efa/CMakeLists.txt @@ -1,4 +1,11 @@ -rdma_provider(efa +rdma_shared_provider(efa libefa.map + 1 1.0.${PACKAGE_VERSION} efa.c verbs.c ) + +publish_headers(infiniband + efadv.h +) + +rdma_pkg_config("efa" "libibverbs" "${CMAKE_THREAD_LIBS_INIT}") diff --git a/providers/efa/efadv.h b/providers/efa/efadv.h new file mode 100644 index 000000000000..a356305681ce --- /dev/null +++ b/providers/efa/efadv.h @@ -0,0 +1,59 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ +/* + * Copyright 2019 Amazon.com, Inc. or its affiliates. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef __EFADV_H__ +#define __EFADV_H__ + +#include +#include + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +enum { + /* Values must match the values in efa-abi.h */ + EFADV_QP_DRIVER_TYPE_SRD = 0, +}; + +struct ibv_qp *efadv_create_driver_qp(struct ibv_pd *ibvpd, + struct ibv_qp_init_attr *attr, + uint32_t driver_qp_type); + +#ifdef __cplusplus +} +#endif + +#endif /* __EFADV_H__ */ diff --git a/providers/efa/libefa.map b/providers/efa/libefa.map new file mode 100644 index 000000000000..110b5a438b80 --- /dev/null +++ b/providers/efa/libefa.map @@ -0,0 +1,7 @@ +/* Export symbols should be added below according to + Documentation/versioning.md document. */ +EFA_1.0 { + global: + efadv_create_driver_qp; + local: *; +}; diff --git a/providers/efa/verbs.c b/providers/efa/verbs.c index b821cf2f5d44..9ce0498dff6c 100644 --- a/providers/efa/verbs.c +++ b/providers/efa/verbs.c @@ -47,6 +47,7 @@ #include #include "efa.h" +#include "efadv.h" #include "verbs.h" int efa_query_device(struct ibv_context *ibvctx, @@ -786,6 +787,16 @@ struct ibv_qp *efa_create_qp(struct ibv_pd *ibvpd, return create_qp(ibvpd, attr, 0); } +struct ibv_qp *efadv_create_driver_qp(struct ibv_pd *ibvpd, + struct ibv_qp_init_attr *attr, + uint32_t driver_qp_type) +{ + if (attr->qp_type != IBV_QPT_DRIVER) + return NULL; + + return create_qp(ibvpd, attr, driver_qp_type); +} + int efa_modify_qp(struct ibv_qp *ibvqp, struct ibv_qp_attr *attr, int attr_mask) { diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec index a3a2f6ded447..6c291aaff8a2 100644 --- a/redhat/rdma-core.spec +++ b/redhat/rdma-core.spec @@ -414,6 +414,7 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh %{_libdir}/libibverbs/*.so %{_libdir}/libmlx5.so.* %{_libdir}/libmlx4.so.* +%{_libdir}/libefa.so.* %config(noreplace) %{_sysconfdir}/libibverbs.d/*.driver %doc %{_docdir}/%{name}-%{version}/libibverbs.md diff --git a/suse/rdma-core.spec b/suse/rdma-core.spec index 8479b06b4262..6b235571ef28 100644 --- a/suse/rdma-core.spec +++ b/suse/rdma-core.spec @@ -34,12 +34,14 @@ Group: Productivity/Networking/Other %define umad_so_major 3 %define mlx4_so_major 1 %define mlx5_so_major 1 +%define efa_so_major 1 %define verbs_lname libibverbs%{verbs_so_major} %define rdmacm_lname librdmacm%{rdmacm_so_major} %define umad_lname libibumad%{umad_so_major} %define mlx4_lname libmlx4-%{mlx4_so_major} %define mlx5_lname libmlx5-%{mlx5_so_major} +%define efa_lname libefa-%{efa_so_major} %ifnarch s390 %arm %define dma_coherent 1 @@ -137,6 +139,7 @@ Requires: %{verbs_lname} = %{version}-%{release} %if 0%{?dma_coherent} Requires: %{mlx4_lname} = %{version}-%{release} Requires: %{mlx5_lname} = %{version}-%{release} +Requires: %{efa_lname} = %{version}-%{release} %endif Requires: rsocket = %{version}-%{release} @@ -181,6 +184,7 @@ Obsoletes: libefa-rdmav2 < %{version}-%{release} %if 0%{?dma_coherent} Requires: %{mlx4_lname} = %{version}-%{release} Requires: %{mlx5_lname} = %{version}-%{release} +Requires: %{efa_lname} = %{version}-%{release} %endif # Recommended packages for rxe_cfg Recommends: ethtool @@ -233,6 +237,13 @@ Group: System/Libraries %description -n %mlx5_lname This package contains the mlx5 runtime library. + +%package -n %efa_lname +Summary: EFA runtime library +Group: System/Libraries + +%description -n %efa_lname +This package contains the efa runtime library. %endif %package -n libibverbs-utils @@ -441,6 +452,9 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh %post -n %mlx5_lname -p /sbin/ldconfig %postun -n %mlx5_lname -p /sbin/ldconfig + +%post -n %efa_lname -p /sbin/ldconfig +%postun -n %efa_lname -p /sbin/ldconfig %endif %post -n %umad_lname -p /sbin/ldconfig @@ -608,6 +622,10 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh %files -n %mlx5_lname %defattr(-,root,root) %{_libdir}/libmlx5*.so.* + +%files -n %efa_lname +%defattr(-,root,root) +%{_libdir}/libefa*.so.* %endif %files -n libibverbs-utils