From patchwork Wed Aug 3 23:27:33 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adit Ranadive X-Patchwork-Id: 9262357 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C72996048B for ; Wed, 3 Aug 2016 23:39:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B608E27FA8 for ; Wed, 3 Aug 2016 23:39:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA6EB28066; Wed, 3 Aug 2016 23:39:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2332027FB6 for ; Wed, 3 Aug 2016 23:39:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932493AbcHCXjc (ORCPT ); Wed, 3 Aug 2016 19:39:32 -0400 Received: from smtp-outbound-1.vmware.com ([208.91.2.12]:44320 "EHLO smtp-outbound-1.vmware.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752301AbcHCXj2 (ORCPT ); Wed, 3 Aug 2016 19:39:28 -0400 Received: from sc9-mailhost2.vmware.com (sc9-mailhost2.vmware.com [10.113.161.72]) by smtp-outbound-1.vmware.com (Postfix) with ESMTP id 6CB9D9867F; Wed, 3 Aug 2016 16:28:18 -0700 (PDT) Received: from EX13-CAS-005.vmware.com (smtp-inbound.vmware.com [10.113.191.55]) by sc9-mailhost2.vmware.com (Postfix) with ESMTP id 6607AB06EF; Wed, 3 Aug 2016 16:28:19 -0700 (PDT) Received: from EX13-CAS-001.vmware.com (10.113.191.51) by EX13-MBX-003.vmware.com (10.113.191.75) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Wed, 3 Aug 2016 16:28:19 -0700 Received: from NAM03-DM3-obe.outbound.protection.outlook.com (10.113.170.11) by EX13-CAS-001.vmware.com (10.113.191.51) with Microsoft SMTP Server (TLS) id 15.0.1156.6 via Frontend Transport; Wed, 3 Aug 2016 16:28:19 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=onevmw.onmicrosoft.com; s=selector1-vmware-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=7b2ShWXqTpz84HmCQGt6/XK4hST/53MDJU9Ch1jAJUI=; b=ArZW5Sp4DiuvWjH2KNZenahAWOcNzka6kw7Ml3KZnAjj+jG52IN7Bflv5VUQ0OM3oqVj1tIXx2PJmanJcvOBXbaskBzh6JLt56f+ncY+ekB2ZZpXczA/0NieGTmgurzqwrkhdrltdvAnOj+u03qy1GFQJOp5uyX417pxZr7ujj4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=aditr@vmware.com; Received: from promb-2s-dhcp95-136.eng.vmware.com (208.91.1.34) by SN2PR0501MB847.namprd05.prod.outlook.com (10.160.14.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.549.15; Wed, 3 Aug 2016 23:28:13 +0000 From: Adit Ranadive To: , , CC: Adit Ranadive , , , , Subject: [PATCH v3 04/15] IB/pvrdma: Add functions for Verbs support Date: Wed, 3 Aug 2016 16:27:33 -0700 Message-ID: <1470266864-16888-5-git-send-email-aditr@vmware.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1470266864-16888-1-git-send-email-aditr@vmware.com> References: <1470266864-16888-1-git-send-email-aditr@vmware.com> MIME-Version: 1.0 X-Originating-IP: [208.91.1.34] X-ClientProxiedBy: CY1PR12CA0084.namprd12.prod.outlook.com (10.163.230.52) To SN2PR0501MB847.namprd05.prod.outlook.com (10.160.14.145) X-MS-Office365-Filtering-Correlation-Id: 0958effc-68e1-4ec5-0686-08d3bbf5d7a5 X-Microsoft-Exchange-Diagnostics: 1; SN2PR0501MB847; 2:2zpYt49zYGDYHmG4/949foToT0MiWBUxYU915+gNsj62nw3N6z1v91v+nnudJn9Iw/GFzkJvb2h+wPIcmCxvByFCy0pCRD2/nyP2zzmTT5pBFzYM3kspE3uJY782LnM0vQFWN5IsuVFrggi38In+SusdrpC6VIMzekzSm25xdOfHbdVwddycM9ncmZ6ZUUOo; 3:3aTgz6t4BuvvcHzSOD889055CpWwVyyQSKSaS4Ray4wVezRVaBYKnC0e0H3EEuE2H9NWG4YHbwwZijfputJX8KQV1QEIofVx0AFeugbzQBvuSEFFW0ODVy2hfkbq/chp; 25:Q7wscD3UqSaD9Hxv/E5THx39Aqz1XHjpDgNVUinWsw7L7fVkLeMLzJw2prd9mXIq9uKoOHYI2bUI1Rfo2M+eqTcuaODZylct1tywrSkavvoEU8SaIRJZ3oP7dPsKY0HJFI8nRPDgzJK9U2j+8xr2NO7207/sqN8xVeg373pHZDUnGUs4Mop7iRfneeE51pn50QXPBHAUUw6oPJsardJHQhhfEgjf0nm6JGxcwlwCtFmI8w6PVt9XSEVT6ee6VA44cciIKJlzB1FFmnrCUDF7r10ke2rJrUbqog1Pb17a5GmSe3pgAsZd88vpUCp81SnqtNdS+vjdzHS6j/sRIfj0SAtxj38cRa+kZU5YEvgO+U4YBTuB+TRg+zkcmIi3Jkd3xkF3T4ziglienHObRuhVQw== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN2PR0501MB847; X-Microsoft-Exchange-Diagnostics: 1; SN2PR0501MB847; 31:dK/8lDkTYm5nOW3f8YWTMiaaUIdYvJM4gSBJi1J8wFuByyXGipw9AZXxV8u8pCuIJawI+EDqaIdzR0oTu11Kfxi+APO8T5WBHITpy1UstNUYrrC1xuOhdHZY5Z0QtAG2Wb1oCN+OEeMsn4Lzwkq9JgtqLU4tQ9wnDwb7gHV60kALS/9JAsK5XIUOI1rKfVZViSLf8lO6q5x59FOqZtJMCN83SO2NGE73UmSHnhn754c=; 20:ArSbP/hxHunsj4xbiLJU73YVYML/wGXqvz438NGkGorszewNG7tEyQ+BFV3glPtJYsqDVFPDu9dsUOXInPWBRqFnQTyThAyouw0frYfqEkHW3kR9XsntRhYaeSg3am+LSzSOaXQzF+fVs4UDBbBwX80YEpFI4gtXWqzyal7gQRg064NFgNDxDyzOYATRXxmUgRtIvzA76RMygHfXpek6dALtSd7belPbdbAx/PXcYsVI6DZs81RAPGmBoX0Q6wnf X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(250305191791016)(61668805478150)(22074186197030); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:SN2PR0501MB847; BCL:0; PCL:0; RULEID:; SRVR:SN2PR0501MB847; X-Microsoft-Exchange-Diagnostics: 1; SN2PR0501MB847; 4:wz+q/ukd0YEPehMwgfKoRw05BRujI4Ha5+w0m3uylQIw4utcf98kkx8paR3sNeUOeH2CpAGiOx3Wll755GFYKQvzlboQjZZ5DX3UAYEt6v/5tx6vH7UTMz1snttcj4IAdKmHSbGYhDCXRrvwaotrl1mBU8YpCN0CtvV/cU1p4VmHqOuO5skn/gW0J8wqILrQRPqWZB0xxcdbVmMqJHRM6ygIZOmfnec62uwdacZzggtkI2ql5sA4NH/31EvV7CdiVejWKvr9aoe4Mde8adfPOc/RAWHY9/T9/feGkK1PljTtRJFQi+wiXv1CjH+2fG7c+goJs8MaAXMRksXud7C+Jj10xnCr1XauGixOx/CDAKTvj1D7LjhUuiKfwzvJfn5ROabqJlQhTOa5zJnpZUHTo7YMeKk07PPYVDR8HGgYonckeAwXlz5TioD6u9rrlK21AwvSr/lEc4ylolK0adMmhwFN7kM5P1gVtxYwGkgldsk= X-Forefront-PRVS: 00235A1EEF X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(189002)(4001450100002)(66066001)(106356001)(189998001)(5890100001)(33646002)(92566002)(107886002)(5001770100001)(77096005)(15975445007)(229853001)(48376002)(50466002)(101416001)(97736004)(2950100001)(42186005)(586003)(47776003)(68736007)(76176999)(50986999)(4001430100002)(2906002)(105586002)(7846002)(7736002)(305945005)(81156014)(50226002)(86362001)(81166006)(8676002)(6116002)(36756003)(3846002)(5003940100001)(19580405001)(19580395003)(2201001)(4326007)(7099028)(2004002); DIR:OUT; SFP:1101; SCL:1; SRVR:SN2PR0501MB847; H:promb-2s-dhcp95-136.eng.vmware.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN2PR0501MB847; 23:yKlz6uNVw15eIP3aJgkABXBymbVr7X563jCIH5f7?= =?us-ascii?Q?0d09RLiJWy+qGB0vRuG7edSpAcKSqyO3UbrwA4xZo+WB1EA6vLRa30kAYYn0?= =?us-ascii?Q?9q4v7bjwjEZ9AnjMftJmeE2eRtGo0CULQpoU1ACwE4bSzJnkX5SCcjB8vuNT?= =?us-ascii?Q?IQNcSWRv1UVZwrijBwQ8ID9QXImRHqde0wXGGrsjKlnWn3EuJo32axHRqC5K?= =?us-ascii?Q?CXTZcn72dB4N1CRhqdlX73XTZY6mEIKEUD+kGSDqyeazErw+TXBx/FT1KY9n?= =?us-ascii?Q?h+pz30BBiqVOcvG66l3pdRgn/fLOPnkAHgfzXVRJUeSD8lep0KhjeetwEuHi?= =?us-ascii?Q?AfgvQslrc287wpBmSmEnhPUOx7LI8KSYpPn0UkUs+Te+SUHNAlK2+UXQ1IdJ?= =?us-ascii?Q?bu0ItYHiDmtjRJKjHV7uOAHf5yTGu7wmc8sN3scz8S3DdbV/BEo7lAPg+sRV?= =?us-ascii?Q?IeciHAAQyXJMAs7RBxfwY+B9qKKl8m6CAa4AqhK/mROt2J42CWBCyj81fUVq?= =?us-ascii?Q?2jFrB5xyHsOmCmtICl0OxHoGKvhJTogIUZ4Fq1NYGLQwIaDuSw4yv3UyXV7j?= =?us-ascii?Q?kABG6wiCMYjNBk4yxTrExDZep053H2dEYx1L6OshX6An8kaJ6OfO7HnwJfjH?= =?us-ascii?Q?MgEWGmIyEEZ2BckUojSzLeLBbbFxr/8xftqf1FfK4HFcaUpReMhLtQOmsq+7?= =?us-ascii?Q?cl9prs4u0Y6Afw6CHGKDPuLRrhqkPeqlqBhpkPaVZwHsScvB3Hw2KSXhU6j7?= =?us-ascii?Q?JtDlNAzCkAtD8S/PTHquu1FIO1WYKjBkmBWOtBT1B4drhRVuXHQw2tmM6AoR?= =?us-ascii?Q?ZJdOG2O66fhq+zcS9s39cCyvJvTqT3qPijV20zUUNo3yHjri0JVkNALMUNj/?= =?us-ascii?Q?AC7txtvdraxxt7jaRRYP7A9jG9ybKvsIWInsU/7oAkN8SRH3r7MD0XdxrYYH?= =?us-ascii?Q?mXd9XMBTptQ/Vbe7a+NQereq2DcIZKNP56/f0PaHDOrlKPqvLu1wQIRh1p1l?= =?us-ascii?Q?8HNpxtuXsPwMlRP2hAdlt61BqZvB5cbld4OT/iFVCGLniYz3LJwoU/x3hoZH?= =?us-ascii?Q?0CbxvM3Ln41IoOBOMsQcndmAKM30+zhwxmnC8IbPHHiYhqw5JZhcTPEi/piv?= =?us-ascii?Q?YQ8pNgTztiimJZxW9+ely5/Z9Bsmr/aQ3TTbnEZoprMi3rZdBreCUv6LL4aZ?= =?us-ascii?Q?xFW/AUAyeeMplCRZwwjbusD0e8jOQwlRZXgJedymIshvUEMG0Ccy6JOhwIBP?= =?us-ascii?Q?o8ciosOpbndRcktaRRA=3D?= X-Microsoft-Exchange-Diagnostics: 1; SN2PR0501MB847; 6:+B2F9qwwZ6Tmz9z0aS+Jm+nNPyb8KYj8HA7BeIV9i28XnQheSJljKXbQdXQr0PYGPAICj3VoBKMxBm5Jq7GKauiBR7I/weme4ykK6VYvMmpX2g6MpgAJ/V+KlLWjfp6LBmA2d7CrmKfkrt0qan5Y71XeEy74+NTAsI4Xe9GDyVyP1vnehZq04V+t/Ht+oQdKLjnz5M0EOeIRSIxzQy1hOO1C17J+O9PWZns1zugB2pb146vMulFdGfQg5YDXDmh0LTEDs6Iw9cpDH2lHnpMvqQmwgaqxInHsW1/PHnfno60=; 5:Az9M37aKae1XUi/NA3NoxLU63/yNFpDKfwJ/NotlCweLTPKKNHWFzfXvbbskPC5YTumsU/wQqm1yGbSNXZIaeeW3CGIC+s9ye9iu73LOUjVf2x9ybUjaOr27UF9vDPRFQe6PeW/s/MO688/YnT8h3A==; 24:HhCm7jJJlLrYryEblLJnlkn1Esv9rOpUF3wDNG3G/Hhh0c2+6gFVdN6v/Ui2abARQXiQR5e5aQoNkU1donHm47QpfF5CVP+29Ni3zQvq4Gk=; 7:nZSQtjOrIx+AMnnofWGyoR5KK05mM5wzsyRDKEmE/ggUz0rWyyFI9aqXZ0KeVArDARM4jCejM9gXVaBVrFV4g+CGmGEp9Tk/7H7MGkn4MSXX/JdJIBqkA0uCWv0trCEL9KUdSGtIxrmfEdSYorH8C5hkdmqjyxYv4vsygrS/nQzbxl+pqX9dQIPRPnUqg/kOeJNX2F8Q+IDM3UXIOM9fHrWJj9OKdwhp1EvPS6RntPJ4Kw8bheWP3xjqvEehCbwW SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; SN2PR0501MB847; 20:W+IHKFoLLUjqKmlNUrj3GCxeZNg0DDO02bTGWaU0QBssrDSOpyrVHZALQGf5DKQ1BOH5RQybi9Q6jGB/73w/SMjs/zuySccmbDrR8pUa1KiOH5KalJg31uhosj5OuGCERmL1LwJJPumB5Rze+I0GxED7jTGwRASU7AmXP/mbFVk= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Aug 2016 23:28:13.7888 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR0501MB847 X-OriginatorOrg: vmware.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch implements the remaining Verbs functions registered with the core RDMA stack. Changes v2->v3: - Removed the boolean from pvrdma_cmd_post call. Reviewed-by: Jorgen Hansen Reviewed-by: George Zhang Reviewed-by: Aditya Sarwade Reviewed-by: Bryan Tan Signed-off-by: Adit Ranadive --- drivers/infiniband/hw/pvrdma/pvrdma_verbs.c | 593 ++++++++++++++++++++++++++++ drivers/infiniband/hw/pvrdma/pvrdma_verbs.h | 108 +++++ 2 files changed, 701 insertions(+) create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_verbs.c create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_verbs.h diff --git a/drivers/infiniband/hw/pvrdma/pvrdma_verbs.c b/drivers/infiniband/hw/pvrdma/pvrdma_verbs.c new file mode 100644 index 0000000..1a90790 --- /dev/null +++ b/drivers/infiniband/hw/pvrdma/pvrdma_verbs.c @@ -0,0 +1,593 @@ +/* + * Copyright (c) 2012-2016 VMware, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of EITHER the GNU General Public License + * version 2 as published by the Free Software Foundation or the BSD + * 2-Clause License. This program is distributed in the hope that it + * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED + * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + * See the GNU General Public License version 2 for more details at + * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html. + * + * You should have received a copy of the GNU General Public License + * along with this program available in the file COPYING in the main + * directory of this source tree. + * + * The BSD 2-Clause License + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include + +#include "pvrdma.h" +#include "pvrdma_user.h" + +/** + * pvrdma_query_device - query device + * @ibdev: the device to query + * @props: the device properties + * @uhw: user data + * + * @return: 0 on success, otherwise negative errno + */ +int pvrdma_query_device(struct ib_device *ibdev, + struct ib_device_attr *props, + struct ib_udata *uhw) +{ + struct pvrdma_dev *dev = to_vdev(ibdev); + + if (uhw->inlen || uhw->outlen) + return -EINVAL; + + memset(props, 0, sizeof(*props)); + + props->fw_ver = dev->dsr->caps.fw_ver; + props->sys_image_guid = dev->dsr->caps.sys_image_guid; + props->max_mr_size = dev->dsr->caps.max_mr_size; + props->page_size_cap = dev->dsr->caps.page_size_cap; + props->vendor_id = dev->dsr->caps.vendor_id; + props->vendor_part_id = dev->pdev->device; + props->hw_ver = dev->dsr->caps.hw_ver; + props->max_qp = dev->dsr->caps.max_qp; + props->max_qp_wr = dev->dsr->caps.max_qp_wr; + props->device_cap_flags = dev->dsr->caps.device_cap_flags; + props->max_sge = dev->dsr->caps.max_sge; + props->max_sge_rd = dev->dsr->caps.max_sge_rd; + props->max_cq = dev->dsr->caps.max_cq; + props->max_cqe = dev->dsr->caps.max_cqe; + props->max_mr = dev->dsr->caps.max_mr; + props->max_pd = dev->dsr->caps.max_pd; + props->max_qp_rd_atom = dev->dsr->caps.max_qp_rd_atom; + props->max_ee_rd_atom = dev->dsr->caps.max_ee_rd_atom; + props->max_res_rd_atom = dev->dsr->caps.max_res_rd_atom; + props->max_qp_init_rd_atom = dev->dsr->caps.max_qp_init_rd_atom; + props->max_ee_init_rd_atom = dev->dsr->caps.max_ee_init_rd_atom; + props->atomic_cap = + dev->dsr->caps.atomic_ops & + (PVRDMA_ATOMIC_OP_COMP_SWAP | PVRDMA_ATOMIC_OP_FETCH_ADD) ? + IB_ATOMIC_HCA : IB_ATOMIC_NONE; + props->masked_atomic_cap = props->atomic_cap; + props->max_ee = dev->dsr->caps.max_ee; + props->max_rdd = dev->dsr->caps.max_rdd; + props->max_mw = dev->dsr->caps.max_mw; + props->max_raw_ipv6_qp = dev->dsr->caps.max_raw_ipv6_qp; + props->max_raw_ethy_qp = dev->dsr->caps.max_raw_ethy_qp; + props->max_mcast_grp = dev->dsr->caps.max_mcast_grp; + props->max_mcast_qp_attach = dev->dsr->caps.max_mcast_qp_attach; + props->max_total_mcast_qp_attach = + dev->dsr->caps.max_total_mcast_qp_attach; + props->max_ah = dev->dsr->caps.max_ah; + props->max_fmr = dev->dsr->caps.max_fmr; + props->max_map_per_fmr = dev->dsr->caps.max_map_per_fmr; + props->max_srq = dev->dsr->caps.max_srq; + props->max_srq_wr = dev->dsr->caps.max_srq_wr; + props->max_srq_sge = dev->dsr->caps.max_srq_sge; + props->max_fast_reg_page_list_len = 0; + props->max_pkeys = dev->dsr->caps.max_pkeys; + props->local_ca_ack_delay = dev->dsr->caps.local_ca_ack_delay; + if ((dev->dsr->caps.bmme_flags & PVRDMA_BMME_FLAG_LOCAL_INV) && + (dev->dsr->caps.bmme_flags & PVRDMA_BMME_FLAG_REMOTE_INV) && + (dev->dsr->caps.bmme_flags & PVRDMA_BMME_FLAG_FAST_REG_WR)) { + props->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; + } + + return 0; +} + +/** + * pvrdma_query_port - query device port attributes + * @ibdev: the device to query + * @port: the port number + * @props: the device properties + * + * @return: 0 on success, otherwise negative errno + */ +int pvrdma_query_port(struct ib_device *ibdev, u8 port, + struct ib_port_attr *props) +{ + struct pvrdma_dev *dev = to_vdev(ibdev); + union pvrdma_cmd_req req; + union pvrdma_cmd_resp rsp; + struct pvrdma_cmd_query_port *cmd = &req.query_port; + struct pvrdma_cmd_query_port_resp *resp = &rsp.query_port_resp; + int err; + + memset(cmd, 0, sizeof(*cmd)); + cmd->hdr.cmd = PVRDMA_CMD_QUERY_PORT; + cmd->port_num = port; + + err = pvrdma_cmd_post(dev, &req, &rsp); + if (err < 0 || rsp.hdr.ack != PVRDMA_CMD_QUERY_PORT_RESP) { + dev_warn(&dev->pdev->dev, "could not query port\n"); + return -EFAULT; + } + + memset(props, 0, sizeof(*props)); + + props->state = pvrdma_port_state_to_ib(resp->attrs.state); + props->max_mtu = pvrdma_mtu_to_ib(resp->attrs.max_mtu); + props->active_mtu = pvrdma_mtu_to_ib(resp->attrs.active_mtu); + props->gid_tbl_len = resp->attrs.gid_tbl_len; + props->port_cap_flags = + pvrdma_port_cap_flags_to_ib(resp->attrs.port_cap_flags); + props->max_msg_sz = resp->attrs.max_msg_sz; + props->bad_pkey_cntr = resp->attrs.bad_pkey_cntr; + props->qkey_viol_cntr = resp->attrs.qkey_viol_cntr; + props->pkey_tbl_len = resp->attrs.pkey_tbl_len; + props->lid = resp->attrs.lid; + props->sm_lid = resp->attrs.sm_lid; + props->lmc = resp->attrs.lmc; + props->max_vl_num = resp->attrs.max_vl_num; + props->sm_sl = resp->attrs.sm_sl; + props->subnet_timeout = resp->attrs.subnet_timeout; + props->init_type_reply = resp->attrs.init_type_reply; + props->active_width = pvrdma_port_width_to_ib(resp->attrs.active_width); + props->active_speed = pvrdma_port_speed_to_ib(resp->attrs.active_speed); + props->phys_state = resp->attrs.phys_state; + + return 0; +} + +/** + * pvrdma_query_gid - query device gid + * @ibdev: the device to query + * @port: the port number + * @index: the index + * @gid: the device gid value + * + * @return: 0 on success, otherwise negative errno + */ +int pvrdma_query_gid(struct ib_device *ibdev, u8 port, int index, + union ib_gid *gid) +{ + struct pvrdma_dev *dev = to_vdev(ibdev); + + if (index >= dev->dsr->caps.gid_tbl_len) + return -EINVAL; + + memcpy(gid, &dev->sgid_tbl[index], sizeof(union ib_gid)); + + return 0; +} + +/** + * pvrdma_query_pkey - query device port's P_Key table + * @ibdev: the device to query + * @port: the port number + * @index: the index + * @pkey: the device P_Key value + * + * @return: 0 on success, otherwise negative errno + */ +int pvrdma_query_pkey(struct ib_device *ibdev, u8 port, u16 index, + u16 *pkey) +{ + int err = 0; + union pvrdma_cmd_req req; + union pvrdma_cmd_resp rsp; + struct pvrdma_cmd_query_pkey *cmd = &req.query_pkey; + + memset(cmd, 0, sizeof(*cmd)); + cmd->hdr.cmd = PVRDMA_CMD_QUERY_PKEY; + cmd->port_num = port; + cmd->index = index; + + err = pvrdma_cmd_post(to_vdev(ibdev), &req, &rsp); + if (err < 0 || rsp.hdr.ack != PVRDMA_CMD_QUERY_PKEY_RESP) { + struct pvrdma_dev *dev = to_vdev(ibdev); + + dev_warn(&dev->pdev->dev, "could not query device pkey\n"); + return -EFAULT; + } + + *pkey = rsp.query_pkey_resp.pkey; + + return 0; +} + +enum rdma_link_layer pvrdma_port_link_layer(struct ib_device *ibdev, + u8 port) +{ + return IB_LINK_LAYER_ETHERNET; +} + +int pvrdma_modify_device(struct ib_device *ibdev, int mask, + struct ib_device_modify *props) +{ + unsigned long flags; + int ret; + + if (mask & ~(IB_DEVICE_MODIFY_SYS_IMAGE_GUID | + IB_DEVICE_MODIFY_NODE_DESC)) { + struct pvrdma_dev *dev = to_vdev(ibdev); + + dev_warn(&dev->pdev->dev, "unsupported device modify mask\n"); + ret = -EOPNOTSUPP; + goto err_out; + } + + if (mask & IB_DEVICE_MODIFY_NODE_DESC) { + spin_lock_irqsave(&to_vdev(ibdev)->desc_lock, flags); + memcpy(ibdev->node_desc, props->node_desc, 64); + spin_unlock_irqrestore(&to_vdev(ibdev)->desc_lock, flags); + } + + if (mask & IB_DEVICE_MODIFY_SYS_IMAGE_GUID) { + mutex_lock(&to_vdev(ibdev)->port_mutex); + to_vdev(ibdev)->sys_image_guid = + cpu_to_be64(props->sys_image_guid); + mutex_unlock(&to_vdev(ibdev)->port_mutex); + } + + return 0; + +err_out: + return ret; +} + +/** + * pvrdma_modify_port - modify device port attributes + * @ibdev: the device to modify + * @port: the port number + * @mask: attributes to modify + * @props: the device properties + * + * @return: 0 on success, otherwise negative errno + */ +int pvrdma_modify_port(struct ib_device *ibdev, u8 port, int mask, + struct ib_port_modify *props) +{ + struct ib_port_attr attr; + struct pvrdma_dev *vdev = to_vdev(ibdev); + int ret; + + mutex_lock(&vdev->port_mutex); + ret = pvrdma_query_port(ibdev, port, &attr); + if (ret) + goto err_out; + + vdev->port_cap_mask |= props->set_port_cap_mask; + vdev->port_cap_mask &= ~props->clr_port_cap_mask; + + if (mask & IB_PORT_SHUTDOWN) + vdev->ib_active = false; + + if (mask & (IB_PORT_INIT_TYPE | IB_PORT_RESET_QKEY_CNTR)) { + ret = -EOPNOTSUPP; + goto err_out; + } + + ret = 0; + +err_out: + mutex_unlock(&vdev->port_mutex); + return ret; +} + +/** + * pvrdma_alloc_ucontext - allocate ucontext + * @ibdev: the IB device + * @udata: user data + * + * @return: the ib_ucontext pointer on success, otherwise errno. + */ +struct ib_ucontext *pvrdma_alloc_ucontext(struct ib_device *ibdev, + struct ib_udata *udata) +{ + struct pvrdma_dev *vdev = to_vdev(ibdev); + struct pvrdma_ucontext *context; + union pvrdma_cmd_req req; + union pvrdma_cmd_resp rsp; + struct pvrdma_cmd_create_uc *cmd = &req.create_uc; + struct pvrdma_cmd_create_uc_resp *resp = &rsp.create_uc_resp; + struct pvrdma_alloc_ucontext_resp uresp; + int ret; + + if (!vdev->ib_active) + return ERR_PTR(-EAGAIN); + + context = kmalloc(sizeof(*context), GFP_KERNEL); + if (!context) + return ERR_PTR(-ENOMEM); + + context->dev = vdev; + ret = pvrdma_uar_alloc(vdev, &context->uar); + if (ret) { + kfree(context); + return ERR_PTR(-ENOMEM); + } + + /* get ctx_handle from host */ + memset(cmd, 0, sizeof(*cmd)); + cmd->pfn = context->uar.pfn; + cmd->hdr.cmd = PVRDMA_CMD_CREATE_UC; + ret = pvrdma_cmd_post(vdev, &req, &rsp); + if (ret < 0 || rsp.hdr.ack != PVRDMA_CMD_CREATE_UC_RESP) { + struct pvrdma_dev *dev = to_vdev(ibdev); + + dev_warn(&dev->pdev->dev, "could not create ucontext\n"); + pvrdma_uar_free(vdev, &context->uar); + kfree(context); + return ERR_PTR(-EFAULT); + } + + context->ctx_handle = resp->ctx_handle; + + /* copy back to user */ + uresp.qp_tab_size = vdev->dsr->caps.max_qp; + ret = ib_copy_to_udata(udata, &uresp, sizeof(uresp)); + if (ret) { + pvrdma_uar_free(vdev, &context->uar); + context->ibucontext.device = ibdev; + pvrdma_dealloc_ucontext(&context->ibucontext); + return ERR_PTR(-EFAULT); + } + + return &context->ibucontext; +} + +/** + * pvrdma_dealloc_ucontext - deallocate ucontext + * @ibcontext: the ucontext + * + * @return: 0 on success, otherwise errno. + */ +int pvrdma_dealloc_ucontext(struct ib_ucontext *ibcontext) +{ + struct pvrdma_ucontext *context = to_vucontext(ibcontext); + union pvrdma_cmd_req req; + struct pvrdma_cmd_destroy_uc *cmd = &req.destroy_uc; + int ret; + + memset(cmd, 0, sizeof(*cmd)); + cmd->hdr.cmd = PVRDMA_CMD_DESTROY_UC; + cmd->ctx_handle = context->ctx_handle; + + ret = pvrdma_cmd_post(context->dev, &req, NULL); + if (ret < 0) { + dev_warn(&context->dev->pdev->dev, + "destroy ucontext failed\n"); + /* Just do it */ + } + pvrdma_uar_free(to_vdev(ibcontext->device), &context->uar); + kfree(context); + + return ret; +} + +/** + * pvrdma_mmap - create mmap region + * @ibcontext: the user context + * @vma: the VMA + * + * @return: 0 on success, otherwise errno. + */ +int pvrdma_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma) +{ + struct pvrdma_ucontext *context = to_vucontext(ibcontext); + unsigned long start = vma->vm_start; + unsigned long size = vma->vm_end - vma->vm_start; + unsigned long offset = vma->vm_pgoff << PAGE_SHIFT; + + dev_dbg(&context->dev->pdev->dev, "create mmap region\n"); + + if ((size != PAGE_SIZE) || (offset & ~PAGE_MASK)) { + dev_warn(&context->dev->pdev->dev, + "invalid params for mmap region\n"); + return -EINVAL; + } + + /* Map UAR to kernel space, VM_LOCKED? */ + vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + if (io_remap_pfn_range(vma, start, context->uar.pfn, size, + vma->vm_page_prot)) + return -EAGAIN; + + return 0; +} + +/** + * pvrdma_alloc_pd - allocate protection domain + * @ibdev: the IB device + * @context: user context + * @udata: user data + * + * @return: the ib_pd protection domain pointer on success, otherwise errno. + */ +struct ib_pd *pvrdma_alloc_pd(struct ib_device *ibdev, + struct ib_ucontext *context, + struct ib_udata *udata) +{ + struct pvrdma_pd *pd; + struct pvrdma_dev *dev = to_vdev(ibdev); + union pvrdma_cmd_req req; + union pvrdma_cmd_resp rsp; + struct pvrdma_cmd_create_pd *cmd = &req.create_pd; + struct pvrdma_cmd_create_pd_resp *resp = &rsp.create_pd_resp; + int ret; + + /* Check allowed max pds */ + if (!atomic_add_unless(&dev->num_pds, 1, dev->dsr->caps.max_pd)) + return ERR_PTR(-EINVAL); + + pd = kmalloc(sizeof(*pd), GFP_KERNEL); + if (!pd) { + atomic_dec(&dev->num_pds); + return ERR_PTR(-ENOMEM); + } + + pd->priviledged = !context; + + memset(cmd, 0, sizeof(*cmd)); + cmd->hdr.cmd = PVRDMA_CMD_CREATE_PD; + cmd->ctx_handle = (context) ? to_vucontext(context)->ctx_handle : 0; + ret = pvrdma_cmd_post(dev, &req, &rsp); + if (ret < 0 || resp->hdr.ack != PVRDMA_CMD_CREATE_PD_RESP) { + dev_warn(&dev->pdev->dev, + "failed to allocate protection domain\n"); + kfree(pd); + atomic_dec(&dev->num_pds); + return ERR_PTR(-EFAULT); + } + + pd->pd_handle = resp->pd_handle; + pd->pdn = resp->pd_handle; + + if (context) { + if (ib_copy_to_udata(udata, &pd->pdn, sizeof(__u32))) { + dev_warn(&dev->pdev->dev, + "failed to copy back protection domain\n"); + pvrdma_dealloc_pd(&pd->ibpd); + atomic_dec(&dev->num_pds); + return ERR_PTR(-EFAULT); + } + } + + /* u32 pd handle */ + return &pd->ibpd; +} + +/** + * pvrdma_dealloc_pd - deallocate protection domain + * @pd: the protection domain to be released + * + * @return: 0 on success, otherwise errno. + */ +int pvrdma_dealloc_pd(struct ib_pd *pd) +{ + struct pvrdma_dev *dev = to_vdev(pd->device); + union pvrdma_cmd_req req; + struct pvrdma_cmd_destroy_pd *cmd = &req.destroy_pd; + int ret; + + memset(cmd, 0, sizeof(*cmd)); + cmd->hdr.cmd = PVRDMA_CMD_DESTROY_PD; + cmd->pd_handle = to_vpd(pd)->pd_handle; + + ret = pvrdma_cmd_post(dev, &req, NULL); + if (ret) + dev_warn(&dev->pdev->dev, + "could not dealloc protection domain\n"); + + kfree(to_vpd(pd)); + atomic_dec(&dev->num_pds); + + return 0; +} + +/** + * pvrdma_create_ah - create an address handle + * @pd: the protection domain + * @ah_attr: the attributes of the AH + * + * @return: the ib_ah pointer on success, oterwise errno. + */ +struct ib_ah *pvrdma_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr) +{ + struct pvrdma_dev *dev = to_vdev(pd->device); + struct pvrdma_ah *ah; + enum rdma_link_layer ll; + + if (!(ah_attr->ah_flags & IB_AH_GRH)) + return ERR_PTR(-EINVAL); + + ll = rdma_port_get_link_layer(pd->device, ah_attr->port_num); + + if (ll != IB_LINK_LAYER_ETHERNET || + rdma_is_multicast_addr((struct in6_addr *)ah_attr->grh.dgid.raw)) + return ERR_PTR(-EINVAL); + + if (!atomic_add_unless(&dev->num_ahs, 1, dev->dsr->caps.max_ah)) + return ERR_PTR(-EINVAL); + + ah = kzalloc(sizeof(*ah), GFP_KERNEL); + if (!ah) { + atomic_dec(&dev->num_ahs); + return ERR_PTR(-ENOMEM); + } + + ah->av.port_pd = to_vpd(pd)->pd_handle | (ah_attr->port_num << 24); + ah->av.src_path_bits = ah_attr->src_path_bits; + ah->av.src_path_bits |= 0x80; + ah->av.gid_index = ah_attr->grh.sgid_index; + ah->av.hop_limit = ah_attr->grh.hop_limit; + ah->av.sl_tclass_flowlabel = (ah_attr->grh.traffic_class << 20) | + ah_attr->grh.flow_label; + memcpy(ah->av.dgid, ah_attr->grh.dgid.raw, 16); + memcpy(ah->av.dmac, ah_attr->dmac, 6); + + ah->ibah.device = pd->device; + ah->ibah.pd = pd; + ah->ibah.uobject = NULL; + + return &ah->ibah; +} + +/** + * pvrdma_destroy_ah - destroy an address handle + * @ah: the address handle to destroyed + * + * @return: 0 on success. + */ +int pvrdma_destroy_ah(struct ib_ah *ah) +{ + struct pvrdma_dev *dev = to_vdev(ah->device); + + kfree(to_vah(ah)); + atomic_dec(&dev->num_ahs); + + return 0; +} + diff --git a/drivers/infiniband/hw/pvrdma/pvrdma_verbs.h b/drivers/infiniband/hw/pvrdma/pvrdma_verbs.h new file mode 100644 index 0000000..e52fbe5 --- /dev/null +++ b/drivers/infiniband/hw/pvrdma/pvrdma_verbs.h @@ -0,0 +1,108 @@ +/* + * Copyright (c) 2012-2016 VMware, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of EITHER the GNU General Public License + * version 2 as published by the Free Software Foundation or the BSD + * 2-Clause License. This program is distributed in the hope that it + * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED + * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + * See the GNU General Public License version 2 for more details at + * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html. + * + * You should have received a copy of the GNU General Public License + * along with this program available in the file COPYING in the main + * directory of this source tree. + * + * The BSD 2-Clause License + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef __PVRDMA_VERBS_H__ +#define __PVRDMA_VERBS_H__ + +int pvrdma_query_device(struct ib_device *ibdev, + struct ib_device_attr *props, + struct ib_udata *udata); +int pvrdma_query_port(struct ib_device *ibdev, u8 port, + struct ib_port_attr *props); +int pvrdma_query_gid(struct ib_device *ibdev, u8 port, + int index, union ib_gid *gid); +int pvrdma_query_pkey(struct ib_device *ibdev, u8 port, + u16 index, u16 *pkey); +enum rdma_link_layer pvrdma_port_link_layer(struct ib_device *ibdev, + u8 port); +int pvrdma_modify_device(struct ib_device *ibdev, int mask, + struct ib_device_modify *props); +int pvrdma_modify_port(struct ib_device *ibdev, u8 port, + int mask, struct ib_port_modify *props); +int pvrdma_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); +struct ib_ucontext *pvrdma_alloc_ucontext(struct ib_device *ibdev, + struct ib_udata *udata); +int pvrdma_dealloc_ucontext(struct ib_ucontext *context); +struct ib_pd *pvrdma_alloc_pd(struct ib_device *ibdev, + struct ib_ucontext *context, + struct ib_udata *udata); +int pvrdma_dealloc_pd(struct ib_pd *ibpd); +struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc); +struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, + u64 virt_addr, int access_flags, + struct ib_udata *udata); +int pvrdma_dereg_mr(struct ib_mr *mr); +struct ib_mr *pvrdma_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type, + u32 max_num_sg); +int pvrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, + int sg_nents, unsigned int *sg_offset); +int pvrdma_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period); +int pvrdma_resize_cq(struct ib_cq *ibcq, int entries, + struct ib_udata *udata); +struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev, + const struct ib_cq_init_attr *attr, + struct ib_ucontext *context, + struct ib_udata *udata); +int pvrdma_resize_cq(struct ib_cq *ibcq, int entries, + struct ib_udata *udata); +int pvrdma_destroy_cq(struct ib_cq *cq); +int pvrdma_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc); +int pvrdma_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify_flags flags); +struct ib_ah *pvrdma_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr); +int pvrdma_destroy_ah(struct ib_ah *ah); +struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, + struct ib_qp_init_attr *init_attr, + struct ib_udata *udata); +int pvrdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, + int attr_mask, struct ib_udata *udata); +int pvrdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr, + int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr); +int pvrdma_destroy_qp(struct ib_qp *qp); +int pvrdma_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, + struct ib_send_wr **bad_wr); +int pvrdma_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr, + struct ib_recv_wr **bad_wr); + +#endif /* __PVRDMA_VERBS_H__ */