From patchwork Tue Jul 12 19:36:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adit Ranadive X-Patchwork-Id: 9226063 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B5ADA60868 for ; Tue, 12 Jul 2016 19:38:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A3E1C27F9C for ; Tue, 12 Jul 2016 19:38:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 986B327F9A; Tue, 12 Jul 2016 19:38:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 08E0227C38 for ; Tue, 12 Jul 2016 19:38:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751655AbcGLThq (ORCPT ); Tue, 12 Jul 2016 15:37:46 -0400 Received: from ex13-edg-ou-002.vmware.com ([208.91.0.190]:29980 "EHLO EX13-EDG-OU-002.vmware.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751645AbcGLTho (ORCPT ); Tue, 12 Jul 2016 15:37:44 -0400 Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Tue, 12 Jul 2016 12:37:18 -0700 Received: from EX13-CAS-008.vmware.com (smtp-inbound.vmware.com [10.113.191.58]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id C99614045A; Tue, 12 Jul 2016 12:37:38 -0700 (PDT) Received: from EX13-CAS-003.vmware.com (10.113.191.53) by EX13-MBX-003.vmware.com (10.113.191.75) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Tue, 12 Jul 2016 12:37:38 -0700 Received: from NAM02-CY1-obe.outbound.protection.outlook.com (10.113.170.11) by EX13-CAS-003.vmware.com (10.113.191.53) with Microsoft SMTP Server (TLS) id 15.0.1156.6 via Frontend Transport; Tue, 12 Jul 2016 12:37:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=onevmw.onmicrosoft.com; s=selector1-vmware-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=S7kvL51HsxaTjMp8nNweZKHN/Y4O8WVGTGJD5Uq4b9U=; b=D+dFoFKCrZxiG2kQlpmLpGiITqY7o6VpLzV7t864lGB5wbVQb5yZZ1QxZOIS5lXvsDAHhE3xXTVdXC0e70cY8v5ClxLi3R36Oya7GI20+XA2elyIxYCkaFXnGmP9MMDKdec4UtNG/TDMIiKtQexZsof86lw5fSXCtnR+QyVA76I= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=aditr@vmware.com; Received: from promb-2s-dhcp95-136.eng.vmware.com (208.91.1.34) by BLUPR0501MB834.namprd05.prod.outlook.com (10.141.251.148) with Microsoft SMTP Server (TLS) id 15.1.539.14; Tue, 12 Jul 2016 19:37:32 +0000 From: Adit Ranadive To: , , CC: Adit Ranadive , , , , Subject: [PATCH v2 05/15] IB/pvrdma: Add UAR support Date: Tue, 12 Jul 2016 12:36:35 -0700 Message-ID: <1468352205-9137-6-git-send-email-aditr@vmware.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1468352205-9137-1-git-send-email-aditr@vmware.com> References: <1468352205-9137-1-git-send-email-aditr@vmware.com> MIME-Version: 1.0 X-Originating-IP: [208.91.1.34] X-ClientProxiedBy: SN1PR10CA0055.namprd10.prod.outlook.com (10.164.10.151) To BLUPR0501MB834.namprd05.prod.outlook.com (10.141.251.148) X-MS-Office365-Filtering-Correlation-Id: 7b0e446c-4562-40c3-6918-08d3aa8bf86f X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB834; 2:JOw1nU5Ueekx4xNY5+vDrCqjSncRN6L9Cb2m0cNDqwIjC/Aefv4CHxak6ozj//2CiboMSD5AmXlXtz3Az1NLcv2V9Zzl+oCrgngUjB0RUVWgstZOA+8HvtxkV36WhyaGkd3mPVijPzrf3hBqG9BAlJRvqtRaVq2l7gfZSGlAcjVtMWRbpTcnvUvgW7b3qPRM; 3:KDhBTrTWO9KPTU4gwgS9GxfWsUg94bAYwXoPW0H7isMbxXku4mJdKrnKf4wPdSlt0+QTQmioKVUY9X8ZtjO4v7hdPfyYDbRO2GRyfpuo9pTwzG04ocRw7LUExJ4/Kn92; 25:4FQb4z+b8CyUVhNWD8ELjxHoLrNdRY2AM8e1jxY/bBXZh5QFw6QEoW+z31GNM/L6RupHkODArSg8DRfAt+FW/KR/vZt70PvlHADpoe7I8N1M5+3nMUJaOgpUPmQCHEHxDFzNo7Np+02/0BVgM2AYASBJkY5WFH7tB9t64eLdSkmwijFItWl+B3opJKRngkNWYAbbBsAhekGWYv2GiaCHLd0GAho6jSe+Jq7HmpoxdBom0c9maciUKgajcKiPkGyTIy2ghxtXSpKjzUilPa9wixTYT/eVnE9gKl28cj10dL4uG05I8PKNRfG2oGlYNpyp2Zg78eN4GS6AAtilpX0pJWlkI+wDg/MDIdNSkI5ZjRoqoq6t3jncudku/tcffvqb/q42XkBh+65WpnpLaLiURVTGZs6raVs6I6CKNuopagc= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0501MB834; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB834; 31:rwYANXS688v6lILVk+4UIjFKegeOT+eNZjCbVUIv3eYHgGr9zzETqdmwJLtb98KolBeaJZITEX6rU52S2If8ryNQ2mymNQUOXNuJRbYBxEXEowBrULnhN/uxtDl5IuA+L7F24gROvNw81H8TVKfsyCvRoej7i3470Y8uC9xKd+w5RVFaoBCJW+yf+GoOJnnLvDGAJEEQhYoOJzbpNm3AAA==; 20:8xTQoF5hyXYzh5/RAfJUjHvOtL8fYnJEoCMEdMs4iTM1F3jqMbW+gPr/7nmaNJUIhDKaubTkq4PM1lYyVt2az9MTfZGHkfECA8jum2o6w5bxpt6VZcqudW6T5kQ3pqLU9jz5YWh+MjxAT+i48QKqHPUjTANOthaDK4CnonpvQb3SrPg4D4NBgvGPZEjc+0Keoaw1urxQOSZa5dZ1Rfl5vQbXcbwyAj80yMsvbv3FVRUQE68evx3tiu0Vhu+t3DU2; 4:W936zbyPw3lBxvgUqi14C6YELNbwl1RJkXUmBWJV1JCehR4fQwV58Q4tHI/RhAKtd2UVnGg4K1WnQd1rRj+nAYwkZAPl2NJ5lkgL4pY5DhJxAOhFbgDBpQjvatUuyMyW9pNKTk2gz3bVQsEqoiGvMPtpybgo4Qbr6jVnpqFtOFEUulZUT0T3E1nv5tBHzzwxkZlY/qfacjaLNfAy9psIsNH8T4JeOd8vQrTSqGeDyyr8S4ro9qlFTuycJ/OwSiZXuWzH7Waudg65lVpoKWcqyj5B8yvmc1G8Skx/o7sA6Cwua/HfTFflva+fO3qxgZy6IHVv2XTMfO/xvx+GzJ/3G45ISbFrVznVn6groAts34iF8BGbIcz4Y71RKba9YLztV2+p9x7/q7DsXIp76h9XsKnXUkziCrJ8xplHkQVEdOGGup1CEpf1nB77vS5IYdqa+eRdjU8IW 2ekmeavYPCfr9II2NDxwL0cHxrKRyK1t4I= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(250305191791016)(61668805478150)(22074186197030); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001); SRVR:BLUPR0501MB834; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0501MB834; X-Forefront-PRVS: 0001227049 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(189002)(305945005)(7736002)(50226002)(86362001)(4326007)(2906002)(42186005)(7846002)(68736007)(33646002)(229853001)(106356001)(50986999)(586003)(3846002)(2201001)(6116002)(15975445007)(97736004)(81166006)(81156014)(48376002)(36756003)(5003940100001)(19580395003)(19580405001)(47776003)(2950100001)(77096005)(66066001)(50466002)(4001450100002)(4001430100002)(101416001)(92566002)(189998001)(76176999)(105586002)(8676002)(107886002)(5001770100001)(7099028)(2004002); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0501MB834; H:promb-2s-dhcp95-136.eng.vmware.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (EX13-EDG-OU-002.vmware.com: aditr@vmware.com does not designate permitted sender hosts) Received-SPF: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0501MB834; 23:LAfC2vG91sioHrgYK7SEgH0TjlwyhLVz3ZraEEW2?= =?us-ascii?Q?SqX05S0gzF8vKXhjeZFdeWWNszi93v+dMYpLqmQ0v54NjeM2Imevn/tu5ASf?= =?us-ascii?Q?z9G+Yzs4YLeH3LRp/VFcHtRIGlYPa1VGX/3UTSgE8jJLry0t1rmSKdjrQmNZ?= =?us-ascii?Q?16EuRnd4VQk6PV3PHl3RIFsp1e9eRcZ85eztj1Yx+tk7yYlNuMVVlwWKcvbz?= =?us-ascii?Q?hbYAHgpKpQ2SJauAy+9x+JGA1hN3IBl5/vLl0GUq/8Q2cjgR5pqgP5gYECvM?= =?us-ascii?Q?c1CZJg+pt9aCjpBJzm+01EkOUyFu8LdhR0N90yetlR8qlC33740wJQiacIho?= =?us-ascii?Q?070J3xB6F9Z0AC86XZNIfDHy1ckhlLFEiXi9A0Rnkz17J2EQe4mBkwQyqX13?= =?us-ascii?Q?9p4IBzZC4DulNWqGgR4ZcWvt2c/upNaNKoWDkSgLFg2KmZX5YcokXwXe/gqj?= =?us-ascii?Q?7KiilIlwQyOGnasl1k3Ug57q/EK0EeShdtaK1ErsRa31VgsnvvKg4W1j55oH?= =?us-ascii?Q?0gnD378mpQ2gP0tgSMdEBays7huMByIBFGSqFLAJLqtPWIAdR+Rq4rF912aE?= =?us-ascii?Q?5r6vUoQFtEut+6Dl949gSaEikqf+42b4Th4GdPvs4t0Ke4VDyVHbuMrKTicR?= =?us-ascii?Q?Y+9FeBOLfIYieHf6m5ywWVEzo4EYWfgF7ZOgIUqXvwP55qQ9er7XbklODiqN?= =?us-ascii?Q?MIMe50OLP8zN40RYFjyAGqROV2tyD4HWohoFWLnk0YvpVXwFMzffjKvWTNqJ?= =?us-ascii?Q?Un/QywNyBMLiAWCyEXXTcEITSGsewc0m2CDlU2X9UlQ+Gma+BtNH6nMxdIeK?= =?us-ascii?Q?VSwQfBa8BtnbVznE77AWnNcWBjb0axGFAH/4rHv6frzsIPI9VpUAkaor5vIA?= =?us-ascii?Q?s6bhxVNXtDv5/1MEoX2W/7DWNR21eavNm5iNL4xS5hAKKhluZU+l8F7yo7Pt?= =?us-ascii?Q?I818X0D59vF3qySBhbtTZbWPihqqGti7zxbhnXEcB2hy/0FqmEtA9aIgYeEW?= =?us-ascii?Q?6oNRbTDO2vnwHW2pVe0fG0BXpb7HIr2n4gwPz7nQHPZ0OcQvHySYSG2Lg4Mk?= =?us-ascii?Q?afDRi8Il6IWxe3Ynk+6nHjng4BsKFCXMb5NlYpAmwJApOXbecSTLDRbsq8MH?= =?us-ascii?Q?In9M6I8yrqYPKUC/WHsFWnR8cc4tR0kEjzJ/ldVtIuaCKkxlrsdNVmnbieys?= =?us-ascii?Q?BwccJyjAg17RWiwiZvLxrP0XstMp/2IFy+l2aLHgiWWLJRtVCLIphB8iDw?= =?us-ascii?Q?=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB834; 6:mqCzl13nqRlHFG/0kRt2ytkAshdX+f/sPv0D0Ivqt98iERcrnuSnSJ86I1gKqsCIUctqcLtIvHjeKQuKRr+NgpzOQ7YSuy0NysaqYiiyWn79sDsNtt3spVDEpCbMgqvsRGXIh0wGjbKOaSLgHp1m3lJFBS+GSKCs+vb2sqjRH5lFRowYqzT966zxY86yzMOpM6sKtFAQentr7Ss50P+9hKQfi+zCoAWrnc6/Ks/xYKnvvHkXqz2K9M4erYnH8xbbEQkaAsiguWetfjtrx6Vxf9xgf6Z7rlLIBSZL5ioxE+w=; 5:nkXrzj4vziQc8UNFcEQ3tSOaABcxOt0q/R+Pw4WK+4jIaAFdb/Iojds3bnh1qMbCcswFTKAMmsp44/P/SS6pIbvvBQoKRavshpPCqe0bFqi2my2arcPMwSRGi2ws59qtxbHJ3ILETjFYYj1SLyjsSw==; 24:9KN2Vhvq0efYyACbkjnQOE58aWb7yNkTIG8ufmv1ZKG06vWo+abhHC0l3BvRceOPr9oPBOhRFUxtwoSgnvCwWdi5uUmFF02vM9K/o8H6Lds=; 7:TwtFG+0mmqNY2IB8XFkpDvajXFYizdxA7Mib8/UPvaswhCEhWZn71OMY6DZB3gKkwT91Xev0zCM2hc+KmRgfH6hdyAnFMbTbvkQavgSUyKw/9bjxrttQX+6rKUCBSq9r5XHJgiA+/sbMzpyB4mDzC6VnUkK1szB2koWx8UuMptxjqIXfNUc9k85+NQZECIygHcgezykQwkzGQU56fPno3PZWoGBLR05tOutKKhA6iBg1RiJk1WXTSxBVbQqi1Yv5 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB834; 20:Qa6PNzZHmfkiNFjmr4Ru6pIQfcl2U9v/mJP/T9lB45L/tpvIDVK1bU5cKZNsBrUfj5Nktb2QNjh1jiiHAXDZXLv92fHp7UdYGh6KiGxf+vECSw0D/4lWtrr1rCK7IniDFK5M+vzhzc+AssI2ikWEbAoANEY/BWiwLSKRboeU1+w= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2016 19:37:32.4280 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0501MB834 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch adds the UAR support for the paravirtual RDMA device. The UAR pages are MMIO pages from the virtual PCI space. We define offsets within this page to provide the fast data-path operations. Reviewed-by: Jorgen Hansen Reviewed-by: George Zhang Reviewed-by: Aditya Sarwade Reviewed-by: Bryan Tan Signed-off-by: Adit Ranadive --- drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c | 128 +++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c diff --git a/drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c b/drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c new file mode 100644 index 0000000..d854f6d --- /dev/null +++ b/drivers/infiniband/hw/pvrdma/pvrdma_doorbell.c @@ -0,0 +1,128 @@ +/* + * Copyright (c) 2012-2016 VMware, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of EITHER the GNU General Public License + * version 2 as published by the Free Software Foundation or the BSD + * 2-Clause License. This program is distributed in the hope that it + * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED + * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + * See the GNU General Public License version 2 for more details at + * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html. + * + * You should have received a copy of the GNU General Public License + * along with this program available in the file COPYING in the main + * directory of this source tree. + * + * The BSD 2-Clause License + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include + +#include "pvrdma.h" + +int pvrdma_uar_table_init(struct pvrdma_dev *dev) +{ + u32 num = dev->dsr->caps.max_uar; + u32 mask = num - 1; + struct pvrdma_id_table *tbl = &dev->uar_table.tbl; + + /* Power of 2 */ + if (num != 1 << (ffs(num) - 1)) + return -EINVAL; + + tbl->last = 0; + tbl->top = 0; + tbl->max = num; + tbl->mask = mask; + spin_lock_init(&tbl->lock); + tbl->table = kcalloc(BITS_TO_LONGS(num), sizeof(long), GFP_KERNEL); + if (!tbl->table) + return -ENOMEM; + + /* 0th UAR is taken by the device. */ + set_bit(0, tbl->table); + + return 0; +} + +void pvrdma_uar_table_cleanup(struct pvrdma_dev *dev) +{ + struct pvrdma_id_table *tbl = &dev->uar_table.tbl; + + kfree(tbl->table); +} + +int pvrdma_uar_alloc(struct pvrdma_dev *dev, struct pvrdma_uar_map *uar) +{ + struct pvrdma_id_table *tbl; + unsigned long flags; + u32 obj; + + tbl = &dev->uar_table.tbl; + + spin_lock_irqsave(&tbl->lock, flags); + obj = find_next_zero_bit(tbl->table, tbl->max, tbl->last); + if (obj >= tbl->max) { + tbl->top = (tbl->top + tbl->max) & tbl->mask; + obj = find_first_zero_bit(tbl->table, tbl->max); + } + + if (obj < tbl->max) { + set_bit(obj, tbl->table); + obj |= tbl->top; + } else { + spin_unlock_irqrestore(&tbl->lock, flags); + return -ENOMEM; + } + + spin_unlock_irqrestore(&tbl->lock, flags); + + uar->index = obj; + uar->pfn = (pci_resource_start(dev->pdev, PVRDMA_PCI_RESOURCE_UAR) >> + PAGE_SHIFT) + uar->index; + + return 0; +} + +void pvrdma_uar_free(struct pvrdma_dev *dev, struct pvrdma_uar_map *uar) +{ + struct pvrdma_id_table *tbl = &dev->uar_table.tbl; + unsigned long flags; + u32 obj; + + obj = uar->index & (tbl->max - 1); + spin_lock_irqsave(&tbl->lock, flags); + clear_bit(obj, tbl->table); + tbl->last = min(tbl->last, obj); + tbl->top = (tbl->top + tbl->max) & tbl->mask; + spin_unlock_irqrestore(&tbl->lock, flags); +}