From patchwork Thu Nov 3 23:44:32 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adit Ranadive X-Patchwork-Id: 9411641 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D38F860585 for ; Thu, 3 Nov 2016 23:49:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C50F12AF86 for ; Thu, 3 Nov 2016 23:49:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B83BB2AFBF; Thu, 3 Nov 2016 23:49:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB4AB2AF86 for ; Thu, 3 Nov 2016 23:49:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933483AbcKCXtN (ORCPT ); Thu, 3 Nov 2016 19:49:13 -0400 Received: from ex13-edg-ou-002.vmware.com ([208.91.0.190]:24645 "EHLO EX13-EDG-OU-002.vmware.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933504AbcKCXtL (ORCPT ); Thu, 3 Nov 2016 19:49:11 -0400 Received: from sc9-mailhost1.vmware.com (10.113.161.71) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Thu, 3 Nov 2016 16:49:04 -0700 Received: from EX13-CAS-005.vmware.com (ex13-cas-005.vmware.com [10.113.191.55]) by sc9-mailhost1.vmware.com (Postfix) with ESMTP id 1C6F6184B8; Thu, 3 Nov 2016 16:49:11 -0700 (PDT) Received: from EX13-CAS-003.vmware.com (10.113.191.53) by EX13-MBX-006.vmware.com (10.113.191.26) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Thu, 3 Nov 2016 16:48:28 -0700 Received: from NAM02-CY1-obe.outbound.protection.outlook.com (10.113.170.11) by EX13-CAS-003.vmware.com (10.113.191.53) with Microsoft SMTP Server (TLS) id 15.0.1156.6 via Frontend Transport; Thu, 3 Nov 2016 16:48:28 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=onevmw.onmicrosoft.com; s=selector1-vmware-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=rIc3uQklSa6a3/5vvReX34A21MsrATWlvOs4lSjJte8=; b=JU1U0dKUdAWus30dUkeIrSqUzMt42CtU4KdTcQGi6mVML6rSyva3FS6rXc6oq5laVuBV0ugE70ELfSm7rrCJc6l4mfYKbBJg7M3LwxfbnozDWF2BfL9AAri9NT/csQ39tXcDm3ARrBdr/R3k3osM45AB0RjVXmMYavvnrZc6iI8= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=aditr@vmware.com; Received: from gandalf.eng.vmware.com (208.91.1.34) by BLUPR0501MB836.namprd05.prod.outlook.com (10.141.251.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.1; Thu, 3 Nov 2016 23:48:22 +0000 From: Adit Ranadive To: , , CC: Adit Ranadive Subject: [PATCH 3/8] libpvrdma: Add completion queue functions Date: Thu, 3 Nov 2016 16:44:32 -0700 Message-ID: <1478216677-6150-4-git-send-email-aditr@vmware.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1478216677-6150-1-git-send-email-aditr@vmware.com> References: <1478216677-6150-1-git-send-email-aditr@vmware.com> MIME-Version: 1.0 X-Originating-IP: [208.91.1.34] X-ClientProxiedBy: BY2PR04CA052.namprd04.prod.outlook.com (10.141.249.170) To BLUPR0501MB836.namprd05.prod.outlook.com (10.141.251.150) X-MS-Office365-Filtering-Correlation-Id: 0a20f2a7-90e0-4212-ce88-08d40443e5c7 X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB836; 2:Vs335ZJUsinWFdON3RweGNYEHtvT9K3k6SzYcjO+7RELOVsYSBM3mktoRqWU26RUzfmBYAAmFp4einUfxkTGfObS9iihbUco4Yq078Klr9fRtHS0IFjlfm1+8G51A6aaSLQgnIT/FdTz3BU4EmVT7TI+TWO59BbLXq1DxTRML0LSkkCLy3jtW3s9U+RssW/GXt+7wbs6HGaCWgfHTkfB5A==; 3:WAtL0DObl2tWFX69+0Rzvkel25vczToLn4NEMIYGiudYcOAiQ16VOAkRkQqjrmRsUiDCpLVKP9V39p9S90MvYIgjWKjTXcc4YQvqetW26+a2ch1rCyp+XpTgqONrvOf3z3DUu4F1WPSsL+fkqYGkmQ== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0501MB836; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB836; 25:ba9KP7exe5BUKimsfWexsFsaVgZs/hUZ6RF+TW8yY2+kKMtCGBI8NxGK1sV3xUPJv1aUkbY1Pye6FuL7NgOhTIjxQUN2Afkqcs4xYcWbu5sKr9pIeX1+HHQET1bgbnKSIa1QXtXlvY3y7zy0WZifs/gDo57s+ccosur3VIqcnLHNhYjpH0OX+aMkfRJIqMlsAkKMgPvhe3TCFpu59z/dFBxwKdgh+uFju+diu6LIG4NLdiVCzj+Z6ZNxmebKtCrR3XYyFfa+RfSSGThOGxixhhlvoxNgFjg8o+p2ejcP8mqa49u3guHt3WmxjIdUgWZmCT/dGJxtYnQtvhbCJSsr8Klg/mtIY6PMD+obCaKsOGZC4741qOWI9jx7V9mbqzjrImWpTf7Vi61pOtkf0Q5dleM5aU+PgBdMiCA6o3nPCuMOYaw0FlHxELl2yI+lRPEuSgEru1UNGM4Kqtwf2V5geVbZ6VH9xbSr6lIUtTLAIOVsGA+4022HkIClx1sR2CvElH3EA7IYAR7SWrPgbajSBza6XesSCH8bNGb3KATa6VqbafFc5Zmg8a4bH48jhk+xBMTbpuKXRiVttCub+LujrDJgvf8MJD3kuPZzRNLce75f5odV0t0HA8CO5WNh37CmrxL/umQIUA9dNcxfKnkLPy7GJF+scLik7G+WJJcgKTw6IRiR23ujd91GS76eTMOuiRtdK+LPH13WrtWsHWPYI5xZywaU9zB1iz5WSB5uLPkrMxn6v8sOgGV5hiXJisjCvIF5re813J5teKTbBVthTn3xZJr2yHBAQpmTgP6kk470gdH6oR7d1ss3GqmR2YcOiR2OYGO5yehUrAHQxSa2gA== X-LD-Processed: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB836; 31:XhqR+KWD7actkKnccVch2w0HB2HT8qcP2y5pon+1YuHo5DWlnm/lAVcwwu94Jy0vci/h/viMPChM9PHUxaTViit2LHETyIYpXnZDJCSFfNVv6IWGeyGpKjm3X0QeBNLHLaboeG5i0ejHincL4hkKUZxV8hSOV9/cdlTL48cE7J/thp98gzPA/7ftIoE6KivnP/WNHWsgPWa9zWLHjrWYMzbQ8gsWDLoSiWkp5PXyFQQXU8L31u61k5gMTP1Ub0HanCqRP2Qzu4vVlIplnPwaAA==; 20:WPfutDogIzok1/1uPYqHiOh/Lm3RNERG2Fxpj/AAQQvRaKg+3oEsAiLVRoQg095OE94sF7IlkwpjSI7ARvvtj440JPNGmoZItphMlFJ8obOwhdgeOJMCX/BBXDN1gEwUXhUVkowyjCaa6W5KBvH5GnRtALfBfh0eV1HP5vqr1fRYq0W9ybRFHkTB3xNpdyZtSh81wMYf/8HHdFp+5zzrh6kZuB2hVqOJX0Q+pkHXLIvVPjO0H8eqww2ZCWuXJe1Bw/k+GjfqwxEmSP0oG8bLi56bdfXQ+ZYa5Wn4duuBGCTpu8pk1pEQ4snLc+mjl9xHBOxAOJVuz3QbFVWAE2EOrQA2z4Ym1oy2sqcrvbFhTdNlZNEXrD9c10iZDpl0Red+00dnpmgN+Kn9GjCS5LX8+T6MgbH3NYxHWKC283pkR8jJtdg1zIuiNt+uOG/p3n8sJpqdtERG1vQPllsJQCjGEoVGLAg8EU7LtiiTb3QlBNouVRYhcZsgTYzTSCzyQK67 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(250305191791016)(61668805478150)(22074186197030); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BLUPR0501MB836; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0501MB836; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB836; 4:JVds1dBhsk3CxE+gV5vQuZkHzBWx3vi3dMxS/OObFVZ0wphfClyMPtqIQHbuquj/kHQiprpuDF4kuAavqsY9o2J1OPaqgyH8aqhkbSeYk0vdKiso+5O4ps/1U/FR6OpsoLJeQI4H/385QocU2W0eYZ1YdjKJt2OUEEnkm9qzSxQ1AWhGrIP9e3BRE4wjQQRIv6dv0um5AICdauK2o0YnuR7zIX25IEowi3jN2bL7jGAqW524Ve87rXlQcm0FqzaIGGMBTJdH472KeIZrgK0XTdXU1XV8cpfdL/IYRbWnsn3vmG5m40sXFsBIzEW6Pk05zIF+IKxKvGblMAVarb+fr1wVeRoBY43ViObnzGCWhl5SsQm/yvC1wxT4xnnTmbFyJY/UXdU2ZeVCWvtCXNmJnfu6iBUvmKqMLC86CgSFr9H+4Jr0tepFRwJWGCFO+O5EI49bISRq9Gk/+halP2XTCjzx8aRiiFpZn7CuadoZ0/l/8GMwnyM0rEdj0OiFdLXxnjX6ew7yjn3e9CMy8xdwow== X-Forefront-PRVS: 011579F31F X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(189002)(15975445007)(8676002)(6116002)(5003940100001)(106356001)(5001770100001)(97736004)(3846002)(586003)(107886002)(19580395003)(19580405001)(81156014)(81166006)(189998001)(66066001)(47776003)(4001430100002)(50226002)(7846002)(92566002)(7736002)(305945005)(2201001)(101416001)(6666003)(42186005)(50986999)(6636002)(86362001)(77096005)(6862003)(48376002)(50466002)(105586002)(76176999)(2950100002)(2906002)(4326007)(68736007)(36756003)(5660300001)(229853001)(33646002)(7099028)(2004002); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0501MB836; H:gandalf.eng.vmware.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (EX13-EDG-OU-002.vmware.com: aditr@vmware.com does not designate permitted sender hosts) Received-SPF: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0501MB836; 23:7DCdpJZc17xP7kK2MLENtlibbDNN/xujc/iel6uB?= =?us-ascii?Q?tI009LQrAyfNB6Ufxx04Z4a4XZT8LsbVbkTA+cD4n3hqZA6e4kjv5xZbSdtt?= =?us-ascii?Q?G7bC4eEfqGD8sncOj6kkfhaAB2CsmyMxnm1QnB9tt1+iyPsm1un86NrIfkeb?= =?us-ascii?Q?iRCOkZRpWBaD+iO6MqYTx7shksT4zjVMVnAi21ayOCZrAdT6qaDpcylQUQac?= =?us-ascii?Q?q/HGZ/bYvM/Csbzev7gGH7nTaFRhPimLO3syZflhCe23fviEwHGskPs47wus?= =?us-ascii?Q?z0uhw44AT73pdzuR6Lpl+ASJPL09aWw5ysCKpDs9kDTYxlSbBnR3cUp0oEZ9?= =?us-ascii?Q?i0kKUT2xSDWjvsgXi/jeiMtW7mUSzLgV6/E82818fI4/068Kqm7Ef0K+bTjN?= =?us-ascii?Q?cmI4QfV1n6/8WgdAYcFvyraHq7PKwbr+NMWp9Q6GC1jDfmNxN+nFcZQWUzfM?= =?us-ascii?Q?uxY4X1YKw9HGzLlV414tg+uXuAICYolpDM5lau+j2Q5GH+oYTpnHgwypYeNk?= =?us-ascii?Q?aJVuLSJXiuhDk1zMQqlOEULgBjnbgr9RdVRXVuOv2bfnrb8K9rbKRUQfSTjF?= =?us-ascii?Q?QB/SOBuOrmMwAzxF9ebImrZkOBHclTNJtkbq3ZWiGsXv5Z9oornsMKYSev7h?= =?us-ascii?Q?JZ8OpbtYrOG3gk5HLUnfQGw8pawpLqDG2ZcIO0BEwKhGW/It4Vo9yWRrOufJ?= =?us-ascii?Q?rhAich/d3XpcRXiktkeTUWZKTBt6i9fyqeI6dlA81If8latlCONxiX/I2KBi?= =?us-ascii?Q?DSJlTxasr0FdO3StuWc2A2V7yhlogMbiHlKN8yitF1qTOsxu9+Le0/vNTS3m?= =?us-ascii?Q?Wos5xF3Li1Nb3VuVnQ2+D90xipltxCLrwgnDtwV/n7KhwWdfm5lKOWJgzj+G?= =?us-ascii?Q?Ds8HwvKZrvstuNNm6CGo9CFlCGmvrXCS04D3m4fpcUpXAHVXTzPCCursXC5e?= =?us-ascii?Q?c4fFnz6wgshs06wNQQ1vvtbqZsP8Tg4+oleydzU6Xp5wuqKKHAKby/CJoRLq?= =?us-ascii?Q?3pbi5TM745HFWKllZGXwPm+ZPxiNdl49LfWxZW0aU7bWEtm378HkJGs5AXuA?= =?us-ascii?Q?zE8/S4D96mppdDnzppJKIZpHqfoPj8TtA9M72nBOhc3I8QNA72JBp18UkjpA?= =?us-ascii?Q?H6g5TeTITEnIGVuUivgRKsgZHVpqQAIv8uGynOH62l+R9CN2t//AavkB7yUw?= =?us-ascii?Q?h1ahksiGDrXS9+9yRT6hSGSe8d5nv/WS/j7SMWLflLLI4DjmG8XpnTsNws3B?= =?us-ascii?Q?4lVCoXt77CmrtVwJ66Y=3D?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB836; 6:IxB5aiqPDFmcYbzuA14gvoPCiuH5Z2rCLYb59CDZKdlbMFTgUu+o/Apn6iebvUPTZevXFC4BuMjC1xeoaqpl84zWds65R626GgV5LAU1Zr8nprItxvRUIHNMlNsvRvJW6YuThX2Js8AwY/tQfUd7WPCLvj287T+r4SqU+8t1uT7IRD6AQN1E9QxqMp0iK8DdwQTpPE5PI7oGjITQBjkjBdVn48RllT4vHNTpSbs47WlNBrqmR3HbTvTMXL+uC7dRtUtKdJN70gzmEuRCzpO+2oYBC0Lj3OlYnTazn+aa7HPq5FjiunLrBxTEZ2jR09x7; 5:hS16amJcfIWxgJK9OgCeF9MFzSg6ISiXsuhBFRdpNVLzrt8DwnfydjMULBEUGx2UU4ljyMdXkqcwmaKaQW0gAJB+Ks3f0+8Wzm0uSlRtH8SCrSwCy619PhmS9PU0KcwIGhS0QwKjRFOk5icF67yIcrKKKs1YDk4mmRAwjfOHiFU=; 24:9AvaeZoDkGJCTXjDkCRS/jSkrtwWt3hf3kB9zLIz4l9sOU5QffkURm+mIgxN8yAcplk1xiQE8xLzdQ3/GMPdUH3/7pZmTgKO6kBN+EMF9Jg= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BLUPR0501MB836; 7:BnSArz3w6LM3tog60SVuci4A13hVGr50Qc9GYxxJVHUclUd3GvpqP3f9LvDgEdnX4QPJZizpyj+F+Usg+h73RxbUm5r3uhF6/jOV8OOvjVMAgTkDHx4JzyizeP3U6Ovl52h8hkSvSmiy87WAVHO146xT0d6xIxVM2wwoQZ3HzV0odRMdllcnOoltv0fbMvez55IXWILeD6qRorbt1cPwfanxVjkFNEjN2csO/mwpkJozqf4HbhpJsCtpVEN+6YJjVD95xoMIw8ZdESVZQVMiDziDl3gKln7a1nUHHBH5i6ExH4EunANgNURX+44mVEY8qdutlf/SEXSv8DSDP1ub0ZutA57BO+iAPSVyZjIexsc=; 20:sdzR3TUgs3rC2KY04XwGnsghg383j0ETRmS2a+X5eVpTc1W7bSCXBFx2Khn/rvojh8VEVq3no97m4tD6lwTc4/zvt/akulvKDJZnKixYpTILnMWWcUL7evTGEs3gRO9BHILUO6G6DsAg1dnSl/aR4VXOdY2NK8+ChNzo5gAgIbw= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2016 23:48:22.4614 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0501MB836 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add support for completion queue creation, destruction, polling and events. Signed-off-by: Adit Ranadive --- providers/pvrdma/cq.c | 287 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 providers/pvrdma/cq.c diff --git a/providers/pvrdma/cq.c b/providers/pvrdma/cq.c new file mode 100644 index 0000000..f99873c --- /dev/null +++ b/providers/pvrdma/cq.c @@ -0,0 +1,287 @@ +/* + * Copyright (c) 2012-2016 VMware, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of EITHER the GNU General Public License + * version 2 as published by the Free Software Foundation or the BSD + * 2-Clause License. This program is distributed in the hope that it + * will be useful, but WITHOUT ANY WARRANTY; WITHOUT EVEN THE IMPLIED + * WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + * See the GNU General Public License version 2 for more details at + * http://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html. + * + * You should have received a copy of the GNU General Public License + * along with this program available in the file COPYING in the main + * directory of this source tree. + * + * The BSD 2-Clause License + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include + +#include "pvrdma.h" + +enum { + CQ_OK = 0, + CQ_EMPTY = -1, + CQ_POLL_ERR = -2, +}; + +enum { + PVRDMA_CQE_IS_SEND_MASK = 0x40, + PVRDMA_CQE_OPCODE_MASK = 0x1f, +}; + +int pvrdma_alloc_cq_buf(struct pvrdma_device *dev, struct pvrdma_cq *cq, + struct pvrdma_buf *buf, int entries) +{ + if (pvrdma_alloc_buf(buf, cq->offset + + entries * (sizeof(struct pvrdma_cqe)), + dev->page_size)) + return -1; + memset(buf->buf, 0, buf->length); + + return 0; +} + +static struct pvrdma_cqe *get_cqe(struct pvrdma_cq *cq, int entry) +{ + return cq->buf.buf + cq->offset + + entry * (sizeof(struct pvrdma_cqe)); +} + +static int pvrdma_poll_one(struct pvrdma_cq *cq, + struct pvrdma_qp **cur_qp, + struct ibv_wc *wc) +{ + struct pvrdma_context *ctx = to_vctx(cq->ibv_cq.context); + int has_data; + unsigned int head; + int tried = 0; + struct pvrdma_cqe *cqe; + +retry: + has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx, + cq->cqe_cnt, &head); + if (has_data == 0) { + unsigned int val; + + if (tried) + return CQ_EMPTY; + + /* Pass down POLL to give physical HCA a chance to poll. */ + val = cq->cqn | PVRDMA_UAR_CQ_POLL; + pvrdma_write_uar_cq(ctx->uar, val); + + tried = 1; + goto retry; + } else if (has_data == -1) { + return CQ_POLL_ERR; + } + + cqe = get_cqe(cq, head); + if (!cqe) + return CQ_EMPTY; + + rmb(); + + if (ctx->qp_tbl[cqe->qp & 0xFFFF]) + *cur_qp = (struct pvrdma_qp *)ctx->qp_tbl[cqe->qp & 0xFFFF]; + else + return CQ_POLL_ERR; + + wc->opcode = pvrdma_wc_opcode_to_ibv(cqe->opcode); + wc->status = pvrdma_wc_status_to_ibv(cqe->status); + wc->wr_id = cqe->wr_id; + wc->qp_num = (*cur_qp)->ibv_qp.qp_num; + wc->byte_len = cqe->byte_len; + wc->imm_data = cqe->imm_data; + wc->src_qp = cqe->src_qp; + wc->wc_flags = cqe->wc_flags; + wc->pkey_index = cqe->pkey_index; + wc->slid = cqe->slid; + wc->sl = cqe->sl; + wc->dlid_path_bits = cqe->dlid_path_bits; + wc->vendor_err = 0; + + /* Update shared ring state. */ + pvrdma_idx_ring_inc(&(cq->ring_state->rx.cons_head), cq->cqe_cnt); + + return CQ_OK; +} + +int pvrdma_poll_cq(struct ibv_cq *ibcq, int num_entries, struct ibv_wc *wc) +{ + struct pvrdma_cq *cq = to_vcq(ibcq); + struct pvrdma_qp *qp; + int npolled = 0; + + if (num_entries < 1 || wc == NULL) + return 0; + + pthread_spin_lock(&cq->lock); + + for (npolled = 0; npolled < num_entries; ++npolled) { + if (pvrdma_poll_one(cq, &qp, wc + npolled) != CQ_OK) + break; + } + + pthread_spin_unlock(&cq->lock); + + return npolled; +} + +void __pvrdma_cq_clean(struct pvrdma_cq *cq, uint32_t qpn) +{ + /* Flush CQEs from specified QP */ + int has_data; + unsigned int head; + + /* Lock held */ + has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx, + cq->cqe_cnt, &head); + + if (unlikely(has_data > 0)) { + int items; + int curr; + int tail = pvrdma_idx(&cq->ring_state->rx.prod_tail, + cq->cqe_cnt); + struct pvrdma_cqe *cqe; + struct pvrdma_cqe *curr_cqe; + + items = (tail > head) ? (tail - head) : + (cq->cqe_cnt - head + tail); + curr = --tail; + while (items-- > 0) { + if (curr < 0) + curr = cq->cqe_cnt - 1; + if (tail < 0) + tail = cq->cqe_cnt - 1; + curr_cqe = get_cqe(cq, curr); + rmb(); + if ((curr_cqe->qp & 0xFFFF) != qpn) { + if (curr != tail) { + cqe = get_cqe(cq, tail); + rmb(); + *cqe = *curr_cqe; + } + tail--; + } else { + pvrdma_idx_ring_inc( + &cq->ring_state->rx.cons_head, + cq->cqe_cnt); + } + curr--; + } + } +} + +void pvrdma_cq_clean(struct pvrdma_cq *cq, uint32_t qpn) +{ + pthread_spin_lock(&cq->lock); + __pvrdma_cq_clean(cq, qpn); + pthread_spin_unlock(&cq->lock); +} + +struct ibv_cq *pvrdma_create_cq(struct ibv_context *context, int cqe, + struct ibv_comp_channel *channel, + int comp_vector) +{ + struct pvrdma_device *dev = to_vdev(context->device); + struct pvrdma_create_cq cmd; + struct pvrdma_create_cq_resp resp; + struct pvrdma_cq *cq; + int ret; + + if (cqe < 1) + return NULL; + + cq = malloc(sizeof(*cq)); + if (!cq) + return NULL; + + /* Extra page for shared ring state */ + cq->offset = dev->page_size; + + if (pthread_spin_init(&cq->lock, PTHREAD_PROCESS_PRIVATE)) + goto err; + + cqe = align_next_power2(cqe); + + if (pvrdma_alloc_cq_buf(dev, cq, &cq->buf, cqe)) + goto err; + + cq->ring_state = cq->buf.buf; + + cmd.buf_addr = (uintptr_t) cq->buf.buf; + cmd.buf_size = cq->buf.length; + ret = ibv_cmd_create_cq(context, cqe, channel, comp_vector, + &cq->ibv_cq, &cmd.ibv_cmd, sizeof(cmd), + &resp.ibv_resp, sizeof(resp)); + if (ret) + goto err_buf; + + cq->cqn = resp.cqn; + cq->cqe_cnt = cq->ibv_cq.cqe; + + return &cq->ibv_cq; + +err_buf: + pvrdma_free_buf(&cq->buf); +err: + free(cq); + + return NULL; +} + +int pvrdma_destroy_cq(struct ibv_cq *cq) +{ + int ret; + + ret = ibv_cmd_destroy_cq(cq); + if (ret) + return ret; + + pvrdma_free_buf(&to_vcq(cq)->buf); + free(to_vcq(cq)); + + return 0; +} + +int pvrdma_req_notify_cq(struct ibv_cq *ibcq, int solicited) +{ + struct pvrdma_context *ctx = to_vctx(ibcq->context); + struct pvrdma_cq *cq = to_vcq(ibcq); + unsigned int val = cq->cqn; + + val |= solicited ? PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM; + pvrdma_write_uar_cq(ctx->uar, val); + + return 0; +}