From patchwork Wed Aug 28 06:09:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng Xu X-Patchwork-Id: 13780694 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21ECC14B945 for ; Wed, 28 Aug 2024 06:09:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825392; cv=none; b=gb6aNMn/VePiw7jxmmrA9k4FNwNeN4wb1fUJkzKraBR9aB6gSrOMhmKSQ7DYwomoQ0okIJlkMq0RhEJkXL5a+n/GR9MOsL4UjHpQgFrCJX9Mww4L12wx/Jb3RcIzqpGGUp3EjsX68yxRoSy7GloVa3tQxHGHfXm9jvvndoIESis= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825392; c=relaxed/simple; bh=kllXqj7/n7qr5ooiWLb4gkupRQZfpra1DehjCKInps0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kQgtObPxK/fUJhxpZWTeWgLd261JULsouTcVpY3taBSejj1M0QgKQCda09uh1BXRY0SkMOETx4AZ6esPPJyxNRQ5V7msIJvF6ovzVaLtevqq1wWwo2cyWNgpg1hnf/n/xijp2Z5G8v1EEfsjtReDWS68i0bT1c9KUhuP3LAMLDM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=GI5ykoAE; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="GI5ykoAE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1724825386; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=3B0UdIQF1Wd35JDJ4rkg/HdwUySaqmCTJOmNmTkeSkU=; b=GI5ykoAEnMgpgfqCujgrgoyGyBAnzskWvDWLS4OpOjp6KJK1nTHcCjD/BWNEDea5iE6DoHzj9tlyke/sEMNiopMQskhR1svijG2YmtLurbN3OUTCwdiQC6RSandaUNgyJzjEOmxORupoiUCvtd5vIlhVZWL0Ws+ctBcS3o/6skA= Received: from localhost(mailfrom:chengyou@linux.alibaba.com fp:SMTPD_---0WDok9M6_1724825385) by smtp.aliyun-inc.com; Wed, 28 Aug 2024 14:09:46 +0800 From: Cheng Xu To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, KaiShen@linux.alibaba.com Subject: [PATCH for-next v2 1/4] RDMA/erdma: Make the device probe process more robust Date: Wed, 28 Aug 2024 14:09:41 +0800 Message-Id: <20240828060944.77829-2-chengyou@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240828060944.77829-1-chengyou@linux.alibaba.com> References: <20240828060944.77829-1-chengyou@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Driver may probe again while hardware is destroying the internal resources allocated for previous probing, which will fail the device probe. To make it more robust, we always issue a reset at the beginning of the device probe process. Signed-off-by: Cheng Xu --- drivers/infiniband/hw/erdma/erdma.h | 1 + drivers/infiniband/hw/erdma/erdma_main.c | 44 +++++++++++++++++++----- 2 files changed, 36 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h index c8bd698e21b0..b5c258f77ca0 100644 --- a/drivers/infiniband/hw/erdma/erdma.h +++ b/drivers/infiniband/hw/erdma/erdma.h @@ -94,6 +94,7 @@ enum { #define ERDMA_CMDQ_TIMEOUT_MS 15000 #define ERDMA_REG_ACCESS_WAIT_MS 20 +#define ERDMA_WAIT_DEV_REST_CNT 50 #define ERDMA_WAIT_DEV_DONE_CNT 500 struct erdma_cmdq { diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c index 7080f8a71ec4..9199058a0b29 100644 --- a/drivers/infiniband/hw/erdma/erdma_main.c +++ b/drivers/infiniband/hw/erdma/erdma_main.c @@ -209,11 +209,30 @@ static void erdma_device_uninit(struct erdma_dev *dev) dma_pool_destroy(dev->resp_pool); } -static void erdma_hw_reset(struct erdma_dev *dev) +static int erdma_hw_reset(struct erdma_dev *dev, bool wait) { u32 ctrl = FIELD_PREP(ERDMA_REG_DEV_CTRL_RESET_MASK, 1); + int i; erdma_reg_write32(dev, ERDMA_REGS_DEV_CTRL_REG, ctrl); + + if (!wait) + return 0; + + for (i = 0; i < ERDMA_WAIT_DEV_REST_CNT; i++) { + if (erdma_reg_read32_filed(dev, ERDMA_REGS_DEV_ST_REG, + ERDMA_REG_DEV_ST_RESET_DONE_MASK)) + break; + + msleep(ERDMA_REG_ACCESS_WAIT_MS); + } + + if (i == ERDMA_WAIT_DEV_REST_CNT) { + dev_err(&dev->pdev->dev, "wait reset done timeout.\n"); + return -ETIME; + } + + return 0; } static int erdma_wait_hw_init_done(struct erdma_dev *dev) @@ -239,6 +258,17 @@ static int erdma_wait_hw_init_done(struct erdma_dev *dev) return 0; } +static int erdma_preinit_check(struct erdma_dev *dev) +{ + u32 version = erdma_reg_read32(dev, ERDMA_REGS_VERSION_REG); + + /* we knows that it is a non-functional function. */ + if (version == 0) + return -ENODEV; + + return erdma_hw_reset(dev, true); +} + static const struct pci_device_id erdma_pci_tbl[] = { { PCI_DEVICE(PCI_VENDOR_ID_ALIBABA, 0x107f) }, {} @@ -248,7 +278,6 @@ static int erdma_probe_dev(struct pci_dev *pdev) { struct erdma_dev *dev; int bars, err; - u32 version; err = pci_enable_device(pdev); if (err) { @@ -287,12 +316,9 @@ static int erdma_probe_dev(struct pci_dev *pdev) goto err_release_bars; } - version = erdma_reg_read32(dev, ERDMA_REGS_VERSION_REG); - if (version == 0) { - /* we knows that it is a non-functional function. */ - err = -ENODEV; + err = erdma_preinit_check(dev); + if (err) goto err_iounmap_func_bar; - } err = erdma_device_init(dev, pdev); if (err) @@ -327,7 +353,7 @@ static int erdma_probe_dev(struct pci_dev *pdev) return 0; err_reset_hw: - erdma_hw_reset(dev); + erdma_hw_reset(dev, false); err_uninit_cmdq: erdma_cmdq_destroy(dev); @@ -364,7 +390,7 @@ static void erdma_remove_dev(struct pci_dev *pdev) struct erdma_dev *dev = pci_get_drvdata(pdev); erdma_ceqs_uninit(dev); - erdma_hw_reset(dev); + erdma_hw_reset(dev, false); erdma_cmdq_destroy(dev); erdma_aeq_destroy(dev); erdma_comm_irq_uninit(dev); From patchwork Wed Aug 28 06:09:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng Xu X-Patchwork-Id: 13780697 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B62691552ED for ; Wed, 28 Aug 2024 06:09:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825397; cv=none; b=qJadrqM06kz2CQ0fytWQzZysQzMPG1wrTQS4fBrFOAp15PCclSY6N+TlQriPTBo8VeHp+iFFP0okX82YkLUgeerGkXCudBWJahXSsZHbvgUb7vz1H/yLqjK1+qsc9Q8n3lURibCaGYwAdt8cJ3AW9mVlfHmt59nkfVO/+8tQWlo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825397; c=relaxed/simple; bh=6bMqRlo4FCpluhTpxFmj78/vmFLXamt5olGAR+6n67s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ix5+uvLJ6Y7tznEn39F9Sh+Nfg6aZwUI2Mdg6RXBXLiiObTRtkPPQPdU4gW82DumVN+/M6+IST/VWbHpEMEphAI2liCpthTq2QnO63hd+/a+klMhiAxSbYZE50/br6rRBfgLwHlP5zi7UUVbZ/iJXmSR6WwKPNX3OTCk5YvjgbY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=kNjfQ3ri; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="kNjfQ3ri" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1724825387; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=8NmTxMOks52GN+L0KbXgVvPlIkLFFlYcNgNhZu6UynY=; b=kNjfQ3riudIz3VcB80gfq4w257vSVGRggpt0EEadLEygIpR9x6uAwXQ8hgheu+aCaMddrHIBx9fGpY+uzn7YJp/5L0tO7xjUIjNlfpSJC6mReq1jSiDkngUDEph7gjjnUOYtPMVccG0nzSfW1kJZSPQ0qZ6pb/BrS02haxXRiHg= Received: from localhost(mailfrom:chengyou@linux.alibaba.com fp:SMTPD_---0WDok9MO_1724825386) by smtp.aliyun-inc.com; Wed, 28 Aug 2024 14:09:47 +0800 From: Cheng Xu To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, KaiShen@linux.alibaba.com Subject: [PATCH for-next v2 2/4] RDMA/erdma: Refactor the initialization and destruction of EQ Date: Wed, 28 Aug 2024 14:09:42 +0800 Message-Id: <20240828060944.77829-3-chengyou@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240828060944.77829-1-chengyou@linux.alibaba.com> References: <20240828060944.77829-1-chengyou@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We extracted the common parts of the initialization/destruction process to make the code cleaner. Signed-off-by: Cheng Xu --- drivers/infiniband/hw/erdma/erdma.h | 3 +- drivers/infiniband/hw/erdma/erdma_cmdq.c | 26 ++----- drivers/infiniband/hw/erdma/erdma_eq.c | 91 ++++++++++++------------ drivers/infiniband/hw/erdma/erdma_main.c | 4 +- 4 files changed, 54 insertions(+), 70 deletions(-) diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h index b5c258f77ca0..f3b648dcf4b6 100644 --- a/drivers/infiniband/hw/erdma/erdma.h +++ b/drivers/infiniband/hw/erdma/erdma.h @@ -275,7 +275,8 @@ void notify_eq(struct erdma_eq *eq); void *get_next_valid_eqe(struct erdma_eq *eq); int erdma_aeq_init(struct erdma_dev *dev); -void erdma_aeq_destroy(struct erdma_dev *dev); +int erdma_eq_common_init(struct erdma_dev *dev, struct erdma_eq *eq, u32 depth); +void erdma_eq_destroy(struct erdma_dev *dev, struct erdma_eq *eq); void erdma_aeq_event_handler(struct erdma_dev *dev); void erdma_ceq_completion_handler(struct erdma_eq_cb *ceq_cb); diff --git a/drivers/infiniband/hw/erdma/erdma_cmdq.c b/drivers/infiniband/hw/erdma/erdma_cmdq.c index 43ff40b5a09d..a3d8922d1ad1 100644 --- a/drivers/infiniband/hw/erdma/erdma_cmdq.c +++ b/drivers/infiniband/hw/erdma/erdma_cmdq.c @@ -158,20 +158,13 @@ static int erdma_cmdq_eq_init(struct erdma_dev *dev) { struct erdma_cmdq *cmdq = &dev->cmdq; struct erdma_eq *eq = &cmdq->eq; + int ret; - eq->depth = cmdq->max_outstandings; - eq->qbuf = dma_alloc_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, - &eq->qbuf_dma_addr, GFP_KERNEL); - if (!eq->qbuf) - return -ENOMEM; - - spin_lock_init(&eq->lock); - atomic64_set(&eq->event_num, 0); + ret = erdma_eq_common_init(dev, eq, cmdq->max_outstandings); + if (ret) + return ret; eq->db = dev->func_bar + ERDMA_REGS_CEQ_DB_BASE_REG; - eq->dbrec = dma_pool_zalloc(dev->db_pool, GFP_KERNEL, &eq->dbrec_dma); - if (!eq->dbrec) - goto err_out; erdma_reg_write32(dev, ERDMA_REGS_CMDQ_EQ_ADDR_H_REG, upper_32_bits(eq->qbuf_dma_addr)); @@ -181,12 +174,6 @@ static int erdma_cmdq_eq_init(struct erdma_dev *dev) erdma_reg_write64(dev, ERDMA_CMDQ_EQ_DB_HOST_ADDR_REG, eq->dbrec_dma); return 0; - -err_out: - dma_free_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, eq->qbuf, - eq->qbuf_dma_addr); - - return -ENOMEM; } int erdma_cmdq_init(struct erdma_dev *dev) @@ -247,10 +234,7 @@ void erdma_cmdq_destroy(struct erdma_dev *dev) clear_bit(ERDMA_CMDQ_STATE_OK_BIT, &cmdq->state); - dma_free_coherent(&dev->pdev->dev, cmdq->eq.depth << EQE_SHIFT, - cmdq->eq.qbuf, cmdq->eq.qbuf_dma_addr); - - dma_pool_free(dev->db_pool, cmdq->eq.dbrec, cmdq->eq.dbrec_dma); + erdma_eq_destroy(dev, &cmdq->eq); dma_free_coherent(&dev->pdev->dev, cmdq->sq.depth << SQEBB_SHIFT, cmdq->sq.qbuf, cmdq->sq.qbuf_dma_addr); diff --git a/drivers/infiniband/hw/erdma/erdma_eq.c b/drivers/infiniband/hw/erdma/erdma_eq.c index 84ccdd8144c9..c277c8f9233c 100644 --- a/drivers/infiniband/hw/erdma/erdma_eq.c +++ b/drivers/infiniband/hw/erdma/erdma_eq.c @@ -80,50 +80,62 @@ void erdma_aeq_event_handler(struct erdma_dev *dev) notify_eq(&dev->aeq); } -int erdma_aeq_init(struct erdma_dev *dev) +int erdma_eq_common_init(struct erdma_dev *dev, struct erdma_eq *eq, u32 depth) { - struct erdma_eq *eq = &dev->aeq; + u32 buf_size = depth << EQE_SHIFT; - eq->depth = ERDMA_DEFAULT_EQ_DEPTH; - - eq->qbuf = dma_alloc_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, - &eq->qbuf_dma_addr, GFP_KERNEL); + eq->qbuf = dma_alloc_coherent(&dev->pdev->dev, buf_size, + &eq->qbuf_dma_addr, + GFP_KERNEL | __GFP_ZERO); if (!eq->qbuf) return -ENOMEM; + eq->dbrec = dma_pool_alloc(dev->db_pool, GFP_KERNEL | __GFP_ZERO, + &eq->dbrec_dma); + if (!eq->dbrec) + goto err_free_qbuf; + spin_lock_init(&eq->lock); atomic64_set(&eq->event_num, 0); atomic64_set(&eq->notify_num, 0); - - eq->db = dev->func_bar + ERDMA_REGS_AEQ_DB_REG; - eq->dbrec = dma_pool_zalloc(dev->db_pool, GFP_KERNEL, &eq->dbrec_dma); - if (!eq->dbrec) - goto err_out; - - erdma_reg_write32(dev, ERDMA_REGS_AEQ_ADDR_H_REG, - upper_32_bits(eq->qbuf_dma_addr)); - erdma_reg_write32(dev, ERDMA_REGS_AEQ_ADDR_L_REG, - lower_32_bits(eq->qbuf_dma_addr)); - erdma_reg_write32(dev, ERDMA_REGS_AEQ_DEPTH_REG, eq->depth); - erdma_reg_write64(dev, ERDMA_AEQ_DB_HOST_ADDR_REG, eq->dbrec_dma); + eq->ci = 0; + eq->depth = depth; return 0; -err_out: - dma_free_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, eq->qbuf, +err_free_qbuf: + dma_free_coherent(&dev->pdev->dev, buf_size, eq->qbuf, eq->qbuf_dma_addr); return -ENOMEM; } -void erdma_aeq_destroy(struct erdma_dev *dev) +void erdma_eq_destroy(struct erdma_dev *dev, struct erdma_eq *eq) { - struct erdma_eq *eq = &dev->aeq; - + dma_pool_free(dev->db_pool, eq->dbrec, eq->dbrec_dma); dma_free_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, eq->qbuf, eq->qbuf_dma_addr); +} - dma_pool_free(dev->db_pool, eq->dbrec, eq->dbrec_dma); +int erdma_aeq_init(struct erdma_dev *dev) +{ + struct erdma_eq *eq = &dev->aeq; + int ret; + + ret = erdma_eq_common_init(dev, &dev->aeq, ERDMA_DEFAULT_EQ_DEPTH); + if (ret) + return ret; + + eq->db = dev->func_bar + ERDMA_REGS_AEQ_DB_REG; + + erdma_reg_write32(dev, ERDMA_REGS_AEQ_ADDR_H_REG, + upper_32_bits(eq->qbuf_dma_addr)); + erdma_reg_write32(dev, ERDMA_REGS_AEQ_ADDR_L_REG, + lower_32_bits(eq->qbuf_dma_addr)); + erdma_reg_write32(dev, ERDMA_REGS_AEQ_DEPTH_REG, eq->depth); + erdma_reg_write64(dev, ERDMA_AEQ_DB_HOST_ADDR_REG, eq->dbrec_dma); + + return 0; } void erdma_ceq_completion_handler(struct erdma_eq_cb *ceq_cb) @@ -234,32 +246,21 @@ static int erdma_ceq_init_one(struct erdma_dev *dev, u16 ceqn) struct erdma_eq *eq = &dev->ceqs[ceqn].eq; int ret; - eq->depth = ERDMA_DEFAULT_EQ_DEPTH; - eq->qbuf = dma_alloc_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, - &eq->qbuf_dma_addr, GFP_KERNEL); - if (!eq->qbuf) - return -ENOMEM; - - spin_lock_init(&eq->lock); - atomic64_set(&eq->event_num, 0); - atomic64_set(&eq->notify_num, 0); + ret = erdma_eq_common_init(dev, eq, ERDMA_DEFAULT_EQ_DEPTH); + if (ret) + return ret; eq->db = dev->func_bar + ERDMA_REGS_CEQ_DB_BASE_REG + (ceqn + 1) * ERDMA_DB_SIZE; - - eq->dbrec = dma_pool_zalloc(dev->db_pool, GFP_KERNEL, &eq->dbrec_dma); - if (!eq->dbrec) { - dma_free_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, - eq->qbuf, eq->qbuf_dma_addr); - return -ENOMEM; - } - - eq->ci = 0; dev->ceqs[ceqn].dev = dev; + dev->ceqs[ceqn].ready = true; /* CEQ indexed from 1, 0 rsvd for CMDQ-EQ. */ ret = create_eq_cmd(dev, ceqn + 1, eq); - dev->ceqs[ceqn].ready = ret ? false : true; + if (ret) { + erdma_eq_destroy(dev, eq); + dev->ceqs[ceqn].ready = false; + } return ret; } @@ -283,9 +284,7 @@ static void erdma_ceq_uninit_one(struct erdma_dev *dev, u16 ceqn) if (err) return; - dma_free_coherent(&dev->pdev->dev, eq->depth << EQE_SHIFT, eq->qbuf, - eq->qbuf_dma_addr); - dma_pool_free(dev->db_pool, eq->dbrec, eq->dbrec_dma); + erdma_eq_destroy(dev, eq); } int erdma_ceqs_init(struct erdma_dev *dev) diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c index 9199058a0b29..d1cb488e7ad4 100644 --- a/drivers/infiniband/hw/erdma/erdma_main.c +++ b/drivers/infiniband/hw/erdma/erdma_main.c @@ -359,7 +359,7 @@ static int erdma_probe_dev(struct pci_dev *pdev) erdma_cmdq_destroy(dev); err_uninit_aeq: - erdma_aeq_destroy(dev); + erdma_eq_destroy(dev, &dev->aeq); err_uninit_comm_irq: erdma_comm_irq_uninit(dev); @@ -392,7 +392,7 @@ static void erdma_remove_dev(struct pci_dev *pdev) erdma_ceqs_uninit(dev); erdma_hw_reset(dev, false); erdma_cmdq_destroy(dev); - erdma_aeq_destroy(dev); + erdma_eq_destroy(dev, &dev->aeq); erdma_comm_irq_uninit(dev); pci_free_irq_vectors(dev->pdev); erdma_device_uninit(dev); From patchwork Wed Aug 28 06:09:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng Xu X-Patchwork-Id: 13780695 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECA07152517 for ; Wed, 28 Aug 2024 06:09:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825393; cv=none; b=D1oXRXc7zDm+QfsfXG3B4UWF+QsIlwpb85I1bt/LslM/E6vTWZYBFHhspL4efXUAwYcLBtkd3WEHXXBp3iMuHHXLHZYzDrwdWwdY1JgzgFixYuGP1mZvF+2W3xe1YVNPXCNL7GrLtC0eP8oqLwtd1GUwnC4ICobU+2sH0j/OIak= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825393; c=relaxed/simple; bh=ckTvPKiI5QEhksb4cEarFV1MS9B0zHPtT2TxBBKjiVg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KlmI36/+UbYsIVkrcpzqr+vJ1qWHumwGlUSkaZEoCEsZHhIR4ej+BPo5EhQ3E3zKNhmXjLq3PQjY+8xYq8mkISSk1XpnlTEqm3vjeHqW88jrymM7chKmm8j6IyKzxmVfE1IULWvr6fH9d1R2K9NsDGl5DpV/vex/rnS9iKkxIBY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=t3I9QyP7; arc=none smtp.client-ip=115.124.30.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="t3I9QyP7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1724825388; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=ILEVsIxtwCVK9Ex+2VK/+fPKofqndLxjLXuNjQIFhMc=; b=t3I9QyP7oSnXI6epQaylsaPh5dDpkJtNQLPnnKUCRqxgKScwIFCVPYlFF9EcwKM4VFd4D/ttnFxt0DsIrH8WeaXePdEp6PlwylM505HQhNDxh1Lhs4wxXAV7mdShQq4Y5HycdQBuDRPAbQdVGfWgeEYmQAWjkZsm0yFdbsFdOZM= Received: from localhost(mailfrom:chengyou@linux.alibaba.com fp:SMTPD_---0WDok9Ma_1724825387) by smtp.aliyun-inc.com; Wed, 28 Aug 2024 14:09:48 +0800 From: Cheng Xu To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, KaiShen@linux.alibaba.com Subject: [PATCH for-next v2 3/4] RDMA/erdma: Add disassociate ucontext support Date: Wed, 28 Aug 2024 14:09:43 +0800 Message-Id: <20240828060944.77829-4-chengyou@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240828060944.77829-1-chengyou@linux.alibaba.com> References: <20240828060944.77829-1-chengyou@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 All IO pages mapped to user space are handled by rdma_user_mmap_io, so add empty stub for disassociate ucontext. Signed-off-by: Cheng Xu --- drivers/infiniband/hw/erdma/erdma_main.c | 1 + drivers/infiniband/hw/erdma/erdma_verbs.c | 4 ++++ drivers/infiniband/hw/erdma/erdma_verbs.h | 1 + 3 files changed, 6 insertions(+) diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c index d1cb488e7ad4..1ccf1b65c02c 100644 --- a/drivers/infiniband/hw/erdma/erdma_main.c +++ b/drivers/infiniband/hw/erdma/erdma_main.c @@ -516,6 +516,7 @@ static const struct ib_device_ops erdma_device_ops = { .dereg_mr = erdma_dereg_mr, .destroy_cq = erdma_destroy_cq, .destroy_qp = erdma_destroy_qp, + .disassociate_ucontext = erdma_disassociate_ucontext, .get_dma_mr = erdma_get_dma_mr, .get_hw_stats = erdma_get_hw_stats, .get_port_immutable = erdma_get_port_immutable, diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c index 40c9b6e46b82..48b08a15e6a8 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.c +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c @@ -1700,6 +1700,10 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, return ret; } +void erdma_disassociate_ucontext(struct ib_ucontext *ibcontext) +{ +} + void erdma_set_mtu(struct erdma_dev *dev, u32 mtu) { struct erdma_cmdq_config_mtu_req req; diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h index 4f02ba06b210..b7478376eb80 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.h +++ b/drivers/infiniband/hw/erdma/erdma_verbs.h @@ -344,6 +344,7 @@ int erdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int mask, struct ib_udata *data); int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata); int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata); +void erdma_disassociate_ucontext(struct ib_ucontext *ibcontext); int erdma_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags); struct ib_mr *erdma_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, u64 virt, int access, struct ib_udata *udata); From patchwork Wed Aug 28 06:09:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng Xu X-Patchwork-Id: 13780696 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF15B1552ED for ; Wed, 28 Aug 2024 06:09:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825393; cv=none; b=koM1KPeiFfehWglGEulLL5ocY3wK1UMUWTlxcIYAVC0BsBjO+R6R6UKOLj0ASvEOTCpLujNIKzCimseHaAgSS/3eda5rjPuXItWuQmctUOUA1hX11dMlAc9O0Gnp9JyzuEyedQZmMaoXrZEheYNuEVDlOuLg3HgqE6nC6dKc8wQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724825393; c=relaxed/simple; bh=atrAcBXqGYOwK9IhJ5JVPvzdvvQ4IF74G4Aivl+S04U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=npMESPmS2KdPOPfDZmfmksKgLrq1IiGXkW5Bj0WCRxHywy8tQ/y0WkP3TM1ExZgy6D2XTUDqnQb6hXkUm2fn9y1/vRdQu/VDeZ86aDXGCtas8YkpgkjgzdMohmwWPDziIr1lajFquLyb1muRPLR2Uhylqg8Ts1dgl6nQ1QHWFzs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=WQyWK9Bv; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="WQyWK9Bv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1724825389; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=PKiV/AnZhD8XWBkIWGSe+NiegrdQamtp1efkIRSVEOQ=; b=WQyWK9BvbWvrBeswBft+LY0fRjamzZeJx7ib3M9WFPUpiv2PnHr55PQWsNf2lnOyiyT3sEmFXKgkcCwlMm1Gx0VCO7Y74ExFnx+05aiyNshOXyGsYqjIbCbwHdeFte7IIcx8rxRTukKUzEY2C3ob6DDMtNAS1ZLCVlLHe6txS48= Received: from localhost(mailfrom:chengyou@linux.alibaba.com fp:SMTPD_---0WDonSTL_1724825388) by smtp.aliyun-inc.com; Wed, 28 Aug 2024 14:09:49 +0800 From: Cheng Xu To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, KaiShen@linux.alibaba.com Subject: [PATCH for-next v2 4/4] RDMA/erdma: Return QP state in erdma_query_qp Date: Wed, 28 Aug 2024 14:09:44 +0800 Message-Id: <20240828060944.77829-5-chengyou@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240828060944.77829-1-chengyou@linux.alibaba.com> References: <20240828060944.77829-1-chengyou@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fix qp_state and cur_qp_state to return correct values in struct ib_qp_attr. Fixes: 155055771704 ("RDMA/erdma: Add verbs implementation") Signed-off-by: Cheng Xu --- drivers/infiniband/hw/erdma/erdma_verbs.c | 25 ++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c index 48b08a15e6a8..de11f0f1adb1 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.c +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c @@ -1544,11 +1544,31 @@ int erdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask, return ret; } +static inline enum ib_qp_state query_qp_state(struct erdma_qp *qp) +{ + switch (qp->attrs.state) { + case ERDMA_QP_STATE_IDLE: + return IB_QPS_INIT; + case ERDMA_QP_STATE_RTR: + return IB_QPS_RTR; + case ERDMA_QP_STATE_RTS: + return IB_QPS_RTS; + case ERDMA_QP_STATE_CLOSING: + return IB_QPS_ERR; + case ERDMA_QP_STATE_TERMINATE: + return IB_QPS_ERR; + case ERDMA_QP_STATE_ERROR: + return IB_QPS_ERR; + default: + return IB_QPS_ERR; + } +} + int erdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr, int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr) { - struct erdma_qp *qp; struct erdma_dev *dev; + struct erdma_qp *qp; if (ibqp && qp_attr && qp_init_attr) { qp = to_eqp(ibqp); @@ -1575,6 +1595,9 @@ int erdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr, qp_init_attr->cap = qp_attr->cap; + qp_attr->qp_state = query_qp_state(qp); + qp_attr->cur_qp_state = query_qp_state(qp); + return 0; }