From patchwork Fri Nov 3 22:56:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Smart X-Patchwork-Id: 10041305 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1F68760384 for ; Fri, 3 Nov 2017 22:57:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 116B9299CE for ; Fri, 3 Nov 2017 22:57:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 06552299D9; Fri, 3 Nov 2017 22:57:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 653B7299E1 for ; Fri, 3 Nov 2017 22:57:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754811AbdKCW5B (ORCPT ); Fri, 3 Nov 2017 18:57:01 -0400 Received: from mail-qt0-f194.google.com ([209.85.216.194]:43959 "EHLO mail-qt0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756069AbdKCW4w (ORCPT ); Fri, 3 Nov 2017 18:56:52 -0400 Received: by mail-qt0-f194.google.com with SMTP id j58so5084480qtj.0; Fri, 03 Nov 2017 15:56:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IuPoH0p6yvrUK8ddj3NO35qh/7RzOdc+y6/xynE+iNA=; b=oNPBFEBzMRNTwTFbJfijet4QTCQYVLzibVoJFMr4hMokhQ50ZQABF/WuU2llaTQLOh AjCM7zguNHBAEtdWfXBaDfrm0kbZ3re6tNndzLyOZtcDp2i4QmgEmycVE+aVQbz6MB/H 8wUA7sxR5WltvEKcQ9xrv0kxfUA+tFj+7zcNrJ8nGuSj8Riig9JvETrCO1Pwa63GtyvF H4ywJgKc9DGPOM3LfwWbORNFECrGhBBnu6Wxeil8/anPiQnJYNSszhVRpZBQEj9cRE9H h7ididRLsZ5XW1oqevXKUY2WROrFYVDjSKbkd0uI7ohYzi0OT/rjtviViK1G724M6tk7 bWNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IuPoH0p6yvrUK8ddj3NO35qh/7RzOdc+y6/xynE+iNA=; b=iE+cPAZbWeeN94RP35qwvlqFikZIaNxk3urW0C7HHU3x1JsgY2n2qgNBlLLHntJl/L /YGGh1IZzkMfY5oYxJ5ZbM0JL2msh7SexKbxdzQcmKVC/KgRELQVKcsO1qv2aP9UNwaG 5/ehXq2k+S/jYitrVvkgCo3KddhNCmB9O66CGHu1WqdsCIFSMCxfMxCWrgerIUR5CMPZ 8zuzvzrlCZl4YIAgLawAEnTqKUYLF/a6D7Og4cLzeDAFeI7jGMSt1p9TnNONFGxqIYAx tgqvZjhBdOz1zRO7P2nja0elF6hMhTxJ6pR1CD4zB6Er6jtCpaevLdukJRqyC5jvznvj F4MQ== X-Gm-Message-State: AMCzsaWm0CxRd5cAMV66gz2o/sd3rIa37QWXpWn6EUdafc73cQmRFxfi lXOIyiKaMEcuHC7Fjs2pt/057Vex X-Google-Smtp-Source: ABhQp+StbY/pQ/I0ORcBSxVPL9+fLbi9Z+MjA6ktK2iJHmi6RXlXthD7+nbJ4RiBsBRpS+j5y6MjDA== X-Received: by 10.237.59.25 with SMTP id p25mr12970728qte.304.1509749811546; Fri, 03 Nov 2017 15:56:51 -0700 (PDT) Received: from pallmd1.broadcom.com ([192.19.255.250]) by smtp.gmail.com with ESMTPSA id v21sm4765626qtv.21.2017.11.03.15.56.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 03 Nov 2017 15:56:51 -0700 (PDT) From: James Smart To: linux-scsi@vger.kernel.org Cc: James Smart , stable@vger.kernel.org, Dick Kennedy , James Smart Subject: [PATCH 15/17] lpfc: Fix driver handling of nvme resources during unload Date: Fri, 3 Nov 2017 15:56:26 -0700 Message-Id: <20171103225628.24716-16-jsmart2021@gmail.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20171103225628.24716-1-jsmart2021@gmail.com> References: <20171103225628.24716-1-jsmart2021@gmail.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP During driver unload, the driver may crash due to NULL pointers. The NULL pointers were due to the driver not protecting itself sufficiently during some of the teardown paths. Additionally, the driver was not waiting for and cleanup up nvme io resources. As such, the driver wasn't making the callbacks to the transport, stalling the transports association teardown. This patch waits for io clean up before tearding down and adds checks for possible NULL pointers. Cc: # 4.12+ Signed-off-by: Dick Kennedy Signed-off-by: James Smart --- drivers/scsi/lpfc/lpfc_crtn.h | 2 + drivers/scsi/lpfc/lpfc_init.c | 18 ++++++++ drivers/scsi/lpfc/lpfc_nvme.c | 96 ++++++++++++++++++++++++++++++++++++++----- 3 files changed, 105 insertions(+), 11 deletions(-) diff --git a/drivers/scsi/lpfc/lpfc_crtn.h b/drivers/scsi/lpfc/lpfc_crtn.h index 7e300734b345..dac33900bf17 100644 --- a/drivers/scsi/lpfc/lpfc_crtn.h +++ b/drivers/scsi/lpfc/lpfc_crtn.h @@ -254,6 +254,8 @@ void lpfc_nvmet_ctxbuf_post(struct lpfc_hba *phba, struct lpfc_nvmet_ctxbuf *ctxp); int lpfc_nvmet_rcv_unsol_abort(struct lpfc_vport *vport, struct fc_frame_header *fc_hdr); +void lpfc_sli_flush_nvme_rings(struct lpfc_hba *phba); +void lpfc_nvme_wait_for_io_drain(struct lpfc_hba *phba); void lpfc_sli4_build_dflt_fcf_record(struct lpfc_hba *, struct fcf_record *, uint16_t); int lpfc_sli4_rq_put(struct lpfc_queue *hq, struct lpfc_queue *dq, diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index 7a06f23a3baf..c466ceb43bc9 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -10136,6 +10136,16 @@ lpfc_sli4_xri_exchange_busy_wait(struct lpfc_hba *phba) int fcp_xri_cmpl = 1; int els_xri_cmpl = list_empty(&phba->sli4_hba.lpfc_abts_els_sgl_list); + /* Driver just aborted IOs during the hba_unset process. Pause + * here to give the HBA time to complete the IO and get entries + * into the abts lists. + */ + msleep(LPFC_XRI_EXCH_BUSY_WAIT_T1 * 5); + + /* Wait for NVME pending IO to flush back to transport. */ + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) + lpfc_nvme_wait_for_io_drain(phba); + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_FCP) fcp_xri_cmpl = list_empty(&phba->sli4_hba.lpfc_abts_scsi_buf_list); @@ -11659,6 +11669,10 @@ lpfc_sli4_prep_dev_for_reset(struct lpfc_hba *phba) /* Flush all driver's outstanding SCSI I/Os as we are to reset */ lpfc_sli_flush_fcp_rings(phba); + /* Flush the outstanding NVME IOs if fc4 type enabled. */ + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) + lpfc_sli_flush_nvme_rings(phba); + /* stop all timers */ lpfc_stop_hba_timers(phba); @@ -11690,6 +11704,10 @@ lpfc_sli4_prep_dev_for_perm_failure(struct lpfc_hba *phba) /* Clean up all driver's outstanding SCSI I/Os */ lpfc_sli_flush_fcp_rings(phba); + + /* Flush the outstanding NVME IOs if fc4 type enabled. */ + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) + lpfc_sli_flush_nvme_rings(phba); } /** diff --git a/drivers/scsi/lpfc/lpfc_nvme.c b/drivers/scsi/lpfc/lpfc_nvme.c index ea7a0e65bbfd..67bd0ff299d1 100644 --- a/drivers/scsi/lpfc/lpfc_nvme.c +++ b/drivers/scsi/lpfc/lpfc_nvme.c @@ -88,6 +88,9 @@ lpfc_nvme_create_queue(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_qhandle *qhandle; char *str; + if (!pnvme_lport->private) + return -ENOMEM; + lport = (struct lpfc_nvme_lport *)pnvme_lport->private; vport = lport->vport; qhandle = kzalloc(sizeof(struct lpfc_nvme_qhandle), GFP_KERNEL); @@ -140,6 +143,9 @@ lpfc_nvme_delete_queue(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_lport *lport; struct lpfc_vport *vport; + if (!pnvme_lport->private) + return; + lport = (struct lpfc_nvme_lport *)pnvme_lport->private; vport = lport->vport; @@ -1265,13 +1271,29 @@ lpfc_nvme_fcp_io_submit(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_buf *lpfc_ncmd; struct lpfc_nvme_rport *rport; struct lpfc_nvme_qhandle *lpfc_queue_info; - struct lpfc_nvme_fcpreq_priv *freqpriv = pnvme_fcreq->private; + struct lpfc_nvme_fcpreq_priv *freqpriv; #ifdef CONFIG_SCSI_LPFC_DEBUG_FS uint64_t start = 0; #endif + /* Validate pointers. LLDD fault handling with transport does + * have timing races. + */ lport = (struct lpfc_nvme_lport *)pnvme_lport->private; + if (unlikely(!lport)) { + ret = -EINVAL; + goto out_fail; + } + vport = lport->vport; + + if (unlikely(!hw_queue_handle)) { + lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_ABTS, + "6129 Fail Abort, NULL hw_queue_handle\n"); + ret = -EINVAL; + goto out_fail; + } + phba = vport->phba; if (vport->load_flag & FC_UNLOADING) { @@ -1284,13 +1306,9 @@ lpfc_nvme_fcp_io_submit(struct nvme_fc_local_port *pnvme_lport, goto out_fail; } - /* Validate pointers. */ - if (!pnvme_lport || !pnvme_rport || !freqpriv) { - lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_IOERR | LOG_NODE, - "6117 No Send:IO submit ptrs NULL, lport %p, " - "rport %p fcreq_priv %p\n", - pnvme_lport, pnvme_rport, freqpriv); - ret = -ENODEV; + freqpriv = pnvme_fcreq->private; + if (unlikely(!freqpriv)) { + ret = -EINVAL; goto out_fail; } @@ -1497,20 +1515,34 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_lport *lport; struct lpfc_vport *vport; struct lpfc_hba *phba; - struct lpfc_nvme_rport *rport; struct lpfc_nvme_buf *lpfc_nbuf; struct lpfc_iocbq *abts_buf; struct lpfc_iocbq *nvmereq_wqe; - struct lpfc_nvme_fcpreq_priv *freqpriv = pnvme_fcreq->private; + struct lpfc_nvme_fcpreq_priv *freqpriv; union lpfc_wqe *abts_wqe; unsigned long flags; int ret_val; + /* Validate pointers. LLDD fault handling with transport does + * have timing races. + */ lport = (struct lpfc_nvme_lport *)pnvme_lport->private; - rport = (struct lpfc_nvme_rport *)pnvme_rport->private; + if (unlikely(!lport)) + return; + vport = lport->vport; + + if (unlikely(!hw_queue_handle)) { + lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_ABTS, + "6129 Fail Abort, HW Queue Handle NULL.\n"); + return; + } + phba = vport->phba; + freqpriv = pnvme_fcreq->private; + if (unlikely(!freqpriv)) + return; if (vport->load_flag & FC_UNLOADING) return; @@ -2678,3 +2710,45 @@ lpfc_sli4_nvme_xri_aborted(struct lpfc_hba *phba, "6312 XRI Aborted xri x%x not found\n", xri); } + +/** + * lpfc_nvme_wait_for_io_drain - Wait for all NVME wqes to complete + * @phba: Pointer to HBA context object. + * + * This function flushes all wqes in the nvme rings and frees all resources + * in the txcmplq. This function does not issue abort wqes for the IO + * commands in txcmplq, they will just be returned with + * IOERR_SLI_DOWN. This function is invoked with EEH when device's PCI + * slot has been permanently disabled. + **/ +void +lpfc_nvme_wait_for_io_drain(struct lpfc_hba *phba) +{ + struct lpfc_sli_ring *pring; + u32 i, wait_cnt = 0; + + if (phba->sli_rev < LPFC_SLI_REV4) + return; + + /* Cycle through all NVME rings and make sure all outstanding + * WQEs have been removed from the txcmplqs. + */ + for (i = 0; i < phba->cfg_nvme_io_channel; i++) { + pring = phba->sli4_hba.nvme_wq[i]->pring; + + /* Retrieve everything on the txcmplq */ + while (!list_empty(&pring->txcmplq)) { + msleep(LPFC_XRI_EXCH_BUSY_WAIT_T1); + wait_cnt++; + + /* The sleep is 10mS. Every ten seconds, + * dump a message. Something is wrong. + */ + if ((wait_cnt % 1000) == 0) { + lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR, + "6178 NVME IO not empty, " + "cnt %d\n", wait_cnt); + } + } + } +}