From patchwork Fri Nov 10 01:08:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Smart X-Patchwork-Id: 10052233 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D4BC26032D for ; Fri, 10 Nov 2017 01:09:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7BE32B20E for ; Fri, 10 Nov 2017 01:09:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCE862B227; Fri, 10 Nov 2017 01:09:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5577B2B20E for ; Fri, 10 Nov 2017 01:09:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755509AbdKJBJH (ORCPT ); Thu, 9 Nov 2017 20:09:07 -0500 Received: from mail-qk0-f193.google.com ([209.85.220.193]:43634 "EHLO mail-qk0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755499AbdKJBI6 (ORCPT ); Thu, 9 Nov 2017 20:08:58 -0500 Received: by mail-qk0-f193.google.com with SMTP id 78so10123880qkz.0; Thu, 09 Nov 2017 17:08:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IuPoH0p6yvrUK8ddj3NO35qh/7RzOdc+y6/xynE+iNA=; b=C7WVinXIDUUgmJxRmY6vbhKiIIkIGFenX3uaioalr7Q/FYGMH96nN4k2NzKb15E+Ss ZAJnc0dZRFzXExrAk7TzrprJ3yXsuxsh1hSNwf8E8PZrQSr8U+GiEL+XyT/zDvw0JUKa 1bOU7ks6aWO9mcq0lNzrw3VLgSUjFyXZ0MVpBsY6YL4bLT7+f5Lc76INIUr8v6Zsx3Uh N5hU6xN65tenFiX93lF4j7CVjMkDgH9x2WrvvFeUYwUBmLqeKNuC+F5mylfgQgnt5tct BucxesLX9zFSpQ2ItkrsMnWLmqYGItIcPwOptY3vTtR4bV2n7kz0QNBdRBN1Jdo6AlJy 7Ilg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IuPoH0p6yvrUK8ddj3NO35qh/7RzOdc+y6/xynE+iNA=; b=EgXJXTBYdL4MOc/vVbYJYeQuS/zvIqrJpWUw+D+GE9YmRle99RIInQWP7C9ZT95aky EwtrX1ec4wQcm1jINlVKQbuKp2X0S06Ii9MIXENcj6m5G2nsuy5TajjmBdBSRteDBGWc PZAuraCp5DVH9AceHvDEkgVA+AEleyAKffXxUi88ewGA+9f7dVLei6Y203VCFMJSQe2T 68jIhTD+TMw2LS6JNPb3KDV0kWZx8ZpWXDwx3U5Ue1LL1LgWpW4l1skHdWBF9JYvsvsG vLgjI/kWRugnoA5u1TNfy8F/wN40AjhGQNGAlhx2f+bAQvu8t0eUiovzD6VMms1P3Jr/ zzOQ== X-Gm-Message-State: AJaThX41SqETxx4gO64iIwDP7UoYiITPcfkkvAYGU6NA1SbW1WLjIBtg p8KlM4E+gjRFRqC5GPYos/Ec+jHG X-Google-Smtp-Source: AGs4zMbNZ0yilLHMKAHK8+ZMXHZvIcn8L2WtlOmYCaWe0wPcUflRZmlr7H5TV0Z5PSW3BB6SOIYQPw== X-Received: by 10.55.133.71 with SMTP id h68mr3774619qkd.17.1510276137488; Thu, 09 Nov 2017 17:08:57 -0800 (PST) Received: from pallmd1.broadcom.com ([192.19.255.250]) by smtp.gmail.com with ESMTPSA id j48sm5661969qta.96.2017.11.09.17.08.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 09 Nov 2017 17:08:57 -0800 (PST) From: James Smart To: linux-scsi@vger.kernel.org Cc: James Smart , stable@vger.kernel.org, Dick Kennedy , James Smart Subject: [PATCH v2 15/17] lpfc: Fix driver handling of nvme resources during unload Date: Thu, 9 Nov 2017 17:08:32 -0800 Message-Id: <20171110010834.4542-16-jsmart2021@gmail.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20171110010834.4542-1-jsmart2021@gmail.com> References: <20171110010834.4542-1-jsmart2021@gmail.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP During driver unload, the driver may crash due to NULL pointers. The NULL pointers were due to the driver not protecting itself sufficiently during some of the teardown paths. Additionally, the driver was not waiting for and cleanup up nvme io resources. As such, the driver wasn't making the callbacks to the transport, stalling the transports association teardown. This patch waits for io clean up before tearding down and adds checks for possible NULL pointers. Cc: # 4.12+ Signed-off-by: Dick Kennedy Signed-off-by: James Smart Reviewed-by: Hannes Reinecke --- drivers/scsi/lpfc/lpfc_crtn.h | 2 + drivers/scsi/lpfc/lpfc_init.c | 18 ++++++++ drivers/scsi/lpfc/lpfc_nvme.c | 96 ++++++++++++++++++++++++++++++++++++++----- 3 files changed, 105 insertions(+), 11 deletions(-) diff --git a/drivers/scsi/lpfc/lpfc_crtn.h b/drivers/scsi/lpfc/lpfc_crtn.h index 7e300734b345..dac33900bf17 100644 --- a/drivers/scsi/lpfc/lpfc_crtn.h +++ b/drivers/scsi/lpfc/lpfc_crtn.h @@ -254,6 +254,8 @@ void lpfc_nvmet_ctxbuf_post(struct lpfc_hba *phba, struct lpfc_nvmet_ctxbuf *ctxp); int lpfc_nvmet_rcv_unsol_abort(struct lpfc_vport *vport, struct fc_frame_header *fc_hdr); +void lpfc_sli_flush_nvme_rings(struct lpfc_hba *phba); +void lpfc_nvme_wait_for_io_drain(struct lpfc_hba *phba); void lpfc_sli4_build_dflt_fcf_record(struct lpfc_hba *, struct fcf_record *, uint16_t); int lpfc_sli4_rq_put(struct lpfc_queue *hq, struct lpfc_queue *dq, diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index 7a06f23a3baf..c466ceb43bc9 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -10136,6 +10136,16 @@ lpfc_sli4_xri_exchange_busy_wait(struct lpfc_hba *phba) int fcp_xri_cmpl = 1; int els_xri_cmpl = list_empty(&phba->sli4_hba.lpfc_abts_els_sgl_list); + /* Driver just aborted IOs during the hba_unset process. Pause + * here to give the HBA time to complete the IO and get entries + * into the abts lists. + */ + msleep(LPFC_XRI_EXCH_BUSY_WAIT_T1 * 5); + + /* Wait for NVME pending IO to flush back to transport. */ + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) + lpfc_nvme_wait_for_io_drain(phba); + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_FCP) fcp_xri_cmpl = list_empty(&phba->sli4_hba.lpfc_abts_scsi_buf_list); @@ -11659,6 +11669,10 @@ lpfc_sli4_prep_dev_for_reset(struct lpfc_hba *phba) /* Flush all driver's outstanding SCSI I/Os as we are to reset */ lpfc_sli_flush_fcp_rings(phba); + /* Flush the outstanding NVME IOs if fc4 type enabled. */ + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) + lpfc_sli_flush_nvme_rings(phba); + /* stop all timers */ lpfc_stop_hba_timers(phba); @@ -11690,6 +11704,10 @@ lpfc_sli4_prep_dev_for_perm_failure(struct lpfc_hba *phba) /* Clean up all driver's outstanding SCSI I/Os */ lpfc_sli_flush_fcp_rings(phba); + + /* Flush the outstanding NVME IOs if fc4 type enabled. */ + if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) + lpfc_sli_flush_nvme_rings(phba); } /** diff --git a/drivers/scsi/lpfc/lpfc_nvme.c b/drivers/scsi/lpfc/lpfc_nvme.c index ea7a0e65bbfd..67bd0ff299d1 100644 --- a/drivers/scsi/lpfc/lpfc_nvme.c +++ b/drivers/scsi/lpfc/lpfc_nvme.c @@ -88,6 +88,9 @@ lpfc_nvme_create_queue(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_qhandle *qhandle; char *str; + if (!pnvme_lport->private) + return -ENOMEM; + lport = (struct lpfc_nvme_lport *)pnvme_lport->private; vport = lport->vport; qhandle = kzalloc(sizeof(struct lpfc_nvme_qhandle), GFP_KERNEL); @@ -140,6 +143,9 @@ lpfc_nvme_delete_queue(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_lport *lport; struct lpfc_vport *vport; + if (!pnvme_lport->private) + return; + lport = (struct lpfc_nvme_lport *)pnvme_lport->private; vport = lport->vport; @@ -1265,13 +1271,29 @@ lpfc_nvme_fcp_io_submit(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_buf *lpfc_ncmd; struct lpfc_nvme_rport *rport; struct lpfc_nvme_qhandle *lpfc_queue_info; - struct lpfc_nvme_fcpreq_priv *freqpriv = pnvme_fcreq->private; + struct lpfc_nvme_fcpreq_priv *freqpriv; #ifdef CONFIG_SCSI_LPFC_DEBUG_FS uint64_t start = 0; #endif + /* Validate pointers. LLDD fault handling with transport does + * have timing races. + */ lport = (struct lpfc_nvme_lport *)pnvme_lport->private; + if (unlikely(!lport)) { + ret = -EINVAL; + goto out_fail; + } + vport = lport->vport; + + if (unlikely(!hw_queue_handle)) { + lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_ABTS, + "6129 Fail Abort, NULL hw_queue_handle\n"); + ret = -EINVAL; + goto out_fail; + } + phba = vport->phba; if (vport->load_flag & FC_UNLOADING) { @@ -1284,13 +1306,9 @@ lpfc_nvme_fcp_io_submit(struct nvme_fc_local_port *pnvme_lport, goto out_fail; } - /* Validate pointers. */ - if (!pnvme_lport || !pnvme_rport || !freqpriv) { - lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_IOERR | LOG_NODE, - "6117 No Send:IO submit ptrs NULL, lport %p, " - "rport %p fcreq_priv %p\n", - pnvme_lport, pnvme_rport, freqpriv); - ret = -ENODEV; + freqpriv = pnvme_fcreq->private; + if (unlikely(!freqpriv)) { + ret = -EINVAL; goto out_fail; } @@ -1497,20 +1515,34 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, struct lpfc_nvme_lport *lport; struct lpfc_vport *vport; struct lpfc_hba *phba; - struct lpfc_nvme_rport *rport; struct lpfc_nvme_buf *lpfc_nbuf; struct lpfc_iocbq *abts_buf; struct lpfc_iocbq *nvmereq_wqe; - struct lpfc_nvme_fcpreq_priv *freqpriv = pnvme_fcreq->private; + struct lpfc_nvme_fcpreq_priv *freqpriv; union lpfc_wqe *abts_wqe; unsigned long flags; int ret_val; + /* Validate pointers. LLDD fault handling with transport does + * have timing races. + */ lport = (struct lpfc_nvme_lport *)pnvme_lport->private; - rport = (struct lpfc_nvme_rport *)pnvme_rport->private; + if (unlikely(!lport)) + return; + vport = lport->vport; + + if (unlikely(!hw_queue_handle)) { + lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_ABTS, + "6129 Fail Abort, HW Queue Handle NULL.\n"); + return; + } + phba = vport->phba; + freqpriv = pnvme_fcreq->private; + if (unlikely(!freqpriv)) + return; if (vport->load_flag & FC_UNLOADING) return; @@ -2678,3 +2710,45 @@ lpfc_sli4_nvme_xri_aborted(struct lpfc_hba *phba, "6312 XRI Aborted xri x%x not found\n", xri); } + +/** + * lpfc_nvme_wait_for_io_drain - Wait for all NVME wqes to complete + * @phba: Pointer to HBA context object. + * + * This function flushes all wqes in the nvme rings and frees all resources + * in the txcmplq. This function does not issue abort wqes for the IO + * commands in txcmplq, they will just be returned with + * IOERR_SLI_DOWN. This function is invoked with EEH when device's PCI + * slot has been permanently disabled. + **/ +void +lpfc_nvme_wait_for_io_drain(struct lpfc_hba *phba) +{ + struct lpfc_sli_ring *pring; + u32 i, wait_cnt = 0; + + if (phba->sli_rev < LPFC_SLI_REV4) + return; + + /* Cycle through all NVME rings and make sure all outstanding + * WQEs have been removed from the txcmplqs. + */ + for (i = 0; i < phba->cfg_nvme_io_channel; i++) { + pring = phba->sli4_hba.nvme_wq[i]->pring; + + /* Retrieve everything on the txcmplq */ + while (!list_empty(&pring->txcmplq)) { + msleep(LPFC_XRI_EXCH_BUSY_WAIT_T1); + wait_cnt++; + + /* The sleep is 10mS. Every ten seconds, + * dump a message. Something is wrong. + */ + if ((wait_cnt % 1000) == 0) { + lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR, + "6178 NVME IO not empty, " + "cnt %d\n", wait_cnt); + } + } + } +}