From patchwork Sat Sep 30 00:34:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Smart X-Patchwork-Id: 9978893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A61456037F for ; Sat, 30 Sep 2017 00:35:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9831E298D1 for ; Sat, 30 Sep 2017 00:35:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8D411298D5; Sat, 30 Sep 2017 00:35:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B7DA298D1 for ; Sat, 30 Sep 2017 00:35:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752649AbdI3AfO (ORCPT ); Fri, 29 Sep 2017 20:35:14 -0400 Received: from mail-qt0-f195.google.com ([209.85.216.195]:48488 "EHLO mail-qt0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752620AbdI3AfM (ORCPT ); Fri, 29 Sep 2017 20:35:12 -0400 Received: by mail-qt0-f195.google.com with SMTP id d13so933041qta.5 for ; Fri, 29 Sep 2017 17:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=idRGwGGEoyGlpBarLU1U8FQ7jQDbPl9cNEmgJ7RgK10=; b=VCmx1ABclCdSzLhRgx30kdAz8noE6XOt21e1/Nn7I/5qG5OhkJMoKUOwvuQsfY13wp /lXtsD8e10qEgFT640KCuxouu9+qSf1oEjectYuKKtxk8aGEHnoHjfWI8TYa2VUauDtw GNv8thB3EnSc5zY7Y1Vb9Y2FwVNPSnDY4OU2wQGVs/cd5rzf8Ou5CODm3A47Q/BzKSLk n5GW/Y+U0VAHfmnlLUhYFnob+BaItrlUvoVvnv4BOgtURcRMJBFA+TsZjv33EEkXQ7Nw 8q8RGLgEDVLY/p7GubGjOxN8iIwO4teU4BmSDckZbcAFQljqbJetjMtuGEiXpQvapdTR Z8kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=idRGwGGEoyGlpBarLU1U8FQ7jQDbPl9cNEmgJ7RgK10=; b=NitX6OgP4N80RWK44WiNpBr/b/Z4QZ1i01f0UiW1gb/wXelLlCXIH5WQ0jNuZhovgg G4O8g7oYix4JQPvs5MGy4V/3quMRDofvIm+y8kciMfaj8QYf7ZiDUEK3wAQW5fRNID0H 4Ir0BHbclUttBatkwXMk66J1gQaH729kWhtuBQQ9XtF78FZs5ISGup/iCc3b0xBRTF4s zgEFUKgS6YTBNpeTUkp9TrW8bV8oohf8dhtn7lqHc5Mm3p3CzfTXcQ3FxOVyvckNyV75 tCyRgAeDwyChBpsdNFH9dC90Rc6fFHEqJl9HKkE2ntRlmfARUdZKaSSjQIf6u3OHsXuK 1xQw== X-Gm-Message-State: AMCzsaWEvW8tt1oQs/BBAW1mOBDvxarWyz+0xq5WRyZwOl7m5XPI0/82 fliBaffB7N0Anw4mArXIUvdneg== X-Google-Smtp-Source: AOwi7QARH1ioowmT8NvTloogpfxZDY9PjpqQGmMjlXICoqa74yR1bozLcJEwGEHqsV6oWzOpJgJajg== X-Received: by 10.200.35.153 with SMTP id q25mr9768198qtq.178.1506731711502; Fri, 29 Sep 2017 17:35:11 -0700 (PDT) Received: from pallmd1.broadcom.com ([192.19.255.250]) by smtp.gmail.com with ESMTPSA id u32sm3596918qte.25.2017.09.29.17.35.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 29 Sep 2017 17:35:11 -0700 (PDT) From: James Smart To: linux-scsi@vger.kernel.org Cc: Dick Kennedy , James Smart Subject: [PATCH v2 17/21] lpfc: Ensure io aborts interlocked with the target. Date: Fri, 29 Sep 2017 17:34:43 -0700 Message-Id: <20170930003447.10747-18-jsmart2021@gmail.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20170930003447.10747-1-jsmart2021@gmail.com> References: <20170930003447.10747-1-jsmart2021@gmail.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Dick Kennedy Before releasing nvme io back to the io stack for possible retry on other paths, ensure the io termination is interlocked with the target device by ensuring the entire ABTS-LS protocol is complete. Additionally, FC-NVME ABTS-LS protocol does not use RRQ. Remove RRQ behavior from ABTS-LS. Signed-off-by: Dick Kennedy Signed-off-by: James Smart Reviewed-by: Johannes Thumshirn --- drivers/scsi/lpfc/lpfc_nvme.c | 59 +++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/drivers/scsi/lpfc/lpfc_nvme.c b/drivers/scsi/lpfc/lpfc_nvme.c index 02ba06f364c4..e3642c1890ea 100644 --- a/drivers/scsi/lpfc/lpfc_nvme.c +++ b/drivers/scsi/lpfc/lpfc_nvme.c @@ -850,7 +850,7 @@ lpfc_nvme_io_cmd_wqe_cmpl(struct lpfc_hba *phba, struct lpfc_iocbq *pwqeIn, } else { lpfc_ncmd->status = (bf_get(lpfc_wcqe_c_status, wcqe) & LPFC_IOCB_STATUS_MASK); - lpfc_ncmd->result = wcqe->parameter; + lpfc_ncmd->result = (wcqe->parameter & IOERR_PARAM_MASK); /* For NVME, the only failure path that results in an * IO error is when the adapter rejects it. All other @@ -884,6 +884,17 @@ lpfc_nvme_io_cmd_wqe_cmpl(struct lpfc_hba *phba, struct lpfc_iocbq *pwqeIn, lpfc_ncmd->status, lpfc_ncmd->result, wcqe->total_data_placed); break; + case IOSTAT_LOCAL_REJECT: + /* Let fall through to set command final state. */ + if (lpfc_ncmd->result == IOERR_ABORT_REQUESTED) + lpfc_printf_vlog(vport, KERN_INFO, + LOG_NVME_IOERR, + "6032 Delay Aborted cmd %p " + "nvme cmd %p, xri x%x, " + "xb %d\n", + lpfc_ncmd, nCmd, + lpfc_ncmd->cur_iocbq.sli4_xritag, + bf_get(lpfc_wcqe_c_xb, wcqe)); default: out_err: lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_IOERR, @@ -930,12 +941,18 @@ lpfc_nvme_io_cmd_wqe_cmpl(struct lpfc_hba *phba, struct lpfc_iocbq *pwqeIn, #endif freqpriv = nCmd->private; freqpriv->nvme_buf = NULL; - nCmd->done(nCmd); + + /* NVME targets need completion held off until the abort exchange + * completes. + */ + if (!lpfc_ncmd->flags & LPFC_SBUF_XBUSY) + nCmd->done(nCmd); spin_lock_irqsave(&phba->hbalock, flags); lpfc_ncmd->nrport = NULL; spin_unlock_irqrestore(&phba->hbalock, flags); + /* Call release with XB=1 to queue the IO into the abort list. */ lpfc_release_nvme_buf(phba, lpfc_ncmd); } @@ -2064,9 +2081,6 @@ lpfc_get_nvme_buf(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp) spin_lock_irqsave(&phba->nvme_buf_list_get_lock, iflag); list_for_each_entry_safe(lpfc_ncmd, lpfc_ncmd_next, &phba->lpfc_nvme_buf_list_get, list) { - if (lpfc_test_rrq_active(phba, ndlp, - lpfc_ncmd->cur_iocbq.sli4_lxritag)) - continue; list_del_init(&lpfc_ncmd->list); found = 1; break; @@ -2079,9 +2093,6 @@ lpfc_get_nvme_buf(struct lpfc_hba *phba, struct lpfc_nodelist *ndlp) spin_unlock(&phba->nvme_buf_list_put_lock); list_for_each_entry_safe(lpfc_ncmd, lpfc_ncmd_next, &phba->lpfc_nvme_buf_list_get, list) { - if (lpfc_test_rrq_active( - phba, ndlp, lpfc_ncmd->cur_iocbq.sli4_lxritag)) - continue; list_del_init(&lpfc_ncmd->list); found = 1; break; @@ -2118,7 +2129,6 @@ lpfc_release_nvme_buf(struct lpfc_hba *phba, struct lpfc_nvme_buf *lpfc_ncmd) spin_lock_irqsave(&phba->sli4_hba.abts_nvme_buf_list_lock, iflag); - lpfc_ncmd->nvmeCmd = NULL; list_add_tail(&lpfc_ncmd->list, &phba->sli4_hba.lpfc_abts_nvme_buf_list); spin_unlock_irqrestore(&phba->sli4_hba.abts_nvme_buf_list_lock, @@ -2486,18 +2496,18 @@ lpfc_nvme_unregister_port(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) * @axri: pointer to the fcp xri abort wcqe structure. * * This routine is invoked by the worker thread to process a SLI4 fast-path - * FCP aborted xri. + * NVME aborted xri. Aborted NVME IO commands are completed to the transport + * here. **/ void lpfc_sli4_nvme_xri_aborted(struct lpfc_hba *phba, struct sli4_wcqe_xri_aborted *axri) { uint16_t xri = bf_get(lpfc_wcqe_xa_xri, axri); - uint16_t rxid = bf_get(lpfc_wcqe_xa_remote_xid, axri); struct lpfc_nvme_buf *lpfc_ncmd, *next_lpfc_ncmd; + struct nvmefc_fcp_req *nvme_cmd = NULL; struct lpfc_nodelist *ndlp; unsigned long iflag = 0; - int rrq_empty = 0; if (!(phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME)) return; @@ -2513,25 +2523,24 @@ lpfc_sli4_nvme_xri_aborted(struct lpfc_hba *phba, spin_unlock( &phba->sli4_hba.abts_nvme_buf_list_lock); - rrq_empty = list_empty(&phba->active_rrq_list); spin_unlock_irqrestore(&phba->hbalock, iflag); ndlp = lpfc_ncmd->ndlp; - if (ndlp) { - lpfc_set_rrq_active( - phba, ndlp, - lpfc_ncmd->cur_iocbq.sli4_lxritag, - rxid, 1); + if (ndlp) lpfc_sli4_abts_err_handler(phba, ndlp, axri); - } lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS, - "6311 XRI Aborted xri x%x tag x%x " - "released\n", - xri, lpfc_ncmd->cur_iocbq.iotag); - + "6311 nvme_cmd %p xri x%x tag x%x " + "abort complete and xri released\n", + lpfc_ncmd->nvmeCmd, xri, + lpfc_ncmd->cur_iocbq.iotag); + + /* Aborted NVME commands are required to not complete + * before the abort exchange command fully completes. + * Once completed, it is available via the put list. + */ + nvme_cmd = lpfc_ncmd->nvmeCmd; + nvme_cmd->done(nvme_cmd); lpfc_release_nvme_buf(phba, lpfc_ncmd); - if (rrq_empty) - lpfc_worker_wake_up(phba); return; } }