From patchwork Thu Jun 1 05:27:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oza Pawandeep X-Patchwork-Id: 9758673 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B00D660390 for ; Thu, 1 Jun 2017 05:28:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6990284FC for ; Thu, 1 Jun 2017 05:28:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 946AA284F1; Thu, 1 Jun 2017 05:28:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 10604284F1 for ; Thu, 1 Jun 2017 05:28:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751184AbdFAF2K (ORCPT ); Thu, 1 Jun 2017 01:28:10 -0400 Received: from mail-qk0-f172.google.com ([209.85.220.172]:33536 "EHLO mail-qk0-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751176AbdFAF2I (ORCPT ); Thu, 1 Jun 2017 01:28:08 -0400 Received: by mail-qk0-f172.google.com with SMTP id y201so28471169qka.0 for ; Wed, 31 May 2017 22:28:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=SbKfa/Rjk4HTmTLolOKyCAyDz8MrSyijKVVbJs+LNxY=; b=RjLRv/CmGoVFUqCPrHe+SI533MMJ96bdkCFZ1G9syGdCmLHGHVXz9F+jzcXONyFq/6 lmfnHgCLPlfQH5u6G5+vHb12gRxAHgueQrVlANNRUQ1pjvESPI7n1OJu2jwPYSPP8WBz 6m1yVQvqmvECIDbbNTnZLEnqjrjLPcZvf5miQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SbKfa/Rjk4HTmTLolOKyCAyDz8MrSyijKVVbJs+LNxY=; b=LojJVsGWv/xwPivlJVcrvBMiw8+PSl7eXm5eE0LNpELy2TgOkwR6aXkJbwqcXLl36+ mefo9VC8+5ckuTn21Y2QH+i8kbzEg678nixAZ/SchSKSL5td9rOWfgTt9zxXdkR/JLQU DJVUQLaZ2PCohJ6yLVNAVkDCEf4i2I0s7gAFlFp2GbfY7v5QytI5Xl2gVCtq0k/c6xlZ 9uILSr83VS4gtlVXAxsQr3sIu4QjJ/tYYbS/T1EHQkyNOb0USMAkNWtgQ85UXncdFwd7 5E6ug255i7Rk3f6nKPRSj6QYbDlk1UgEXXiQ0dPR5VOVOQgSceIm+EU9nET+gp7d28at IukQ== X-Gm-Message-State: AODbwcAjPPK+Yc2VaeFBwtAyUBV19IF8Fgj4dQRAyg7DK9bIklk+kf+y 1/XghSBgEwAGBVSC X-Received: by 10.55.58.67 with SMTP id h64mr13632455qka.9.1496294887019; Wed, 31 May 2017 22:28:07 -0700 (PDT) Received: from anjanavk-OptiPlex-7010.dhcp.avagotech.net ([192.19.237.250]) by smtp.gmail.com with ESMTPSA id d34sm12307026qta.59.2017.05.31.22.28.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 31 May 2017 22:28:06 -0700 (PDT) From: Oza Pawandeep To: Bjorn Helgaas , Ray Jui , Scott Branden , Jon Mason , bcm-kernel-feedback-list@broadcom.com, Oza Pawandeep , Andy Gospodarek , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Oza Pawandeep Subject: [PATCH 1/2] PCI: iproc: Retry request when CRS returned from EP Date: Thu, 1 Jun 2017 10:57:50 +0530 Message-Id: <1496294871-5836-2-git-send-email-oza.oza@broadcom.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1496294871-5836-1-git-send-email-oza.oza@broadcom.com> References: <1496294871-5836-1-git-send-email-oza.oza@broadcom.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For Configuration Requests only, following reset it is possible for a device to terminate the request but indicate that it is temporarily unable to process the Request, but will be able to process the Request in the future – in this case, the Configuration Request Retry Status 10 (CRS) Completion Status is used SPDK user space NVMe driver reinitializes NVMe which causes reset, while doing this some configuration requests get NAKed by Endpoint (NVMe). Current iproc PCI driver is agnostic about it. PAXB will forward the NAKed response in stipulated AXI code. NVMe spec defines this timeout in 500 ms units, and this only happens if controller has been in reset, or with new firmware, or in abrupt shutdown case. Meanwhile config access could result into retry. This patch fixes the problem, and attempts to read again in case of PAXB forwarding the NAK. It implements iproc_pcie_config_read which gets called for Stingray. Otherwise it falls back to PCI generic APIs. Signed-off-by: Oza Pawandeep Reviewed-by: Ray Jui Reviewed-by: Scott Branden diff --git a/drivers/pci/host/pcie-iproc.c b/drivers/pci/host/pcie-iproc.c index 0f39bd2..05a3647 100644 --- a/drivers/pci/host/pcie-iproc.c +++ b/drivers/pci/host/pcie-iproc.c @@ -68,6 +68,9 @@ #define APB_ERR_EN_SHIFT 0 #define APB_ERR_EN BIT(APB_ERR_EN_SHIFT) +#define CFG_RETRY_STATUS 0xffff0001 +#define CFG_RETRY_STATUS_TIMEOUT_US 500000 /* 500 milli-seconds. */ + /* derive the enum index of the outbound/inbound mapping registers */ #define MAP_REG(base_reg, index) ((base_reg) + (index) * 2) @@ -448,6 +451,47 @@ static inline void iproc_pcie_apb_err_disable(struct pci_bus *bus, } } +static int iproc_pcie_cfg_retry(void __iomem *cfg_data_p) +{ + int timeout = CFG_RETRY_STATUS_TIMEOUT_US; + unsigned int ret; + + do { + ret = readl(cfg_data_p); + if (ret == CFG_RETRY_STATUS) + udelay(1); + else + return PCIBIOS_SUCCESSFUL; + } while (timeout--); + + return PCIBIOS_DEVICE_NOT_FOUND; +} + +static void __iomem *iproc_pcie_map_ep_cfg_reg(struct iproc_pcie *pcie, + unsigned int busno, + unsigned int slot, + unsigned int fn, + int where) +{ + u16 offset; + u32 val; + + /* EP device access */ + val = (busno << CFG_ADDR_BUS_NUM_SHIFT) | + (slot << CFG_ADDR_DEV_NUM_SHIFT) | + (fn << CFG_ADDR_FUNC_NUM_SHIFT) | + (where & CFG_ADDR_REG_NUM_MASK) | + (1 & CFG_ADDR_CFG_TYPE_MASK); + + iproc_pcie_write_reg(pcie, IPROC_PCIE_CFG_ADDR, val); + offset = iproc_pcie_reg_offset(pcie, IPROC_PCIE_CFG_DATA); + + if (iproc_pcie_reg_is_invalid(offset)) + return NULL; + + return (pcie->base + offset); +} + /** * Note access to the configuration registers are protected at the higher layer * by 'pci_lock' in drivers/pci/access.c @@ -499,13 +543,48 @@ static void __iomem *iproc_pcie_map_cfg_bus(struct pci_bus *bus, return (pcie->base + offset); } +static int iproc_pcie_config_read(struct pci_bus *bus, unsigned int devfn, + int where, int size, u32 *val) +{ + struct iproc_pcie *pcie = iproc_data(bus); + unsigned int slot = PCI_SLOT(devfn); + unsigned int fn = PCI_FUNC(devfn); + unsigned int busno = bus->number; + void __iomem *cfg_data_p; + int ret; + + /* root complex access. */ + if (busno == 0) + return pci_generic_config_read32(bus, devfn, where, size, val); + + cfg_data_p = iproc_pcie_map_ep_cfg_reg(pcie, busno, slot, fn, where); + + if (!cfg_data_p) + return PCIBIOS_DEVICE_NOT_FOUND; + + ret = iproc_pcie_cfg_retry(cfg_data_p); + if (ret) + return ret; + + *val = readl(cfg_data_p); + + if (size <= 2) + *val = (*val >> (8 * (where & 3))) & ((1 << (size * 8)) - 1); + + return PCIBIOS_SUCCESSFUL; +} + static int iproc_pcie_config_read32(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *val) { int ret; + struct iproc_pcie *pcie = iproc_data(bus); iproc_pcie_apb_err_disable(bus, true); - ret = pci_generic_config_read32(bus, devfn, where, size, val); + if (pcie->type == IPROC_PCIE_PAXB_V2) + ret = iproc_pcie_config_read(bus, devfn, where, size, val); + else + ret = pci_generic_config_read32(bus, devfn, where, size, val); iproc_pcie_apb_err_disable(bus, false); return ret;