From patchwork Fri Dec 11 12:15:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Thompson X-Patchwork-Id: 11968025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7675AC4361B for ; Fri, 11 Dec 2020 12:16:44 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A7A522D2A for ; Fri, 11 Dec 2020 12:16:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A7A522D2A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=8gGVHosGTFBQxzwhqLWbDYQp1AxsktAKioxw239rExo=; b=uqA/GV+PBmlEfwvwQOhsdA31IL CGlL5nnaj8gm5iO9RdKsQKJ2ZUs6zZNOjGBpZxef4qxA4yfRrYQ+sNQaYGvxK9t7HcCQnHD8vcvKq 4mOyi0P2AzZYg2yPaubDrPxgxqOveJk7gobkyqoWIEcEwnGpSvyxnyAmxy9y3p8c2zSi7PZQRsdha 1wxCFGEh4f2X1rgElkAWPGKn/UGc/yUz7zZEy0DzJ+0e85NkXpi/ELJCEsxz0SKVkdyPZKW69Vzom rx0c13YeFA+NgGy8MFPpwaVucU1fgW7aWLrsB+szc8RrlGfu4SCDkVQQ/jX8gPpjE/Ugkfg8f0onb hs9kYfwg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1knhKO-0000Jk-8Q; Fri, 11 Dec 2020 12:15:16 +0000 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1knhKL-0000FX-Il for linux-arm-kernel@lists.infradead.org; Fri, 11 Dec 2020 12:15:14 +0000 Received: by mail-wm1-x343.google.com with SMTP id g25so4571407wmh.1 for ; Fri, 11 Dec 2020 04:15:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qSQYi3dLtVwnUE7Pl/TP3BDVgqP8TWHMbxJBiJCII0E=; b=Dx5LivaHRPPR2HXGN6NhnuCdmBQJvKcZWNlLrTLzZFJwcB6RzJfmUfHHX4C4GRDH+l Fcl2VrSCqpI3weTIOmxR4ST1Ab+nn2xscu6tLq74x6PXQBQG65J05JwnSJmobYeE9qJM dVtzrRd4Izqw/X9RQc7u7y0Gf07xW1suebY6PqHPbvgQpw/9xyyNskglhVAb3CQtCOJJ yvkfzspi1t+d/nNAM30Lby/YgWlVGRWwtFu2aJkl7Qkl4q0eInKA1+hgGxL3Zq2Yij0n ylb9TP1E0wB3Gi46R4nOboIrRT4ILrik9FL+7bvCj7CtuSdXcbyObivBqO36KuGP9+qn d1Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qSQYi3dLtVwnUE7Pl/TP3BDVgqP8TWHMbxJBiJCII0E=; b=XEaiy2nV0KXvokH0JI4vxEZhH74T3A3/bs2ACMgrqYYGc710TaNRNLvYLb7ruy6P8A 9zPewOGgHWB7SSGQRUJ7Ig2G0dYtdE6I9MGoNtsrrnB2mly64B20e3hl6g/FxKsBy1I6 yglL4dcG8uxHMFYJoVTI+yFYQjLJ/p1wZ4R6d4BO9P8S/TscjVj0BtrOJcmCWzOa0hzE zZUtxgXqmbFC7pA8cQr2ljYy9Rc5pX8SW+XIBmze4UM99hlTSVjcLBF0EVWOKW6lHil8 IcTmSB6XCtXTmS+KcXgRilE4OQeVrDfX+rp1gu0QSElc89amYnN5zqbUitdmN6UIfZPq nSxg== X-Gm-Message-State: AOAM532bMSlqobMQfp6I70l4Pgi5740AAn/3EyV65kpJmDpW++ToCw8C ADN5FGxx6eS7kOUR4CMv7lmiVJl7sRH7Sw== X-Google-Smtp-Source: ABdhPJzOVm94SMfkeQ9hfL9CF7fkbPYJoEd7mWVHor1srR0OHVa66W4FFAeAs+pfP5iEXHtZs3DlKw== X-Received: by 2002:a7b:cc12:: with SMTP id f18mr13429738wmh.110.1607688910750; Fri, 11 Dec 2020 04:15:10 -0800 (PST) Received: from maple.lan (cpc141216-aztw34-2-0-cust174.18-1.cable.virginm.net. [80.7.220.175]) by smtp.gmail.com with ESMTPSA id s63sm16668115wms.18.2020.12.11.04.15.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Dec 2020 04:15:10 -0800 (PST) From: Daniel Thompson To: Minghuan Lian , Mingkai Hu , Roy Zang Subject: [RFC HACK PATCH] PCI: dwc: layerscape: Hack around enumeration problems with Honeycomb LX2K Date: Fri, 11 Dec 2020 12:15:07 +0000 Message-Id: <20201211121507.28166-1-daniel.thompson@linaro.org> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201211_071513_751821_EC843C72 X-CRM114-Status: GOOD ( 21.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Herring , Daniel Thompson , Lorenzo Pieralisi , patches@linaro.org, linux-pci@vger.kernel.org, Jon Nettleton , linux-kernel@vger.kernel.org, Bjorn Helgaas , linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org I have been chasing down a problem enumerating an NVMe drive on a Honeycomb LX2K (NXP LX2160A). Specifically the drive can only enumerate successfully if the we are emitting lots of console messages via a UART. If the system is booted with `quiet` set then enumeration fails. I guessed this would be due to the timing impact of printk-to-UART and tried to find out where a delay could be added to provoke a successful enumeration. This patch contains the results. The delay length (1ms) was selected by binary searching downwards until the delay was not effective for these devices (Honeycomb LX2K and a Western Digital WD Blue SN550). I have also included the workaround twice (conditionally compiled). The first change is the *latest* possible code path that we can deploy a delay whilst the second is the earliest place I could find. The summary is that the critical window were we are currently relying on a call to the console UART code can "mend" the driver runs from calling dw_pcie_setup_rc() in host init to just before we read the state in the link up callback. Signed-off-by: Daniel Thompson --- Notes: This patch is RFC (and HACK) because I don't have much clue *why* this patch works... merely that this is the smallest possible change I need to replicate the current accidental printk() workaround. Perhaps one could argue that RFC here stands for request-for-clue. All my observations and changes here are empirical and I don't know how best to turn them into something that is not a hack! BTW I noticed many other pcie-designware drivers take advantage of a function called dw_pcie_wait_for_link() in their init paths... but my naive attempts to add it to the layerscape driver results in non-booting systems so I haven't embarrassed myself by including that in the patch! drivers/pci/controller/dwc/pci-layerscape.c | 35 +++++++++++++++++++++ 1 file changed, 35 insertions(+) base-commit: 0477e92881850d44910a7e94fc2c46f96faa131f -- 2.29.2 diff --git a/drivers/pci/controller/dwc/pci-layerscape.c b/drivers/pci/controller/dwc/pci-layerscape.c index f24f79a70d9a..c354904b90ef 100644 --- a/drivers/pci/controller/dwc/pci-layerscape.c +++ b/drivers/pci/controller/dwc/pci-layerscape.c @@ -22,6 +22,8 @@ #include "pcie-designware.h" +#define WORKAROUND_LATEST_POSSIBLE + /* PEX1/2 Misc Ports Status Register */ #define SCFG_PEXMSCPORTSR(pex_idx) (0x94 + (pex_idx) * 4) #define LTSSM_STATE_SHIFT 20 @@ -113,10 +115,31 @@ static int ls_pcie_link_up(struct dw_pcie *pci) struct ls_pcie *pcie = to_ls_pcie(pci); u32 state; + /* + * Strictly speaking *this* (before the ioread32) is the latest + * point a simple delay can be effective. If we move the delay + * after the ioread32 then the NVMe does not enumerate. + * + * However this function appears to be frequently called so an + * unconditional delay here causes noticeable delay at boot + * time. Hence we implement the workaround by retrying the read + * after a short delay if we think we might need to return false. + */ + state = (ioread32(pcie->lut + pcie->drvdata->lut_dbg) >> pcie->drvdata->ltssm_shift) & LTSSM_STATE_MASK; +#ifdef WORKAROUND_LATEST_POSSIBLE + if (state < LTSSM_PCIE_L0) { + /* see comment above */ + mdelay(1); + state = (ioread32(pcie->lut + pcie->drvdata->lut_dbg) >> + pcie->drvdata->ltssm_shift) & + LTSSM_STATE_MASK; + } +#endif + if (state < LTSSM_PCIE_L0) return 0; @@ -152,6 +175,18 @@ static int ls_pcie_host_init(struct pcie_port *pp) dw_pcie_setup_rc(pp); +#ifdef WORKAROUND_EARLIEST_POSSIBLE + /* + * This is the earliest point the delay is effective. + * If we move it before dw_pcie_setup_rc() then the + * NVMe does not enumerate. + * + * 500us is too short to reliably work around the issue + * hence adopting 1000us here. + */ + mdelay(1); +#endif + return 0; }