From patchwork Tue Jul 23 09:25:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054189 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F087E1580 for ; Tue, 23 Jul 2019 09:26:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE7121FFBE for ; Tue, 23 Jul 2019 09:26:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D2A1526E51; Tue, 23 Jul 2019 09:26:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 590AA26E47 for ; Tue, 23 Jul 2019 09:26:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387820AbfGWJZ4 (ORCPT ); Tue, 23 Jul 2019 05:25:56 -0400 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:59453 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726432AbfGWJZy (ORCPT ); Tue, 23 Jul 2019 05:25:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563873953; x=1595409953; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=c0w4B2FUQGPRXAvRw5cgS9W93te+n8egQX1yzR84ahs=; b=jfLp8S1wAkKnWy+HKPWRd+4Sixe0A8gKM6SGuwLesdfSGeBEoP2BoBkC PJ2RclM2hxx2HWtHmILHW6T3x7HuapkXohChESrVFoCTRHUg1ccDGsRsD YvPs++io2T4jhB/82fSuJ2lwHSlq7WmUuhheYWahLeOZMkrVCDgQrx5Cf w=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="406177491" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-d0be17ee.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 23 Jul 2019 09:25:51 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-d0be17ee.us-west-2.amazon.com (Postfix) with ESMTPS id B10B4A2629; Tue, 23 Jul 2019 09:25:50 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:25:50 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.160.245) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:25:45 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 1/8] PCI: Add Amazon's Annapurna Labs vendor ID Date: Tue, 23 Jul 2019 12:25:26 +0300 Message-ID: <20190723092529.11310-2-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.245] X-ClientProxiedBy: EX13D24UWB004.ant.amazon.com (10.43.161.4) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add Amazon's Annapurna Labs vendor ID to pci_ids.h. Signed-off-by: Jonathan Chocron Acked-by: Bjorn Helgaas Reviewed-by: Andrew Murray --- include/linux/pci_ids.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h index 40015609c4b5..63dfa4bace57 100644 --- a/include/linux/pci_ids.h +++ b/include/linux/pci_ids.h @@ -2569,6 +2569,8 @@ #define PCI_VENDOR_ID_ASMEDIA 0x1b21 +#define PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS 0x1c36 + #define PCI_VENDOR_ID_CIRCUITCO 0x1cc8 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD 0x0001 From patchwork Tue Jul 23 09:25:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054183 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7FB9912 for ; Tue, 23 Jul 2019 09:26:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5F9E1FFBE for ; Tue, 23 Jul 2019 09:26:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9089D26E47; Tue, 23 Jul 2019 09:26:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F26211FFBE for ; Tue, 23 Jul 2019 09:26:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387885AbfGWJZ7 (ORCPT ); Tue, 23 Jul 2019 05:25:59 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:11924 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387874AbfGWJZ7 (ORCPT ); Tue, 23 Jul 2019 05:25:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563873957; x=1595409957; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=1lQKq3hxKQkr7WB4YDlui/W/5OP1jSDsOKg2GvSonVs=; b=S/zNy/O6LcFd2tP9auIHemYbpGbSu1mRJC3Ynn1RpJB2/LuTLS610Ko8 iBGgVB4b1L2V85QSBCrUIkCahp/bamHRajhu9VSwbC07gUdkwQCJEjny4 If2jbaEbLuw5GSmje8aCSeBlVSgl7Dqmte9A1kH2dIH3Yci3Al7C8BXDW o=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="812905024" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 23 Jul 2019 09:25:55 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com (Postfix) with ESMTPS id 8BBFAA293F; Tue, 23 Jul 2019 09:25:55 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:25:55 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.160.245) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:25:50 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 2/8] PCI: Add ACS quirk for Amazon Annapurna Labs root ports Date: Tue, 23 Jul 2019 12:25:27 +0300 Message-ID: <20190723092529.11310-3-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.245] X-ClientProxiedBy: EX13D24UWB004.ant.amazon.com (10.43.161.4) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Ali Saidi The Amazon's Annapurna Labs root ports don't advertise an ACS capability, but they don't allow peer-to-peer transactions and do validate bus numbers through the SMMU. Additionally, it's not possible for one RP to pass traffic to another RP. Signed-off-by: Ali Saidi Signed-off-by: Jonathan Chocron Reviewed-by: Gustavo Pimentel --- drivers/pci/quirks.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 208aacf39329..23672680dba7 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -4366,6 +4366,23 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags) return ret; } +static int pci_quirk_al_acs(struct pci_dev *dev, u16 acs_flags) +{ + /* + * Amazon's Annapurna Labs root ports don't include an ACS capability, + * but do include ACS-like functionality. The hardware doesn't support + * peer-to-peer transactions via the root port and each has a unique + * segment number. + * Additionally, the root ports cannot send traffic to each other. + */ + acs_flags &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_SV | PCI_ACS_UF); + + if (pci_pcie_type(dev) != PCI_EXP_TYPE_ROOT_PORT) + return -ENOTTY; + + return acs_flags ? 0 : 1; +} + /* * Sunrise Point PCH root ports implement ACS, but unfortunately as shown in * the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2, @@ -4559,6 +4576,8 @@ static const struct pci_dev_acs_enabled { { PCI_VENDOR_ID_AMPERE, 0xE00A, pci_quirk_xgene_acs }, { PCI_VENDOR_ID_AMPERE, 0xE00B, pci_quirk_xgene_acs }, { PCI_VENDOR_ID_AMPERE, 0xE00C, pci_quirk_xgene_acs }, + /* Amazon Annapurna Labs */ + { PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, pci_quirk_al_acs }, { 0 } }; From patchwork Tue Jul 23 09:25:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054193 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BF38912 for ; Tue, 23 Jul 2019 09:26:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B98C1FFBE for ; Tue, 23 Jul 2019 09:26:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5FDF126E51; Tue, 23 Jul 2019 09:26:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07DEE1FFBE for ; Tue, 23 Jul 2019 09:26:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387895AbfGWJ0H (ORCPT ); Tue, 23 Jul 2019 05:26:07 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:35518 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726397AbfGWJ0H (ORCPT ); Tue, 23 Jul 2019 05:26:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563873967; x=1595409967; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=yPw/cZErk19SmOK/njpJRQGVQoT+U5XdF/Qgj3DD9Ng=; b=IQaFz+RariQRRupHd74Covvcfp0e9lMtaq41vJAsn05u8xq254kGNnuO 45qYW/+PE/sJlilgCc1729fUilToWGiDWeEDdaUVOHmDZYC1n1/0uBbpU 81ywdl5kP8jc3m5A8L0feuM7V72/I0qn138jVv1ACjOk1ZuzsVoXrlEOT Y=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="817888383" Received: from sea3-co-svc-lb6-vlan2.sea.amazon.com (HELO email-inbound-relay-2c-397e131e.us-west-2.amazon.com) ([10.47.22.34]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 23 Jul 2019 09:26:01 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-397e131e.us-west-2.amazon.com (Postfix) with ESMTPS id CE906A28CD; Tue, 23 Jul 2019 09:26:00 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:26:00 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.160.245) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:25:55 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 3/8] PCI/VPD: Add VPD release quirk for Amazon's Annapurna Labs Root Port Date: Tue, 23 Jul 2019 12:25:28 +0300 Message-ID: <20190723092529.11310-4-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.245] X-ClientProxiedBy: EX13D24UWB004.ant.amazon.com (10.43.161.4) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The Amazon Annapurna Labs PCIe Root Port exposes the VPD capability, but there is no actual support for it. The reason for not using the already existing quirk_blacklist_vpd() is that, although this fails pci_vpd_read/write, the 'vpd' sysfs entry still exists. When running lspci -vv, for example, this results in the following error: pcilib: sysfs_read_vpd: read failed: Input/output error This quirk removes the sysfs entry, which avoids the error print. Signed-off-by: Jonathan Chocron Reviewed-by: Gustavo Pimentel --- drivers/pci/vpd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/drivers/pci/vpd.c b/drivers/pci/vpd.c index 4963c2e2bd4c..c23a8ec08db9 100644 --- a/drivers/pci/vpd.c +++ b/drivers/pci/vpd.c @@ -644,4 +644,20 @@ static void quirk_chelsio_extend_vpd(struct pci_dev *dev) DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID, quirk_chelsio_extend_vpd); +static void quirk_al_vpd_release(struct pci_dev *dev) +{ + if (dev->vpd) { + pci_vpd_release(dev); + dev->vpd = NULL; + pci_warn(dev, FW_BUG "Releasing VPD capability (No support for VPD read/write transactions)\n"); + } +} + +/* + * The 0031 device id is reused for other non Root Port device types, + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class. + */ +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, + PCI_CLASS_BRIDGE_PCI, 8, quirk_al_vpd_release); + #endif From patchwork Tue Jul 23 09:25:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054197 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83C711399 for ; Tue, 23 Jul 2019 09:26:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73B801FFBE for ; Tue, 23 Jul 2019 09:26:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6761526E47; Tue, 23 Jul 2019 09:26:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D58731FFBE for ; Tue, 23 Jul 2019 09:26:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387921AbfGWJ0J (ORCPT ); Tue, 23 Jul 2019 05:26:09 -0400 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:40507 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387874AbfGWJ0I (ORCPT ); Tue, 23 Jul 2019 05:26:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563873967; x=1595409967; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=2GJlucczclkdq9MNyN1luyz8JSNTe9FaZ7yShbzyobA=; b=kXkFul/CZfT056i6s1XzC6GPmnTFgUg6uflrFkeNGMI4ClYCxiYp+Kwv 5MPEeENYNhhHFlJzjwblMsqhhuTOwK3R8tfQJWW679eWUQcCx1D5P7N+C +KqoEN89exVDLOl4/XlQj1xn/W9UAjxIYcYLPNKUo5kHJC0N8Hpz4MhEm 0=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="406177514" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 23 Jul 2019 09:26:06 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com (Postfix) with ESMTPS id 7748BA20A4; Tue, 23 Jul 2019 09:26:05 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:26:05 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.160.245) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:25:59 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 4/8] PCI: Add quirk to disable MSI-X support for Amazon's Annapurna Labs Root Port Date: Tue, 23 Jul 2019 12:25:29 +0300 Message-ID: <20190723092529.11310-5-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.245] X-ClientProxiedBy: EX13D24UWB004.ant.amazon.com (10.43.161.4) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The Root Port (identified by [1c36:0032]) doesn't support MSI-X. On some platforms it is configured to not advertise the capability at all, while on others it (mistakenly) does. This causes a panic during initialization by the pcieport driver, since it tries to configure the MSI-X capability. Specifically, when trying to access the MSI-X table a "non-existing addr" exception occurs. Example stacktrace snippet: [ 1.632363] SError Interrupt on CPU2, code 0xbf000000 -- SError [ 1.632364] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33 [ 1.632365] Hardware name: Annapurna Labs Alpine V3 EVP (DT) [ 1.632365] pstate: 80000005 (Nzcv daif -PAN -UAO) [ 1.632366] pc : __pci_enable_msix_range+0x4e4/0x608 [ 1.632367] lr : __pci_enable_msix_range+0x498/0x608 [ 1.632367] sp : ffffff80117db700 [ 1.632368] x29: ffffff80117db700 x28: 0000000000000001 [ 1.632370] x27: 0000000000000001 x26: 0000000000000000 [ 1.632372] x25: ffffffd3e9d8c0b0 x24: 0000000000000000 [ 1.632373] x23: 0000000000000000 x22: 0000000000000000 [ 1.632375] x21: 0000000000000001 x20: 0000000000000000 [ 1.632376] x19: ffffffd3e9d8c000 x18: ffffffffffffffff [ 1.632378] x17: 0000000000000000 x16: 0000000000000000 [ 1.632379] x15: ffffff80116496c8 x14: ffffffd3e9844503 [ 1.632380] x13: ffffffd3e9844502 x12: 0000000000000038 [ 1.632382] x11: ffffffffffffff00 x10: 0000000000000040 [ 1.632384] x9 : ffffff801165e270 x8 : ffffff801165e268 [ 1.632385] x7 : 0000000000000002 x6 : 00000000000000b2 [ 1.632387] x5 : ffffffd3e9d8c2c0 x4 : 0000000000000000 [ 1.632388] x3 : 0000000000000000 x2 : 0000000000000000 [ 1.632390] x1 : 0000000000000000 x0 : ffffffd3e9844680 [ 1.632392] Kernel panic - not syncing: Asynchronous SError Interrupt [ 1.632393] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc1-Jonny-14847-ge76f1d4a1828-dirty #33 [ 1.632394] Hardware name: Annapurna Labs Alpine V3 EVP (DT) [ 1.632394] Call trace: [ 1.632395] dump_backtrace+0x0/0x140 [ 1.632395] show_stack+0x14/0x20 [ 1.632396] dump_stack+0xa8/0xcc [ 1.632396] panic+0x140/0x334 [ 1.632397] nmi_panic+0x6c/0x70 [ 1.632398] arm64_serror_panic+0x74/0x88 [ 1.632398] __pte_error+0x0/0x28 [ 1.632399] el1_error+0x84/0xf8 [ 1.632400] __pci_enable_msix_range+0x4e4/0x608 [ 1.632400] pci_alloc_irq_vectors_affinity+0xdc/0x150 [ 1.632401] pcie_port_device_register+0x2b8/0x4e0 [ 1.632402] pcie_portdrv_probe+0x34/0xf0 Signed-off-by: Jonathan Chocron Reviewed-by: Gustavo Pimentel --- drivers/pci/quirks.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 23672680dba7..11f843aa96b3 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -2925,6 +2925,21 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0x10a1, quirk_msi_intx_disable_qca_bug); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_ATTANSIC, 0xe091, quirk_msi_intx_disable_qca_bug); + +/* + * Amazon's Annapurna Labs 1c36:0031 Root Ports don't support MSI-X, so it + * should be disabled on platforms where the device (mistakenly) advertises it. + * + * The 0031 device id is reused for other non Root Port device types, + * therefore the quirk is registered for the PCI_CLASS_BRIDGE_PCI class. + */ +static void quirk_al_msi_disable(struct pci_dev *dev) +{ + dev->no_msi = 1; + pci_warn(dev, "Disabling MSI-X\n"); +} +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMAZON_ANNAPURNA_LABS, 0x0031, + PCI_CLASS_BRIDGE_PCI, 8, quirk_al_msi_disable); #endif /* CONFIG_PCI_MSI */ /* From patchwork Tue Jul 23 09:27:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054199 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0BE2A1399 for ; Tue, 23 Jul 2019 09:27:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF85A1FFBE for ; Tue, 23 Jul 2019 09:27:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E3A2426E47; Tue, 23 Jul 2019 09:27:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 844061FFBE for ; Tue, 23 Jul 2019 09:27:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728031AbfGWJ1b (ORCPT ); Tue, 23 Jul 2019 05:27:31 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:52938 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727408AbfGWJ1a (ORCPT ); Tue, 23 Jul 2019 05:27:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563874050; x=1595410050; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=b531SE6MRYpUt8mqMq6kBzbDUygX4l0/zWJlNrWTyTY=; b=BvJfvYvwcgjfN2d6+V5bI632bWVUO3FhL9XJlC/xiYyJfjz3hKklpTPt SWj6akCCMQGut0ommegz8xfJIohQGko6a3wHKF4QPJlOxvMzeDOedJAuJ A8Duypw1EZ/RKXAAV1Tzh/fDOz9mDBwz31Vb6unDl2TNPyFBk0DDt9NgQ c=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="775794056" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 23 Jul 2019 09:27:28 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com (Postfix) with ESMTPS id CF43DA272C; Tue, 23 Jul 2019 09:27:27 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:27 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.161.85) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:22 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 5/8] dt-bindings: PCI: Add Amazon's Annapurna Labs PCIe host bridge binding Date: Tue, 23 Jul 2019 12:27:08 +0300 Message-ID: <20190723092711.11786-1-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.85] X-ClientProxiedBy: EX13D27UWB003.ant.amazon.com (10.43.161.195) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Document Amazon's Annapurna Labs PCIe host bridge. Signed-off-by: Jonathan Chocron --- .../devicetree/bindings/pci/pcie-al.txt | 45 +++++++++++++++++++ MAINTAINERS | 3 +- 2 files changed, 47 insertions(+), 1 deletion(-) create mode 100644 Documentation/devicetree/bindings/pci/pcie-al.txt diff --git a/Documentation/devicetree/bindings/pci/pcie-al.txt b/Documentation/devicetree/bindings/pci/pcie-al.txt new file mode 100644 index 000000000000..89876190eb5a --- /dev/null +++ b/Documentation/devicetree/bindings/pci/pcie-al.txt @@ -0,0 +1,45 @@ +* Amazon Annapurna Labs PCIe host bridge + +Amazon's Annapurna Labs PCIe Host Controller is based on the Synopsys DesignWare +PCI core. +It shares common functions with the PCIe DesignWare core driver and inherits +common properties defined in Documentation/devicetree/bindings/pci/designware-pcie.txt. +Properties of the host controller node that differ from it are: + +- compatible: + Usage: required + Value type: + Definition: Value should contain + - "amazon,al-pcie" + +- reg: + Usage: required + Value type: + Definition: Register ranges as listed in the reg-names property + +- reg-names: + Usage: required + Value type: + Definition: Must include the following entries + - "config" PCIe ECAM space + - "controller" AL proprietary registers + - "dbi" Designware PCIe registers + +Example: + + pcie-external0: pcie@fb600000 { + compatible = "amazon,al-pcie"; + reg = <0x0 0xfb600000 0x0 0x00100000 + 0x0 0xfd800000 0x0 0x00010000 + 0x0 0xfd810000 0x0 0x00001000>; + reg-names = "config", "controller", "dbi"; + bus-range = <0 255>; + device_type = "pci"; + #address-cells = <3>; + #size-cells = <2>; + #interrupt-cells = <1>; + interrupts = ; + interrupt-map-mask = <0x00 0 0 7>; + interrupt-map = <0x0000 0 0 1 &gic GIC_SPI 41 IRQ_TYPE_LEVEL_HIGH>; /* INTa */ + ranges = <0x02000000 0x0 0xc0010000 0x0 0xc0010000 0x0 0x07ff0000>; + }; diff --git a/MAINTAINERS b/MAINTAINERS index 5a6137df3f0e..29cca14a05a6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12201,10 +12201,11 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/ S: Supported F: drivers/pci/controller/ -PCIE DRIVER FOR ANNAPURNA LABS +PCIE DRIVER FOR AMAZON ANNAPURNA LABS M: Jonathan Chocron L: linux-pci@vger.kernel.org S: Maintained +F: Documentation/devicetree/bindings/pci/pcie-al.txt F: drivers/pci/controller/dwc/pcie-al.c PCIE DRIVER FOR AMLOGIC MESON From patchwork Tue Jul 23 09:27:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054203 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF716912 for ; Tue, 23 Jul 2019 09:27:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE53326E47 for ; Tue, 23 Jul 2019 09:27:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C236028372; Tue, 23 Jul 2019 09:27:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E625126E47 for ; Tue, 23 Jul 2019 09:27:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729969AbfGWJ1f (ORCPT ); Tue, 23 Jul 2019 05:27:35 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:21677 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727408AbfGWJ1e (ORCPT ); Tue, 23 Jul 2019 05:27:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563874054; x=1595410054; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=JoLVQV5FUknmMd598i8M4m89lRFK0Bb4+obRxzGMdGk=; b=F0GSMOyuyQcetupdC2cX4zwW+198gVvYuxOyX0lSmfgdM/fFgNPZsg0/ HgCVjFUds/p4f67U9261QPW6VQ0mpcnct+JxAPzkDmwffHZ9VT3eN7EfG 0nk3IkwdmFcLEzTL5XejflNk2GA0r/y3WMdsxXON5Y+kdQug8pmjXPvtx U=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="812905261" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-2b-c300ac87.us-west-2.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 23 Jul 2019 09:27:33 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-c300ac87.us-west-2.amazon.com (Postfix) with ESMTPS id 060F8A067E; Tue, 23 Jul 2019 09:27:32 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:32 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.161.85) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:27 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 6/8] PCI: al: Add support for DW based driver type Date: Tue, 23 Jul 2019 12:27:09 +0300 Message-ID: <20190723092711.11786-2-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.85] X-ClientProxiedBy: EX13D27UWB003.ant.amazon.com (10.43.161.195) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This driver is DT based and utilizes the DesignWare APIs. It allows using a smaller ECAM range for a larger bus range - usually an entire bus uses 1MB of address space, but the driver can use it for a larger number of buses. All link initializations are handled by the boot FW. Signed-off-by: Jonathan Chocron Reviewed-by: Gustavo Pimentel --- drivers/pci/controller/dwc/Kconfig | 12 + drivers/pci/controller/dwc/pcie-al.c | 367 +++++++++++++++++++++++++++ 2 files changed, 379 insertions(+) diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig index 6ea778ae4877..3c6094cbcc3b 100644 --- a/drivers/pci/controller/dwc/Kconfig +++ b/drivers/pci/controller/dwc/Kconfig @@ -230,4 +230,16 @@ config PCIE_UNIPHIER Say Y here if you want PCIe controller support on UniPhier SoCs. This driver supports LD20 and PXs3 SoCs. +config PCIE_AL + bool "Amazon Annapurna Labs PCIe controller" + depends on OF && (ARM64 || COMPILE_TEST) + depends on PCI_MSI_IRQ_DOMAIN + select PCIE_DW_HOST + help + Say Y here to enable support of the Amazon's Annapurna Labs PCIe + controller IP on Amazon SoCs. The PCIe controller uses the DesignWare + core plus Annapurna Labs proprietary hardware wrappers. This is + required only for DT-based platforms. ACPI platforms with the + Annapurna Labs PCIe controller don't need to enable this. + endmenu diff --git a/drivers/pci/controller/dwc/pcie-al.c b/drivers/pci/controller/dwc/pcie-al.c index 3ab58f0584a8..3ffdd3c97617 100644 --- a/drivers/pci/controller/dwc/pcie-al.c +++ b/drivers/pci/controller/dwc/pcie-al.c @@ -91,3 +91,370 @@ struct pci_ecam_ops al_pcie_ops = { }; #endif /* defined(CONFIG_ACPI) && defined(CONFIG_PCI_QUIRKS) */ + +#ifdef CONFIG_PCIE_AL + +#include +#include "pcie-designware.h" + +#define AL_PCIE_REV_ID_2 2 +#define AL_PCIE_REV_ID_3 3 +#define AL_PCIE_REV_ID_4 4 + +#define AXI_BASE_OFFSET 0x0 + +#define DEVICE_ID_OFFSET 0x16c + +#define DEVICE_REV_ID 0x0 +#define DEVICE_REV_ID_DEV_ID_MASK GENMASK(31, 16) + +#define DEVICE_REV_ID_DEV_ID_X4 0 +#define DEVICE_REV_ID_DEV_ID_X8 2 +#define DEVICE_REV_ID_DEV_ID_X16 4 + +#define OB_CTRL_REV1_2_OFFSET 0x0040 +#define OB_CTRL_REV3_5_OFFSET 0x0030 + +#define CFG_TARGET_BUS 0x0 +#define CFG_TARGET_BUS_MASK_MASK GENMASK(7, 0) +#define CFG_TARGET_BUS_BUSNUM_MASK GENMASK(15, 8) + +#define CFG_CONTROL 0x4 +#define CFG_CONTROL_SUBBUS_MASK GENMASK(15, 8) +#define CFG_CONTROL_SEC_BUS_MASK GENMASK(23, 16) + +struct al_pcie_reg_offsets { + unsigned int ob_ctrl; +}; + +struct al_pcie_target_bus_cfg { + u8 reg_val; + u8 reg_mask; + u8 ecam_mask; +}; + +struct al_pcie { + struct dw_pcie *pci; + void __iomem *controller_base; /* base of PCIe unit (not DW core) */ + struct device *dev; + resource_size_t ecam_size; + unsigned int controller_rev_id; + struct al_pcie_reg_offsets reg_offsets; + struct al_pcie_target_bus_cfg target_bus_cfg; +}; + +#define PCIE_ECAM_DEVFN(x) (((x) & 0xff) << 12) + +#define to_al_pcie(x) dev_get_drvdata((x)->dev) + +static inline u32 al_pcie_controller_readl(struct al_pcie *pcie, u32 offset) +{ + return readl(pcie->controller_base + offset); +} + +static inline void al_pcie_controller_writel(struct al_pcie *pcie, u32 offset, + u32 val) +{ + writel(val, pcie->controller_base + offset); +} + +static int al_pcie_rev_id_get(struct al_pcie *pcie, unsigned int *rev_id) +{ + u32 dev_rev_id_val; + u32 dev_id_val; + + dev_rev_id_val = al_pcie_controller_readl(pcie, AXI_BASE_OFFSET + + DEVICE_ID_OFFSET + + DEVICE_REV_ID); + dev_id_val = FIELD_GET(DEVICE_REV_ID_DEV_ID_MASK, dev_rev_id_val); + + switch (dev_id_val) { + case DEVICE_REV_ID_DEV_ID_X4: + *rev_id = AL_PCIE_REV_ID_2; + break; + case DEVICE_REV_ID_DEV_ID_X8: + *rev_id = AL_PCIE_REV_ID_3; + break; + case DEVICE_REV_ID_DEV_ID_X16: + *rev_id = AL_PCIE_REV_ID_4; + break; + default: + dev_err(pcie->dev, "Unsupported dev_id_val (0x%x)\n", + dev_id_val); + return -EINVAL; + } + + dev_dbg(pcie->dev, "dev_id_val: 0x%x\n", dev_id_val); + + return 0; +} + +static int al_pcie_reg_offsets_set(struct al_pcie *pcie) +{ + switch (pcie->controller_rev_id) { + case AL_PCIE_REV_ID_2: + pcie->reg_offsets.ob_ctrl = OB_CTRL_REV1_2_OFFSET; + break; + case AL_PCIE_REV_ID_3: + case AL_PCIE_REV_ID_4: + pcie->reg_offsets.ob_ctrl = OB_CTRL_REV3_5_OFFSET; + break; + default: + dev_err(pcie->dev, "Unsupported controller rev_id: 0x%x\n", + pcie->controller_rev_id); + return -EINVAL; + } + + return 0; +} + +static inline void al_pcie_target_bus_set(struct al_pcie *pcie, + u8 target_bus, + u8 mask_target_bus) +{ + u32 reg; + + reg = FIELD_PREP(CFG_TARGET_BUS_MASK_MASK, mask_target_bus) | + FIELD_PREP(CFG_TARGET_BUS_BUSNUM_MASK, target_bus); + + al_pcie_controller_writel(pcie, AXI_BASE_OFFSET + + pcie->reg_offsets.ob_ctrl + CFG_TARGET_BUS, + reg); +} + +static void __iomem *al_pcie_conf_addr_map(struct al_pcie *pcie, + unsigned int busnr, + unsigned int devfn) +{ + struct al_pcie_target_bus_cfg *target_bus_cfg = &pcie->target_bus_cfg; + unsigned int busnr_ecam = busnr & target_bus_cfg->ecam_mask; + unsigned int busnr_reg = busnr & target_bus_cfg->reg_mask; + struct pcie_port *pp = &pcie->pci->pp; + void __iomem *pci_base_addr; + + pci_base_addr = (void __iomem *)((uintptr_t)pp->va_cfg0_base + + (busnr_ecam << 20) + + PCIE_ECAM_DEVFN(devfn)); + + if (busnr_reg != target_bus_cfg->reg_val) { + dev_dbg(pcie->pci->dev, "Changing target bus busnum val from 0x%x to 0x%x\n", + target_bus_cfg->reg_val, busnr_reg); + target_bus_cfg->reg_val = busnr_reg; + al_pcie_target_bus_set(pcie, + target_bus_cfg->reg_val, + target_bus_cfg->reg_mask); + } + + return pci_base_addr; +} + +static int al_pcie_rd_other_conf(struct pcie_port *pp, struct pci_bus *bus, + unsigned int devfn, int where, int size, + u32 *val) +{ + struct dw_pcie *pci = to_dw_pcie_from_pp(pp); + struct al_pcie *pcie = to_al_pcie(pci); + unsigned int busnr = bus->number; + void __iomem *pci_addr; + int rc; + + pci_addr = al_pcie_conf_addr_map(pcie, busnr, devfn); + + rc = dw_pcie_read(pci_addr + where, size, val); + + dev_dbg(pci->dev, "%d-byte config read from %04x:%02x:%02x.%d offset 0x%x (pci_addr: 0x%px) - val:0x%x\n", + size, pci_domain_nr(bus), bus->number, + PCI_SLOT(devfn), PCI_FUNC(devfn), where, + (pci_addr + where), *val); + + return rc; +} + +static int al_pcie_wr_other_conf(struct pcie_port *pp, struct pci_bus *bus, + unsigned int devfn, int where, int size, + u32 val) +{ + struct dw_pcie *pci = to_dw_pcie_from_pp(pp); + struct al_pcie *pcie = to_al_pcie(pci); + unsigned int busnr = bus->number; + void __iomem *pci_addr; + int rc; + + pci_addr = al_pcie_conf_addr_map(pcie, busnr, devfn); + + rc = dw_pcie_write(pci_addr + where, size, val); + + dev_err(pci->dev, "%d-byte config write to %04x:%02x:%02x.%d offset 0x%x (pci_addr: 0x%px) - val:0x%x\n", + size, pci_domain_nr(bus), bus->number, + PCI_SLOT(devfn), PCI_FUNC(devfn), where, + (pci_addr + where), val); + + return rc; +} + +static void al_pcie_config_prepare(struct al_pcie *pcie) +{ + struct al_pcie_target_bus_cfg *target_bus_cfg; + struct pcie_port *pp = &pcie->pci->pp; + unsigned int ecam_bus_mask; + u32 cfg_control_offset; + u8 subordinate_bus; + u8 secondary_bus; + u32 cfg_control; + u32 reg; + + target_bus_cfg = &pcie->target_bus_cfg; + + ecam_bus_mask = (pcie->ecam_size >> 20) - 1; + if (ecam_bus_mask > 255) { + dev_warn(pcie->dev, "ECAM window size is larger than 256MB. Cutting off at 256\n"); + ecam_bus_mask = 255; + } + + /* This portion is taken from the transaction address */ + target_bus_cfg->ecam_mask = ecam_bus_mask; + /* This portion is taken from the cfg_target_bus reg */ + target_bus_cfg->reg_mask = ~target_bus_cfg->ecam_mask; + target_bus_cfg->reg_val = pp->busn->start & target_bus_cfg->reg_mask; + + al_pcie_target_bus_set(pcie, target_bus_cfg->reg_val, + target_bus_cfg->reg_mask); + + secondary_bus = pp->busn->start + 1; + subordinate_bus = pp->busn->end; + + /* Set the valid values of secondary and subordinate buses */ + cfg_control_offset = AXI_BASE_OFFSET + pcie->reg_offsets.ob_ctrl + + CFG_CONTROL; + + cfg_control = al_pcie_controller_readl(pcie, cfg_control_offset); + + reg = cfg_control & + ~(CFG_CONTROL_SEC_BUS_MASK | CFG_CONTROL_SUBBUS_MASK); + + reg |= FIELD_PREP(CFG_CONTROL_SUBBUS_MASK, subordinate_bus) | + FIELD_PREP(CFG_CONTROL_SEC_BUS_MASK, secondary_bus); + + al_pcie_controller_writel(pcie, cfg_control_offset, reg); +} + +static int al_pcie_host_init(struct pcie_port *pp) +{ + struct dw_pcie *pci = to_dw_pcie_from_pp(pp); + struct al_pcie *pcie = to_al_pcie(pci); + int link_up; + int rc; + + rc = al_pcie_rev_id_get(pcie, &pcie->controller_rev_id); + if (rc) + return rc; + + rc = al_pcie_reg_offsets_set(pcie); + if (rc) + return rc; + + al_pcie_config_prepare(pcie); + + return 0; +} + +static const struct dw_pcie_host_ops al_pcie_host_ops = { + .rd_other_conf = al_pcie_rd_other_conf, + .wr_other_conf = al_pcie_wr_other_conf, + .host_init = al_pcie_host_init, +}; + +static int al_add_pcie_port(struct pcie_port *pp, + struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + int ret; + + pp->ops = &al_pcie_host_ops; + + ret = dw_pcie_host_init(pp); + if (ret) { + dev_err(dev, "failed to initialize host\n"); + return ret; + } + + return 0; +} + +static const struct dw_pcie_ops dw_pcie_ops = { +}; + +static int al_pcie_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct resource *controller_res; + struct resource *ecam_res; + struct resource *dbi_res; + struct al_pcie *al_pcie; + struct dw_pcie *pci; + int ret; + + al_pcie = devm_kzalloc(dev, sizeof(*al_pcie), GFP_KERNEL); + if (!al_pcie) + return -ENOMEM; + + pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL); + if (!pci) + return -ENOMEM; + + pci->dev = dev; + pci->ops = &dw_pcie_ops; + + al_pcie->pci = pci; + al_pcie->dev = dev; + + dbi_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "dbi"); + pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_res); + if (IS_ERR(pci->dbi_base)) { + dev_err(dev, "couldn't remap dbi base %pR\n", dbi_res); + return PTR_ERR(pci->dbi_base); + } + + ecam_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config"); + if (!ecam_res) { + dev_err(dev, "couldn't find 'config' reg in DT\n"); + return -ENOENT; + } + al_pcie->ecam_size = resource_size(ecam_res); + + controller_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, + "controller"); + al_pcie->controller_base = devm_ioremap_resource(dev, controller_res); + if (IS_ERR(al_pcie->controller_base)) { + dev_err(dev, "couldn't remap controller base %pR\n", + controller_res); + return PTR_ERR(al_pcie->controller_base); + } + + dev_dbg(dev, "From DT: dbi_base: %pR, controller_base: %pR\n", + dbi_res, controller_res); + + platform_set_drvdata(pdev, al_pcie); + + ret = al_add_pcie_port(&pci->pp, pdev); + + return ret; +} + +static const struct of_device_id al_pcie_of_match[] = { + { .compatible = "amazon,al-pcie", + }, + {}, +}; + +static struct platform_driver al_pcie_driver = { + .driver = { + .name = "al-pcie", + .of_match_table = al_pcie_of_match, + .suppress_bind_attrs = true, + }, + .probe = al_pcie_probe, +}; +builtin_platform_driver(al_pcie_driver); + +#endif /* CONFIG_PCIE_AL*/ From patchwork Tue Jul 23 09:27:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054205 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15E5C1399 for ; Tue, 23 Jul 2019 09:27:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 056252022B for ; Tue, 23 Jul 2019 09:27:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED66326E51; Tue, 23 Jul 2019 09:27:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E6292022B for ; Tue, 23 Jul 2019 09:27:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729735AbfGWJ1j (ORCPT ); Tue, 23 Jul 2019 05:27:39 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:21677 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727408AbfGWJ1j (ORCPT ); Tue, 23 Jul 2019 05:27:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563874058; x=1595410058; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=sNQOZhJUfbz+nC7+KNBC3O+NbzfXKJgSiWivUBW5VoI=; b=qcASuAkIsMZsG9v/nw6IcqIhr3Y/1w/FqHW1qY2T1aWFj2nocQS0/N9q gflEIeS6gDssttBZ32bdi8ZnIq4rqFQssQSYETY9x2QfbWlXI/kkCgQQ2 bGgraaEsjeQFuJ5SJof6tzvdd+baQCu9mDzSE8Cg05J5wu8JDAfPzRfAd s=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="812905282" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-2c-6f38efd9.us-west-2.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 23 Jul 2019 09:27:38 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-6f38efd9.us-west-2.amazon.com (Postfix) with ESMTPS id D4609A1C2A; Tue, 23 Jul 2019 09:27:37 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:37 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.161.85) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:32 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 7/8] PCI: dw: Add validation that PCIe core is set to correct mode Date: Tue, 23 Jul 2019 12:27:10 +0300 Message-ID: <20190723092711.11786-3-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.85] X-ClientProxiedBy: EX13D27UWB003.ant.amazon.com (10.43.161.195) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some PCIe controllers can be set to either Host or EP according to some early boot FW. To make sure there is no discrepancy (e.g. FW configured the port to EP mode while the DT specifies it as a host bridge or vice versa), a check has been added for each mode. Signed-off-by: Jonathan Chocron Acked-by: Gustavo Pimentel --- drivers/pci/controller/dwc/pcie-designware-ep.c | 8 ++++++++ drivers/pci/controller/dwc/pcie-designware-host.c | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c index 2bf5a35c0570..00e59a134b93 100644 --- a/drivers/pci/controller/dwc/pcie-designware-ep.c +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c @@ -531,6 +531,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) int ret; u32 reg; void *addr; + u8 hdr_type; unsigned int nbars; unsigned int offset; struct pci_epc *epc; @@ -543,6 +544,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) return -EINVAL; } + hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE); + if (hdr_type != PCI_HEADER_TYPE_NORMAL) { + dev_err(pci->dev, "PCIe controller is not set to EP mode (hdr_type:0x%x)!\n", + hdr_type); + return -EIO; + } + ret = of_property_read_u32(np, "num-ib-windows", &ep->num_ib_windows); if (ret < 0) { dev_err(dev, "Unable to read *num-ib-windows* property\n"); diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index f93252d0da5b..d2ca748e4c85 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -323,6 +323,7 @@ int dw_pcie_host_init(struct pcie_port *pp) struct pci_bus *child; struct pci_host_bridge *bridge; struct resource *cfg_res; + u8 hdr_type; int ret; raw_spin_lock_init(&pci->pp.lock); @@ -396,6 +397,13 @@ int dw_pcie_host_init(struct pcie_port *pp) } } + hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE); + if (hdr_type != PCI_HEADER_TYPE_BRIDGE) { + dev_err(pci->dev, "PCIe controller is not set to bridge type (hdr_type: 0x%x)!\n", + hdr_type); + return -EIO; + } + pp->mem_base = pp->mem->start; if (!pp->va_cfg0_base) { From patchwork Tue Jul 23 09:27:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chocron, Jonathan" X-Patchwork-Id: 11054207 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29A521399 for ; Tue, 23 Jul 2019 09:27:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19ABB1FFBE for ; Tue, 23 Jul 2019 09:27:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0B1BB26E51; Tue, 23 Jul 2019 09:27:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ECD311FFBE for ; Tue, 23 Jul 2019 09:27:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730347AbfGWJ1q (ORCPT ); Tue, 23 Jul 2019 05:27:46 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:31440 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727408AbfGWJ1q (ORCPT ); Tue, 23 Jul 2019 05:27:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1563874065; x=1595410065; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=r3UejI0fEyx4nycQ3vf7JrpqMGHRZyoys9vsmeuSdCk=; b=WpYunE6QvhuTBTT0kkM5lBnZ+I0Xcbplq16FNSNp3Cg+2wMmCyRyV7I4 JPqjjkOFzS+H4LZu5mbcGU9p85vScXytLx0h6e+Z4rZohlzHNFF21lVZI +upfWVJLzI2iyKs3dcWLiPrVjHnMdFT9Kc7PnBQrW7aYlwZBDDT2Qe7LJ M=; X-IronPort-AV: E=Sophos;i="5.64,298,1559520000"; d="scan'208";a="687150634" Received: from sea3-co-svc-lb6-vlan2.sea.amazon.com (HELO email-inbound-relay-2a-69849ee2.us-west-2.amazon.com) ([10.47.22.34]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 23 Jul 2019 09:27:43 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-69849ee2.us-west-2.amazon.com (Postfix) with ESMTPS id BE1F9A2490; Tue, 23 Jul 2019 09:27:42 +0000 (UTC) Received: from EX13D13UWA001.ant.amazon.com (10.43.160.136) by EX13MTAUWA001.ant.amazon.com (10.43.160.58) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:42 +0000 Received: from u9ff250417f405e.ant.amazon.com (10.43.161.85) by EX13D13UWA001.ant.amazon.com (10.43.160.136) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 23 Jul 2019 09:27:37 +0000 From: Jonathan Chocron To: , , , , , CC: , , , , , , , , , , , Subject: [PATCH v3 8/8] PCI: dw: Add support for PCI_PROBE_ONLY/PCI_REASSIGN_ALL_BUS flags Date: Tue, 23 Jul 2019 12:27:11 +0300 Message-ID: <20190723092711.11786-4-jonnyc@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190723092529.11310-1-jonnyc@amazon.com> References: <20190723092529.11310-1-jonnyc@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.85] X-ClientProxiedBy: EX13D27UWB003.ant.amazon.com (10.43.161.195) To EX13D13UWA001.ant.amazon.com (10.43.160.136) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This basically aligns the usage of PCI_PROBE_ONLY and PCI_REASSIGN_ALL_BUS in dw_pcie_host_init() with the logic in pci_host_common_probe(). Now it will be possible to control via the devicetree whether to just probe the PCI bus (in cases where FW already configured it) or to fully configure it. Signed-off-by: Jonathan Chocron --- .../pci/controller/dwc/pcie-designware-host.c | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index d2ca748e4c85..0a294d8aa21a 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -342,6 +342,8 @@ int dw_pcie_host_init(struct pcie_port *pp) if (!bridge) return -ENOMEM; + of_pci_check_probe_only(); + ret = devm_of_pci_get_host_bridge_resources(dev, 0, 0xff, &bridge->windows, &pp->io_base); if (ret) @@ -474,6 +476,10 @@ int dw_pcie_host_init(struct pcie_port *pp) pp->root_bus_nr = pp->busn->start; + /* Do not reassign bus nums if probe only */ + if (!pci_has_flag(PCI_PROBE_ONLY)) + pci_add_flags(PCI_REASSIGN_ALL_BUS); + bridge->dev.parent = dev; bridge->sysdata = pp; bridge->busnr = pp->root_bus_nr; @@ -490,11 +496,20 @@ int dw_pcie_host_init(struct pcie_port *pp) if (pp->ops->scan_bus) pp->ops->scan_bus(pp); - pci_bus_size_bridges(pp->root_bus); - pci_bus_assign_resources(pp->root_bus); + /* + * We insert PCI resources into the iomem_resource and + * ioport_resource trees in either pci_bus_claim_resources() + * or pci_bus_assign_resources(). + */ + if (pci_has_flag(PCI_PROBE_ONLY)) { + pci_bus_claim_resources(pp->root_bus); + } else { + pci_bus_size_bridges(pp->root_bus); + pci_bus_assign_resources(pp->root_bus); - list_for_each_entry(child, &pp->root_bus->children, node) - pcie_bus_configure_settings(child); + list_for_each_entry(child, &pp->root_bus->children, node) + pcie_bus_configure_settings(child); + } pci_bus_add_devices(pp->root_bus); return 0;