From patchwork Fri Nov 22 08:52:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882845 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7C90178CEC for ; Fri, 22 Nov 2024 07:50:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261857; cv=none; b=bQh+3eRpiYxc13jIPVVolUTRjZ/Klvmat71heKKPjRi8nBB9X23lhJzPP4NJKOIk02ONTXGtRC93uzAHcHXk4X0wmCsUtQGCa+Puu4l0mpANAWfhNqMyKSgz+WVZJU2zY8rzLZkhh7wIKWty5aKk5RFVv36oY/CcpSd87CUIUMw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261857; c=relaxed/simple; bh=3DKmPqixFjXnSu8i4KQqEtMLJM564Px14iu08s9bNdw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DKGokiz/wulDFh0tRRaji5YGWooKHB5pnLzBlq7wf7psqwdAUFK4BvCCKMLxHSFBG+1MhI4Qjtx4dCCZXlS29I74FXfTFh4IPhRy5hrHCUTrGlv70NgGNAH95ERTjd8OgD7/BWxYD0IrsmG9VikNyEZQtKQZ1Ro9LNh5RCEBaRo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Dq1UIM3/; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Dq1UIM3/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261856; x=1763797856; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3DKmPqixFjXnSu8i4KQqEtMLJM564Px14iu08s9bNdw=; b=Dq1UIM3/E5lpPCZHx50z2zCnZ3DzsjdGszqF4i0vkLdOXRHF6g+5P8Bi IBamF6+ZRUXhV56OXIosG7vBcfYmVezvGbYkTL8PcwCU0FYbs1lXLwqxC yquH7F1EwV5FFjFcc3MPBhcz9Ct6goOVSJWK0nOssGgDTc/WswtHhXW/X PcGKp9P70F1flNbOwfm6U0EuHrpQ2Ceiqt9kI08EtiQhoSANBbV6iqbR9 +xYtGDJH/2++ZYA6Px3uow+GiYZMV/GStchT9aYWuLnbqSt7pBcwe96vh kkwSzvWWosEIcJUXmuaieW5pqTDa7YxG9+SjJAsfbFVgBa0YiPiN3sHVt A==; X-CSE-ConnectionGUID: eBG3k4zsRaeav7kAd2h5vg== X-CSE-MsgGUID: w9UvvzK5QhqpgIoI0TOCow== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156822" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156822" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:50:55 -0800 X-CSE-ConnectionGUID: 5MsoUFoiTqygoHubQL1JEw== X-CSE-MsgGUID: sFwXEtyWQtieqxKXkJvw/A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301622" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:50:53 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 1/8] PCI: vmd: Add vmd_bus_enumeration() Date: Fri, 22 Nov 2024 09:52:08 +0100 Message-Id: <20241122085215.424736-2-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the vmd bus enumeration code to a new helper vmd_bus_enumeration(). No functional changes. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 89 ++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 40 deletions(-) mode change 100644 => 100755 drivers/pci/controller/vmd.c diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c old mode 100644 new mode 100755 index a726de0af011..fb66910f9204 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -778,6 +778,54 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) return 0; } +static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) +{ + struct pci_bus *child; + struct pci_dev *dev; + int ret; + + vmd_acpi_begin(); + + pci_scan_child_bus(bus); + vmd_domain_reset(vmd_from_bus(bus)); + + /* + * When Intel VMD is enabled, the OS does not discover the Root Ports + * owned by Intel VMD within the MMCFG space. pci_reset_bus() applies + * a reset to the parent of the PCI device supplied as argument. This + * is why we pass a child device, so the reset can be triggered at + * the Intel bridge level and propagated to all the children in the + * hierarchy. + */ + list_for_each_entry(child, &bus->children, node) { + if (!list_empty(&child->devices)) { + dev = list_first_entry(&child->devices, struct pci_dev, + bus_list); + ret = pci_reset_bus(dev); + if (ret) + pci_warn(dev, "can't reset device: %d\n", ret); + + break; + } + } + + pci_assign_unassigned_bus_resources(bus); + + pci_walk_bus(bus, vmd_pm_enable_quirk, &features); + + /* + * VMD root buses are virtual and don't return true on pci_is_pcie() + * and will fail pcie_bus_configure_settings() early. It can instead be + * run on each of the real root ports. + */ + list_for_each_entry(child, &bus->children, node) + pcie_bus_configure_settings(child); + + pci_bus_add_devices(bus); + + vmd_acpi_end(); +} + static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) { struct pci_sysdata *sd = &vmd->sysdata; @@ -787,8 +835,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) LIST_HEAD(resources); resource_size_t offset[2] = {0}; resource_size_t membar2_offset = 0x2000; - struct pci_bus *child; - struct pci_dev *dev; int ret; /* @@ -928,45 +974,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, "domain"), "Can't create symlink to domain\n"); - vmd_acpi_begin(); - - pci_scan_child_bus(vmd->bus); - vmd_domain_reset(vmd); + vmd_bus_enumeration(vmd->bus, features); - /* When Intel VMD is enabled, the OS does not discover the Root Ports - * owned by Intel VMD within the MMCFG space. pci_reset_bus() applies - * a reset to the parent of the PCI device supplied as argument. This - * is why we pass a child device, so the reset can be triggered at - * the Intel bridge level and propagated to all the children in the - * hierarchy. - */ - list_for_each_entry(child, &vmd->bus->children, node) { - if (!list_empty(&child->devices)) { - dev = list_first_entry(&child->devices, - struct pci_dev, bus_list); - ret = pci_reset_bus(dev); - if (ret) - pci_warn(dev, "can't reset device: %d\n", ret); - - break; - } - } - - pci_assign_unassigned_bus_resources(vmd->bus); - - pci_walk_bus(vmd->bus, vmd_pm_enable_quirk, &features); - - /* - * VMD root buses are virtual and don't return true on pci_is_pcie() - * and will fail pcie_bus_configure_settings() early. It can instead be - * run on each of the real root ports. - */ - list_for_each_entry(child, &vmd->bus->children, node) - pcie_bus_configure_settings(child); - - pci_bus_add_devices(vmd->bus); - - vmd_acpi_end(); return 0; } From patchwork Fri Nov 22 08:52:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882846 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84F2C166F26 for ; Fri, 22 Nov 2024 07:50:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261859; cv=none; b=YWC6sc5kzf2IEJnwm6TSQ4BfFZaKovfPsJ+sJm+wxLyIARb3psWZLAGLTngZmaYMns7d2S19JYojJiKO/kKKEIfwAJPYeAfYI/AhOgnBoAN2ZmHqnK58SDXevO80qkgufFRf33LdXX3i7nbadE3jiDxJjWFXt0/BTG1G/rApNp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261859; c=relaxed/simple; bh=diyyFkRFsnHMtBwhlD2nl8ygTX7RBDpPre7ieomvH5o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=b2cp5cDg/dCKeCUnYVnlAByYnSHibJyOpmAxSOEvHkh7NpVOYd63vmSjrS/rT83UH75sJzty127qIfW/cdfsSM9lR07NKbmGy7W2kvMVxQmnYnjXij4Y/VOazEj3mbap0OQjKaGOdRfenkGn4l7xEBFXGOGcAb23CHETxsHbkw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=eTXoCA4G; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="eTXoCA4G" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261859; x=1763797859; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=diyyFkRFsnHMtBwhlD2nl8ygTX7RBDpPre7ieomvH5o=; b=eTXoCA4GD7Ph2WVv5aCWUSaCnbrSbk72ySYpH+gjNBKvZO+2IgjdNTwW DTlRGu/623K0IFPrwy4Ep3XSiZpGcTD2TkNWFwxGw0nwJiGOiSruzdUje ZhMiB98GjC+39vTnFuke9CpQTExH4DpH9dbEwmyRipdAWziGh7d3lok8T IXoIH+mRGw1XGR/vQVCu/cTonaVYcLB52geTHorUM32AFWMQlC0XESX7X 3xayQ5hNt2ECRerVWVWw9Snl8x/+syv5oEETzGylx7rxySFaHUTaw2N0h gkT473OFF/MJ54TkcMbtBqa4YJX1MqWxar5/+JCWvoyRy8giZz5aqvARo w==; X-CSE-ConnectionGUID: mh/42Pt1Tna1rRNx/5f+ww== X-CSE-MsgGUID: 8XOyLJddRryD6J2im2rJIA== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156831" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156831" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:50:58 -0800 X-CSE-ConnectionGUID: gZUw4heTSAGqlYBkBVUTDg== X-CSE-MsgGUID: sNXxoAD3RiqQJO7uwE9lfw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301635" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:50:56 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 2/8] PCI: vmd: Add vmd_configure_cfgbar() Date: Fri, 22 Nov 2024 09:52:09 +0100 Message-Id: <20241122085215.424736-3-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the VMD CFGBAR initialization code to a new helper vmd_configure_cfgbar(). No functional changes. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index fb66910f9204..bb09114068f5 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -778,6 +778,18 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) return 0; } +static void vmd_configure_cfgbar(struct vmd_dev *vmd) +{ + struct resource *res = &vmd->dev->resource[VMD_CFGBAR]; + + vmd->resources[0] = (struct resource){ + .name = "VMD CFGBAR", + .start = vmd->busn_start, + .end = vmd->busn_start + (resource_size(res) >> 20) - 1, + .flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED, + }; +} + static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) { struct pci_bus *child; @@ -864,13 +876,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) return ret; } - res = &vmd->dev->resource[VMD_CFGBAR]; - vmd->resources[0] = (struct resource) { - .name = "VMD CFGBAR", - .start = vmd->busn_start, - .end = vmd->busn_start + (resource_size(res) >> 20) - 1, - .flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED, - }; + vmd_configure_cfgbar(vmd); /* * If the window is below 4GB, clear IORESOURCE_MEM_64 so we can From patchwork Fri Nov 22 08:52:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882847 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9799D17DFE0 for ; Fri, 22 Nov 2024 07:51:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261863; cv=none; b=AzIdTNEifsYBCqXX1GXzmZe2INHy3C5FLf+vx0m5UTw8TXOTKESA3hg1opNmC5VcbBHgzrRhRbuXIsFVP+0yLDcOfnWq/X0YjERr5nZnMOS01uLQuRyX4GwAam72UascrylCF8w9Puobh4bmByQ2fQ5TdESVFCSUMHyK7ROOYUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261863; c=relaxed/simple; bh=qtEgykn61K7e2fFHwpoVA/qTyRjaiMysvp9hNjuT8HA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=oLvRzZSdWkzRr56PyJ8JU3fdplrrT6eunlNrvfazzz3mJL9tlmvmwFPLqn89nnd75f4z49yYEgecvb60omp+dlXRf0/7uXNUPagw4P0ZfBKTn9tV++wqcnVnaeXnxFFMzzdKLIs/G5R0SydfO3KB4yHrlC15jCrwcsJjwkos7uc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZouATo1v; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZouATo1v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261862; x=1763797862; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qtEgykn61K7e2fFHwpoVA/qTyRjaiMysvp9hNjuT8HA=; b=ZouATo1vOAAE46nWImi2pnawdNM/NqhR+G8pcBYX4aB3iz2xITJ6n9t0 MZ/lLQ8R0EjCBQgbAz8qeTt1xYr0mMrGqxnKdgURgFxBJa4CDo+FO1lOv myw4aY3EfQqPB5cTeBBhockDuKZeWNRpmT2A61tBBZycBh+zYGs+lvQrR S54imXOJi7GWPFM5n3TjPuLFCMGnIGaOVwS9qshySB9grDuZc5YJbJPrh f3h2YdBGsm2W2zxjPwnZFkRwzH6qNYHoYjGEm4tuhfPWrAZsQTuyznbIN xyq7+myWVsd1nRPcCSvjrFEaq4hH8zmpCB/dE3CvPeWCCCHmLQS6bo8xi A==; X-CSE-ConnectionGUID: KSSFpPdGSOKDhtIf2JSV6g== X-CSE-MsgGUID: fUguWWQFTGS7JOMJRNetWA== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156845" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156845" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:01 -0800 X-CSE-ConnectionGUID: 9F4bmDhBR1WPq44FGfcHnA== X-CSE-MsgGUID: k1ysoP0dTYSKiml1htL4GA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301651" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:50:59 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 3/8] PCI: vmd: Add vmd_configure_membar() and vmd_configure_membar1_membar2() Date: Fri, 22 Nov 2024 09:52:10 +0100 Message-Id: <20241122085215.424736-4-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the MEMBAR1 and MEMBAR2 registry initialization code to new helpers vmd_configure_membar() and vmd_configure_membar1_membar2(). No functional changes. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 80 ++++++++++++++++++++++-------------- 1 file changed, 50 insertions(+), 30 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index bb09114068f5..240be800ae96 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -790,6 +790,48 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd) }; } +/* + * vmd_configure_membar - Configure VMD MemBAR register, which points + * to MMIO address assigned by the OS or BIOS. + * @vmd: the VMD device + * @resource_number: resource buffer number to be filled in + * @membar_number: number of the MemBAR + * @start_offset: 4K aligned offset applied to start of VMD’s MEMBAR MMIO space + * @end_offset: 4K aligned offset applied to end of VMD’s MEMBAR MMIO space + * + * Function fills resource buffer inside the VMD structure. + */ +static void vmd_configure_membar(struct vmd_dev *vmd, u8 resource_number, + u8 membar_number, resource_size_t start_offset, + resource_size_t end_offset) +{ + u32 upper_bits; + unsigned long flags; + + struct resource *res = &vmd->dev->resource[membar_number]; + + upper_bits = upper_32_bits(res->end); + flags = res->flags & ~IORESOURCE_SIZEALIGN; + if (!upper_bits) + flags &= ~IORESOURCE_MEM_64; + + vmd->resources[resource_number] = (struct resource){ + .name = kasprintf(GFP_KERNEL, "VMD MEMBAR%d", + membar_number / 2), + .start = res->start + start_offset, + .end = res->end - end_offset, + .flags = flags, + .parent = res, + }; +} + +static void vmd_configure_membar1_membar2(struct vmd_dev *vmd, + resource_size_t mbar2_ofs) +{ + vmd_configure_membar(vmd, 1, VMD_MEMBAR1, 0, 0); + vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0); +} + static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) { struct pci_bus *child; @@ -841,9 +883,6 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) { struct pci_sysdata *sd = &vmd->sysdata; - struct resource *res; - u32 upper_bits; - unsigned long flags; LIST_HEAD(resources); resource_size_t offset[2] = {0}; resource_size_t membar2_offset = 0x2000; @@ -890,36 +929,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * * The only way we could use a 64-bit non-prefetchable MEMBAR is * if its address is <4GB so that we can convert it to a 32-bit - * resource. To be visible to the host OS, all VMD endpoints must + * resource. To be visible to the host OS, all VMD endpoints must * be initially configured by platform BIOS, which includes setting - * up these resources. We can assume the device is configured + * up these resources. We can assume the device is configured * according to the platform needs. */ - res = &vmd->dev->resource[VMD_MEMBAR1]; - upper_bits = upper_32_bits(res->end); - flags = res->flags & ~IORESOURCE_SIZEALIGN; - if (!upper_bits) - flags &= ~IORESOURCE_MEM_64; - vmd->resources[1] = (struct resource) { - .name = "VMD MEMBAR1", - .start = res->start, - .end = res->end, - .flags = flags, - .parent = res, - }; - - res = &vmd->dev->resource[VMD_MEMBAR2]; - upper_bits = upper_32_bits(res->end); - flags = res->flags & ~IORESOURCE_SIZEALIGN; - if (!upper_bits) - flags &= ~IORESOURCE_MEM_64; - vmd->resources[2] = (struct resource) { - .name = "VMD MEMBAR2", - .start = res->start + membar2_offset, - .end = res->end, - .flags = flags, - .parent = res, - }; + vmd_configure_membar1_membar2(vmd, membar2_offset); sd->vmd_dev = vmd->dev; sd->domain = vmd_find_free_domain(); @@ -1060,6 +1075,11 @@ static void vmd_remove(struct pci_dev *dev) pci_stop_root_bus(vmd->bus); sysfs_remove_link(&vmd->dev->dev.kobj, "domain"); pci_remove_root_bus(vmd->bus); + + /* CFGBAR is static, does not require releasing memory */ + kfree(vmd->resources[1].name); + kfree(vmd->resources[2].name); + vmd_cleanup_srcu(vmd); vmd_detach_resources(vmd); vmd_remove_irq_domain(vmd); From patchwork Fri Nov 22 08:52:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882848 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50B8D189BBA for ; Fri, 22 Nov 2024 07:51:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261869; cv=none; b=XQHH8BFoa8rhBhTOFciytuEK06SpODt5cAGsSiBHd8Akvybwi+IiWKOApR+vcty1Ige5eW6ZYMyNcGblJMtrulqqpPI29tTUE32fADBVamEPXdDCzyyqhR0n45Ov4nH7DMhgUAnBvfNCJnMoip1O5B4bkxc6ayjFjeztxyvpLlM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261869; c=relaxed/simple; bh=3Mgp6dlUFfVDt8/DzVLC+bpjwKWY72oDMKYB3Be9j1Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kdnut2F1CJDTDn5pZYlM34xPJ7WKhaKerwSVZMNTGCE3Sk08K96yX/nNUobrcSo6PScfAkJXQxc2L6sY+KpvKQTW7tXvWaKh3qQdutFMkvQQguX+QbbHdBNOOv90Hv1KnPN8ILQW80JLU3xbPZxmldR3LZiXX8DEtEU33QhA4gU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=UcvbflXT; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="UcvbflXT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261867; x=1763797867; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3Mgp6dlUFfVDt8/DzVLC+bpjwKWY72oDMKYB3Be9j1Q=; b=UcvbflXTd7+fndADUrFfbJV8SO5bSsruoMzoWOESKvcfKxBanG0vqT4z Fj7XOV/dSj7Qg4v40nfYvD9URX2/9OKCQd+n46uPqn/gGKP5d5DxsB6IJ 2eAZewd0cjwhzp13DMRVjp1e2CLySD1uEtMvwhUxpcWjsYEWZDOqjINR0 p05jPEyK+f74lyA5DX6vPsggFBBbwv8LBJq0kVVZEtRmn97bCAZdISv2Y l+kjoN8Y7gxBMF5llfDEShJ4QDMCVL42SFvFjj1C0KUAcGG8nzkZYg8bn B9tPAySXjASzgnPQy92QBnn/ZOIFIEflCir1FPkpvoT0wMVkBMIyDCObd A==; X-CSE-ConnectionGUID: p1oaTlscSTqJu8wxuxTa6g== X-CSE-MsgGUID: 8cxqTlA+S/yp6+8uqQjIvw== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156859" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156859" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:04 -0800 X-CSE-ConnectionGUID: GZ/e753/TTeP9/y17Nz+lg== X-CSE-MsgGUID: 7YS3Gs/uR9qdp1pPOlL7DA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301666" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:02 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 4/8] PCI: vmd: Add vmd_create_bus() Date: Fri, 22 Nov 2024 09:52:11 +0100 Message-Id: <20241122085215.424736-5-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the VMD bus initialization code to a new helper vmd_create_bus(). No functional changes. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 54 +++++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 20 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 240be800ae96..24ca19a28ba7 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -832,6 +832,36 @@ static void vmd_configure_membar1_membar2(struct vmd_dev *vmd, vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0); } +static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd, + resource_size_t *offset) +{ + LIST_HEAD(resources); + + pci_add_resource(&resources, &vmd->resources[0]); + pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]); + pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]); + + vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start, + &vmd_ops, sd, &resources); + if (!vmd->bus) { + pci_free_resource_list(&resources); + vmd_remove_irq_domain(vmd); + return -ENODEV; + } + + vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus), + to_pci_host_bridge(vmd->bus->bridge)); + + vmd_attach_resources(vmd); + if (vmd->irq_domain) + dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); + else + dev_set_msi_domain(&vmd->bus->dev, + dev_get_msi_domain(&vmd->dev->dev)); + + return 0; +} + static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) { struct pci_bus *child; @@ -883,7 +913,6 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) { struct pci_sysdata *sd = &vmd->sysdata; - LIST_HEAD(resources); resource_size_t offset[2] = {0}; resource_size_t membar2_offset = 0x2000; int ret; @@ -970,28 +999,13 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) vmd_set_msi_remapping(vmd, false); } - pci_add_resource(&resources, &vmd->resources[0]); - pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]); - pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]); + ret = vmd_create_bus(vmd, sd, offset); - vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start, - &vmd_ops, sd, &resources); - if (!vmd->bus) { - pci_free_resource_list(&resources); - vmd_remove_irq_domain(vmd); - return -ENODEV; + if (ret) { + pci_err(vmd->dev, "Can't create bus: %d\n", ret); + return ret; } - vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus), - to_pci_host_bridge(vmd->bus->bridge)); - - vmd_attach_resources(vmd); - if (vmd->irq_domain) - dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); - else - dev_set_msi_domain(&vmd->bus->dev, - dev_get_msi_domain(&vmd->dev->dev)); - WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, "domain"), "Can't create symlink to domain\n"); From patchwork Fri Nov 22 08:52:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882849 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30C0F189F32 for ; Fri, 22 Nov 2024 07:51:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261870; cv=none; b=J2VWAoaVC/BsShEopAiI6FFP6o5lJYrcsU3MD0O2IozxsnRXHRS814PFl9TJD0WSRy0Pc4LAmYPIQWySNFrT3pIfmL7WriVsMmFapchYKfhhMMGIooLaRCCajm4+Ic0VqgLHbN+2Ur3jVyJBziYzzgg70CmXuF3iEFTTQ0dev0Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261870; c=relaxed/simple; bh=vUyexQXC1vY4+ibo9vbwwaeZrvHZOYuBjyfDgTuUtFI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JO9iZcqD6+SjNple+0hJqwRloXPaUlNsNqgeqpOIST4UkUe0yAZ+T9tTzQ1yT+SG+hKRFG0oVYJ6FWey4h/aXVcnDbY5M4Syf5zj+BFKBhlTA+HOA7VJZQ0QZifH9ypRyCH6Vc/U192cMAEYW1PWYSeXS/PZQRx9wnU+uHBE654= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=WHOxelGu; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="WHOxelGu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261868; x=1763797868; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vUyexQXC1vY4+ibo9vbwwaeZrvHZOYuBjyfDgTuUtFI=; b=WHOxelGun7ga+3c2JKZWJwEpDLed7v5BDkzT3HSa0+8O0/HIaXQrO+9H Cc61r251PENtvkhM+ElfDQWlu2gKoPqt9T03LhKtbBVYJz8gTWXUCJ0rV ixQDqgM7xuEKUMs7l9fdX2bdBIiphhQQNvG22wAqlBK7FBtXP00XOSo1T O/9xlL2mIPIVcoS5+KX2C3U8CxrXbFjaVVySLl5EgwQsVX+PGp0RMTnCH 2bkoHDllAVSvtrT6PjwjcrCwaPXcai4MpBAiI2w7+LJEZkE6f8AwdBpiu epVi4Kd2sus/z8BGmI/0felfuSUU4zI5cWN+B+a8J6/GfrWJCSzzTit/T Q==; X-CSE-ConnectionGUID: UQzj5ZRISCe3s87Y0flqdw== X-CSE-MsgGUID: 8tiEDUhVSuCAfNHkxqs+Tw== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156865" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156865" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:08 -0800 X-CSE-ConnectionGUID: uuFHHS1kRcuftLIwhsxfAg== X-CSE-MsgGUID: ZnFxHPHLRMChAGnMnWEKuA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301676" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:06 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 5/8] PCI: vmd: Replace hardcoded values with enum and defines Date: Fri, 22 Nov 2024 09:52:12 +0100 Message-Id: <20241122085215.424736-6-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add enum vmd_resource type to replace hardcoded values. Add defines for vmd bus start number based on VMD restriction value. No functional changes. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 42 ++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 24ca19a28ba7..1cd55c3686f3 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -23,6 +23,10 @@ #define VMD_MEMBAR1 2 #define VMD_MEMBAR2 4 +#define VMD_RESTRICT_0_BUS_START 0 +#define VMD_RESTRICT_1_BUS_START 128 +#define VMD_RESTRICT_2_BUS_START 224 + #define PCI_REG_VMCAP 0x40 #define BUS_RESTRICT_CAP(vmcap) (vmcap & 0x1) #define PCI_REG_VMCONFIG 0x44 @@ -34,6 +38,13 @@ #define MB2_SHADOW_OFFSET 0x2000 #define MB2_SHADOW_SIZE 16 +enum vmd_resource { + VMD_RES_CFGBAR = 0, /* VMD Bus0 Config BAR */ + VMD_RES_MBAR_1, /* VMD Bus0 Resource MemBAR 1 */ + VMD_RES_MBAR_2, /* VMD Bus0 Resource MemBAR 2 */ + VMD_RES_COUNT +}; + enum vmd_features { /* * Device may contain registers which hint the physical location of the @@ -132,7 +143,7 @@ struct vmd_dev { struct vmd_irq_list *irqs; struct pci_sysdata sysdata; - struct resource resources[3]; + struct resource resources[VMD_RES_COUNT]; struct irq_domain *irq_domain; struct pci_bus *bus; u8 busn_start; @@ -516,7 +527,7 @@ static inline void vmd_acpi_end(void) { } static void vmd_domain_reset(struct vmd_dev *vmd) { - u16 bus, max_buses = resource_size(&vmd->resources[0]); + u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]); u8 dev, functions, fn, hdr_type; char __iomem *base; @@ -564,8 +575,8 @@ static void vmd_domain_reset(struct vmd_dev *vmd) static void vmd_attach_resources(struct vmd_dev *vmd) { - vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1]; - vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[2]; + vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1]; + vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2]; } static void vmd_detach_resources(struct vmd_dev *vmd) @@ -655,13 +666,13 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd) switch (BUS_RESTRICT_CFG(reg)) { case 0: - vmd->busn_start = 0; + vmd->busn_start = VMD_RESTRICT_0_BUS_START; break; case 1: - vmd->busn_start = 128; + vmd->busn_start = VMD_RESTRICT_1_BUS_START; break; case 2: - vmd->busn_start = 224; + vmd->busn_start = VMD_RESTRICT_2_BUS_START; break; default: pci_err(dev, "Unknown Bus Offset Setting (%d)\n", @@ -782,7 +793,7 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd) { struct resource *res = &vmd->dev->resource[VMD_CFGBAR]; - vmd->resources[0] = (struct resource){ + vmd->resources[VMD_RES_CFGBAR] = (struct resource){ .name = "VMD CFGBAR", .start = vmd->busn_start, .end = vmd->busn_start + (resource_size(res) >> 20) - 1, @@ -801,7 +812,8 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd) * * Function fills resource buffer inside the VMD structure. */ -static void vmd_configure_membar(struct vmd_dev *vmd, u8 resource_number, +static void vmd_configure_membar(struct vmd_dev *vmd, + enum vmd_resource resource_number, u8 membar_number, resource_size_t start_offset, resource_size_t end_offset) { @@ -837,9 +849,11 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd, { LIST_HEAD(resources); - pci_add_resource(&resources, &vmd->resources[0]); - pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]); - pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]); + pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]); + pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1], + offset[0]); + pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2], + offset[1]); vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start, &vmd_ops, sd, &resources); @@ -1091,8 +1105,8 @@ static void vmd_remove(struct pci_dev *dev) pci_remove_root_bus(vmd->bus); /* CFGBAR is static, does not require releasing memory */ - kfree(vmd->resources[1].name); - kfree(vmd->resources[2].name); + kfree(vmd->resources[VMD_RES_MBAR_1].name); + kfree(vmd->resources[VMD_RES_MBAR_2].name); vmd_cleanup_srcu(vmd); vmd_detach_resources(vmd); From patchwork Fri Nov 22 08:52:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882850 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D31A18A6C3 for ; Fri, 22 Nov 2024 07:51:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261873; cv=none; b=t6fplfUYO4UEOgeegRojNGWGi9qDiUlAQ26S/hQM8aejPjcCoz9ICLc51JOfzaUFQlkDQB6zNoLRvz99xR433fTG82TvmUHUUSSdqkMabWumBzHRJ7yihA+5ogfkSAXL1riHDAy5iPc7Uv1CCasr+QAn9yEBC16k/i8MDcGdxMY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261873; c=relaxed/simple; bh=u6dNy0VKAxxVzrmtq1KTAiKG4i0N9UITP4zAQjTiv08=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=j9+ursdYiB79BUb7OuekgmyFWc38QbHJ5kqjkHny+4foL/e7LZVpFBfgR8L0/k/YYW2RVsGsUJHc6k49o6+L87GEnUCpp2u1GPuRXZfCNVB7Jn0hEK8Af/b/PFR+E9dnV5EmkPSMvf5UTLlevP+RGMuBL5h+5SvkawzD+N+tMJ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Qi6r3NiC; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Qi6r3NiC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261871; x=1763797871; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=u6dNy0VKAxxVzrmtq1KTAiKG4i0N9UITP4zAQjTiv08=; b=Qi6r3NiC1egWOkXsmIGCStJ6FhODQWz5U1UNK63bI5e/nYMp9qRoq3vF 4fahmgoNC4p/IHLRLoRGOrD4By+ORNPXZm5MObVVL9hOmC0d7bjbycgVu k6equlqZdAJj1F0X/i8AcwqXz/ZNnxelT5XWwFj5D21wWqoX6J6fQRGR6 N5i36UQBWr67N/uawg726T7tKQYCTEDl/7H/F/2fFWM9H/FZANJfWFRDy IPkF8GbJrZI7YP/CQF+Ni6NyGylmjFml65RfnK5Qg7dRkXy52NBgRfkDz GVQYQb89WyhtJbZtWHGWPpgfGw9ZyWHrTj5MKL7vA2KVbuHeRthE3/T0G Q==; X-CSE-ConnectionGUID: vYbGXeDqRNecx1Scv8qF/Q== X-CSE-MsgGUID: GftnSFBoSUOdFQwMharirw== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156881" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156881" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:11 -0800 X-CSE-ConnectionGUID: 2lcGcvThTO+OycOq6hulig== X-CSE-MsgGUID: CDRy7n1pSxSlHBftfkPBxA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301694" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:09 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 6/8] PCI: vmd: Convert bus and busn_start to an array Date: Fri, 22 Nov 2024 09:52:13 +0100 Message-Id: <20241122085215.424736-7-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert bus and busn_start from scalar to an array to support multiple VMD buses in the future. No functional changes. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 49 +++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 1cd55c3686f3..6d8397b5ebee 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -45,6 +45,11 @@ enum vmd_resource { VMD_RES_COUNT }; +enum vmd_rootbus { + VMD_BUS_0 = 0, + VMD_BUS_COUNT +}; + enum vmd_features { /* * Device may contain registers which hint the physical location of the @@ -145,8 +150,8 @@ struct vmd_dev { struct pci_sysdata sysdata; struct resource resources[VMD_RES_COUNT]; struct irq_domain *irq_domain; - struct pci_bus *bus; - u8 busn_start; + struct pci_bus *bus[VMD_BUS_COUNT]; + u8 busn_start[VMD_BUS_COUNT]; u8 first_vec; char *name; int instance; @@ -389,7 +394,7 @@ static void vmd_remove_irq_domain(struct vmd_dev *vmd) static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus, unsigned int devfn, int reg, int len) { - unsigned int busnr_ecam = bus->number - vmd->busn_start; + unsigned int busnr_ecam = bus->number - vmd->busn_start[VMD_BUS_0]; u32 offset = PCIE_ECAM_OFFSET(busnr_ecam, devfn, reg); if (offset + len >= resource_size(&vmd->dev->resource[VMD_CFGBAR])) @@ -666,13 +671,13 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd) switch (BUS_RESTRICT_CFG(reg)) { case 0: - vmd->busn_start = VMD_RESTRICT_0_BUS_START; + vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_0_BUS_START; break; case 1: - vmd->busn_start = VMD_RESTRICT_1_BUS_START; + vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_1_BUS_START; break; case 2: - vmd->busn_start = VMD_RESTRICT_2_BUS_START; + vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START; break; default: pci_err(dev, "Unknown Bus Offset Setting (%d)\n", @@ -795,8 +800,9 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd) vmd->resources[VMD_RES_CFGBAR] = (struct resource){ .name = "VMD CFGBAR", - .start = vmd->busn_start, - .end = vmd->busn_start + (resource_size(res) >> 20) - 1, + .start = vmd->busn_start[VMD_BUS_0], + .end = vmd->busn_start[VMD_BUS_0] + + (resource_size(res) >> 20) - 1, .flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED, }; } @@ -855,22 +861,24 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd, pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2], offset[1]); - vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start, - &vmd_ops, sd, &resources); - if (!vmd->bus) { + vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev, + vmd->busn_start[VMD_BUS_0], + &vmd_ops, sd, &resources); + if (!vmd->bus[VMD_BUS_0]) { pci_free_resource_list(&resources); vmd_remove_irq_domain(vmd); return -ENODEV; } - vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus), - to_pci_host_bridge(vmd->bus->bridge)); + vmd_copy_host_bridge_flags( + pci_find_host_bridge(vmd->dev->bus), + to_pci_host_bridge(vmd->bus[VMD_BUS_0]->bridge)); vmd_attach_resources(vmd); if (vmd->irq_domain) - dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain); + dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, vmd->irq_domain); else - dev_set_msi_domain(&vmd->bus->dev, + dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, dev_get_msi_domain(&vmd->dev->dev)); return 0; @@ -1020,10 +1028,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) return ret; } - WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, - "domain"), "Can't create symlink to domain\n"); + WARN(sysfs_create_link(&vmd->dev->dev.kobj, + &vmd->bus[VMD_BUS_0]->dev.kobj, "domain"), + "Can't create symlink to domain\n"); - vmd_bus_enumeration(vmd->bus, features); + vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features); return 0; } @@ -1100,9 +1109,9 @@ static void vmd_remove(struct pci_dev *dev) { struct vmd_dev *vmd = pci_get_drvdata(dev); - pci_stop_root_bus(vmd->bus); + pci_stop_root_bus(vmd->bus[VMD_BUS_0]); sysfs_remove_link(&vmd->dev->dev.kobj, "domain"); - pci_remove_root_bus(vmd->bus); + pci_remove_root_bus(vmd->bus[VMD_BUS_0]); /* CFGBAR is static, does not require releasing memory */ kfree(vmd->resources[VMD_RES_MBAR_1].name); From patchwork Fri Nov 22 08:52:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882851 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C2C8187855 for ; Fri, 22 Nov 2024 07:51:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261877; cv=none; b=D3s8QYj4HER1LiBsfVtXp//PzteZTFPgRXiQVOis75f+xiZP6L6LprlJd1LMF4jb8hN3xRHVfSHWCE2748jGeExgYjHLkag+m7O4oe+qOB7vi0XeSli+odDA/BEX2SSQPnZ7PAjOnqDvCLv36PfnWi+iQY6zxz/d9GoEVttL9zQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261877; c=relaxed/simple; bh=RUIybfcv5ZJj+gIvsuJgLUORJSbRjrfMSlu4lMTHL10=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gpS4+ljc15iT9k/Q2nX3oLoorKVfzxxNL9XbL08V4VWbujGP+N8lCk/5YNvRpe1/lPmUNtxeh5x7ERC4Sk6KAh7uNAhcYQE1NeYsPjLQEGS7cFLrVAicLelTweR7UB7srx1Lq7jj+FEzu64xi4XZdt9pN0XuhpGVuhvimFOB5A4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ICm1e6N1; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ICm1e6N1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261874; x=1763797874; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RUIybfcv5ZJj+gIvsuJgLUORJSbRjrfMSlu4lMTHL10=; b=ICm1e6N1mM3q511aLIz6vP1JVp9bVY7ofBn43ssaRviNNu++KPkPpNiV p3kim4akuXtmSrl+plpvseHyfTRklPJEL8RHT1/cHH5g9a+5XQIk5oVyr N6TxXRdj9uX5WQwqa/c93Qa6JKr4QlaHDwB8oefQRSUkBVqX5j7r5yLoW cFa/8Lr8+12Og15Bb8F0ttnXFqBz3Ood0dDXj7KRWVmuZZ9kkYSNrIr76 mRQ1VFlmQV4r3YithALl/vKlz+Tq9cnUIdEbgZLpR+LjobrTxj5+GpQRz IXSZ76qGpNQXMN84hrb8/29vUNG/7C1mbtoDgBWlg0PGG06fib1kY0LiJ Q==; X-CSE-ConnectionGUID: tJPrYc3PTs+ontvAa+IlXg== X-CSE-MsgGUID: CAbTRuBrSyGxthV/xmTTHg== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156889" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156889" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:14 -0800 X-CSE-ConnectionGUID: F8ZXgAHhTYu4dGM68prMnA== X-CSE-MsgGUID: UKqjf98MQweI98Nfmp6V4A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301704" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:12 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 7/8] PCI: vmd: Add support for second rootbus under VMD Date: Fri, 22 Nov 2024 09:52:14 +0100 Message-Id: <20241122085215.424736-8-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Starting from Intel Arrow Lake VMD enhancement introduces second rootbus support with fixed root bus number (0x80). It means that all 3 MMIO BARs exposed by VMD are shared now between both buses (current BUS0 and new BUS1). Add new BUS1 enumeration and divide MMIO space to be shared between both rootbuses. Due to enumeration issues with rootbus hardwired to a fixed non-zero value, this patch will work with a workaround proposed in next patch. Without workaround user won't see attached devices for BUS1 rootbus. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 208 ++++++++++++++++++++++++++++++----- 1 file changed, 180 insertions(+), 28 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 6d8397b5ebee..6cd14c28fd4e 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -26,6 +26,7 @@ #define VMD_RESTRICT_0_BUS_START 0 #define VMD_RESTRICT_1_BUS_START 128 #define VMD_RESTRICT_2_BUS_START 224 +#define VMD_RESTRICT_3_BUS_START 225 #define PCI_REG_VMCAP 0x40 #define BUS_RESTRICT_CAP(vmcap) (vmcap & 0x1) @@ -38,15 +39,33 @@ #define MB2_SHADOW_OFFSET 0x2000 #define MB2_SHADOW_SIZE 16 +#define VMD_PRIMARY_BUS0 0x00 +#define VMD_PRIMARY_BUS1 0x80 +#define VMD_BUSRANGE0 0xc8 +#define VMD_BUSRANGE1 0xcc +#define VMD_MEMBAR1_OFFSET 0xd0 +#define VMD_MEMBAR2_OFFSET1 0xd8 +#define VMD_MEMBAR2_OFFSET2 0xdc +#define VMD_BUS_END(busr) ((busr >> 8) & 0xff) +#define VMD_BUS_START(busr) (busr & 0x00ff) + +/* + * Add VMD resources for BUS1, it will share the same MMIO space with + * previous VMD resources. + */ enum vmd_resource { VMD_RES_CFGBAR = 0, /* VMD Bus0 Config BAR */ VMD_RES_MBAR_1, /* VMD Bus0 Resource MemBAR 1 */ VMD_RES_MBAR_2, /* VMD Bus0 Resource MemBAR 2 */ + VMD_RES_BUS1_CFGBAR, /* VMD Bus1 Config BAR */ + VMD_RES_BUS1_MBAR_1, /* VMD Bus1 Resource MemBAR 1 */ + VMD_RES_BUS1_MBAR_2, /* VMD Bus1 Resource MemBAR 2 */ VMD_RES_COUNT }; enum vmd_rootbus { VMD_BUS_0 = 0, + VMD_BUS_1, VMD_BUS_COUNT }; @@ -90,6 +109,12 @@ enum vmd_features { * proper power management of the SoC. */ VMD_FEAT_BIOS_PM_QUIRK = (1 << 5), + + /* + * Starting from Intel Arrow Lake, VMD devices have their VMD rootports + * connected to additional BUS1 rootport. + */ + VMD_FEAT_HAS_BUS1_ROOTBUS = (1 << 6) }; #define VMD_BIOS_PM_QUIRK_LTR 0x1003 /* 3145728 ns */ @@ -97,7 +122,8 @@ enum vmd_features { #define VMD_FEATS_CLIENT (VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | \ VMD_FEAT_HAS_BUS_RESTRICTIONS | \ VMD_FEAT_OFFSET_FIRST_VECTOR | \ - VMD_FEAT_BIOS_PM_QUIRK) + VMD_FEAT_BIOS_PM_QUIRK | \ + VMD_FEAT_HAS_BUS1_ROOTBUS) static DEFINE_IDA(vmd_instance_ida); @@ -155,6 +181,7 @@ struct vmd_dev { u8 first_vec; char *name; int instance; + bool bus1_rootbus; }; static inline struct vmd_dev *vmd_from_bus(struct pci_bus *bus) @@ -532,11 +559,16 @@ static inline void vmd_acpi_end(void) { } static void vmd_domain_reset(struct vmd_dev *vmd) { - u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]); + u16 bus, bus_cnt = resource_size(&vmd->resources[VMD_RES_CFGBAR]); u8 dev, functions, fn, hdr_type; char __iomem *base; - for (bus = 0; bus < max_buses; bus++) { + if (vmd->bus1_rootbus) { + bus_cnt += resource_size(&vmd->resources[VMD_RES_BUS1_CFGBAR]); + bus_cnt += 2; + } + + for (bus = 0; bus < bus_cnt; bus++) { for (dev = 0; dev < 32; dev++) { base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus, PCI_DEVFN(dev, 0), 0); @@ -582,12 +614,24 @@ static void vmd_attach_resources(struct vmd_dev *vmd) { vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1]; vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2]; + + if (vmd->bus1_rootbus) { + vmd->resources[VMD_RES_MBAR_1].sibling = + &vmd->resources[VMD_RES_BUS1_MBAR_1]; + vmd->resources[VMD_RES_MBAR_2].sibling = + &vmd->resources[VMD_RES_BUS1_MBAR_2]; + } } static void vmd_detach_resources(struct vmd_dev *vmd) { vmd->dev->resource[VMD_MEMBAR1].child = NULL; vmd->dev->resource[VMD_MEMBAR2].child = NULL; + + if (vmd->bus1_rootbus) { + vmd->resources[VMD_RES_MBAR_1].sibling = NULL; + vmd->resources[VMD_RES_MBAR_2].sibling = NULL; + } } /* @@ -660,7 +704,7 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint, return 0; } -static int vmd_get_bus_number_start(struct vmd_dev *vmd) +static int vmd_get_bus_number_start(struct vmd_dev *vmd, unsigned long features) { struct pci_dev *dev = vmd->dev; u16 reg; @@ -679,6 +723,19 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd) case 2: vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START; break; + case 3: + if (!(features & VMD_FEAT_HAS_BUS1_ROOTBUS)) { + pci_err(dev, "VMD Bus Restriction detected type %d, but BUS1 Rootbus is not supported, aborting.\n", + BUS_RESTRICT_CFG(reg)); + return -ENODEV; + } + + /* BUS0 start number */ + vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START; + /* BUS1 start number */ + vmd->busn_start[VMD_BUS_1] = VMD_RESTRICT_3_BUS_START; + vmd->bus1_rootbus = true; + break; default: pci_err(dev, "Unknown Bus Offset Setting (%d)\n", BUS_RESTRICT_CFG(reg)); @@ -805,6 +862,30 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd) (resource_size(res) >> 20) - 1, .flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED, }; + + if (vmd->bus1_rootbus) { + u16 bus0_range = 0; + u16 bus1_range = 0; + + pci_read_config_word(vmd->dev, VMD_BUSRANGE0, &bus0_range); + pci_read_config_word(vmd->dev, VMD_BUSRANGE1, &bus1_range); + + /* + * Resize BUS0 CFGBAR range to make space for BUS1 + * owned devices by adjusting range end with value stored in + * VMD_BUSRANGE0 register. + */ + vmd->resources[VMD_RES_CFGBAR].start = VMD_BUS_START(bus0_range); + vmd->resources[VMD_RES_CFGBAR].end = VMD_BUS_END(bus0_range); + + vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){ + .name = "VMD CFGBAR BUS1", + .start = VMD_BUS_START(bus1_range), + .end = VMD_BUS_END(bus1_range), + .flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED, + .parent = res, + }; + } } /* @@ -834,8 +915,9 @@ static void vmd_configure_membar(struct vmd_dev *vmd, flags &= ~IORESOURCE_MEM_64; vmd->resources[resource_number] = (struct resource){ - .name = kasprintf(GFP_KERNEL, "VMD MEMBAR%d", - membar_number / 2), + .name = kasprintf( + GFP_KERNEL, "VMD MEMBAR%d %s", membar_number / 2, + resource_number > VMD_RES_MBAR_2 ? "BUS1" : ""), .start = res->start + start_offset, .end = res->end - end_offset, .flags = flags, @@ -846,41 +928,80 @@ static void vmd_configure_membar(struct vmd_dev *vmd, static void vmd_configure_membar1_membar2(struct vmd_dev *vmd, resource_size_t mbar2_ofs) { - vmd_configure_membar(vmd, 1, VMD_MEMBAR1, 0, 0); - vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0); + if (vmd->bus1_rootbus) { + u32 bus1_mbar1_ofs = 0; + u64 bus1_mbar2_ofs = 0; + u32 reg; + + pci_read_config_dword(vmd->dev, VMD_MEMBAR1_OFFSET, + &bus1_mbar1_ofs); + + pci_read_config_dword(vmd->dev, VMD_MEMBAR2_OFFSET1, ®); + bus1_mbar2_ofs = reg; + + pci_read_config_dword(vmd->dev, VMD_MEMBAR2_OFFSET2, ®); + bus1_mbar2_ofs |= (u64)reg << 32; + + /* + * Resize BUS MEMBAR1 and MEMBAR2 ranges to make space + * for BUS1 owned devices by adjusting range end with values + * stored in VMD_MEMBAR1_OFFSET and VMD_MEMBAR2_OFFSET registers + */ + vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, + bus1_mbar1_ofs); + vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2, + mbar2_ofs, bus1_mbar2_ofs - mbar2_ofs); + + vmd_configure_membar(vmd, VMD_RES_BUS1_MBAR_1, VMD_MEMBAR1, + bus1_mbar1_ofs, 0); + vmd_configure_membar(vmd, VMD_RES_BUS1_MBAR_2, VMD_MEMBAR2, + mbar2_ofs + bus1_mbar2_ofs, 0); + } else { + vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0); + vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2, + mbar2_ofs, 0); + } } -static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd, - resource_size_t *offset) +static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number, + struct pci_sysdata *sd, resource_size_t *offset) { + u8 cfgbar = bus_number * 3; + u8 membar1 = cfgbar + 1; + u8 membar2 = cfgbar + 2; + struct pci_bus *vmd_bus; LIST_HEAD(resources); - pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]); - pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1], + pci_add_resource(&resources, &vmd->resources[cfgbar]); + pci_add_resource_offset(&resources, &vmd->resources[membar1], offset[0]); - pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2], + pci_add_resource_offset(&resources, &vmd->resources[membar2], offset[1]); - vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev, - vmd->busn_start[VMD_BUS_0], - &vmd_ops, sd, &resources); - if (!vmd->bus[VMD_BUS_0]) { + vmd_bus = pci_create_root_bus(&vmd->dev->dev, + vmd->busn_start[bus_number], &vmd_ops, sd, + &resources); + + if (!vmd_bus) { pci_free_resource_list(&resources); - vmd_remove_irq_domain(vmd); + + if (bus_number == VMD_PRIMARY_BUS0) + vmd_remove_irq_domain(vmd); return -ENODEV; } - vmd_copy_host_bridge_flags( - pci_find_host_bridge(vmd->dev->bus), - to_pci_host_bridge(vmd->bus[VMD_BUS_0]->bridge)); + vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus), + to_pci_host_bridge(vmd_bus->bridge)); vmd_attach_resources(vmd); if (vmd->irq_domain) - dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, vmd->irq_domain); + dev_set_msi_domain(&vmd_bus->dev, vmd->irq_domain); else - dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, + dev_set_msi_domain(&vmd_bus->dev, dev_get_msi_domain(&vmd->dev->dev)); + vmd->bus[bus_number] = vmd_bus; + return 0; } @@ -893,7 +1014,9 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features) vmd_acpi_begin(); pci_scan_child_bus(bus); - vmd_domain_reset(vmd_from_bus(bus)); + + if (bus->primary == VMD_PRIMARY_BUS0) + vmd_domain_reset(vmd_from_bus(bus)); /* * When Intel VMD is enabled, the OS does not discover the Root Ports @@ -961,7 +1084,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * limits the bus range to between 0-127, 128-255, or 224-255 */ if (features & VMD_FEAT_HAS_BUS_RESTRICTIONS) { - ret = vmd_get_bus_number_start(vmd); + ret = vmd_get_bus_number_start(vmd, features); if (ret) return ret; } @@ -1021,7 +1144,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) vmd_set_msi_remapping(vmd, false); } - ret = vmd_create_bus(vmd, sd, offset); + ret = vmd_create_bus(vmd, VMD_BUS_0, sd, offset); if (ret) { pci_err(vmd->dev, "Can't create bus: %d\n", ret); @@ -1034,6 +1157,27 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features); + if (vmd->bus1_rootbus) { + ret = vmd_create_bus(vmd, VMD_BUS_1, sd, offset); + if (ret) { + pci_err(vmd->dev, "Can't create BUS1: %d\n", ret); + return ret; + } + + /* + * Primary bus is not set by pci_create_root_bus(), it is + * updated here + */ + vmd->bus[VMD_BUS_1]->primary = VMD_PRIMARY_BUS1; + + WARN(sysfs_create_link(&vmd->dev->dev.kobj, + &vmd->bus[VMD_BUS_1]->dev.kobj, + "domain1"), + "Can't create symlink to domain1\n"); + + vmd_bus_enumeration(vmd->bus[VMD_BUS_1], features); + } + return 0; } @@ -1113,10 +1257,18 @@ static void vmd_remove(struct pci_dev *dev) sysfs_remove_link(&vmd->dev->dev.kobj, "domain"); pci_remove_root_bus(vmd->bus[VMD_BUS_0]); - /* CFGBAR is static, does not require releasing memory */ + /* CFGBARs are static, do not require releasing memory */ kfree(vmd->resources[VMD_RES_MBAR_1].name); kfree(vmd->resources[VMD_RES_MBAR_2].name); + if (vmd->bus1_rootbus) { + pci_stop_root_bus(vmd->bus[VMD_BUS_1]); + sysfs_remove_link(&vmd->dev->dev.kobj, "domain1"); + pci_remove_root_bus(vmd->bus[VMD_BUS_1]); + kfree(vmd->resources[VMD_RES_BUS1_MBAR_1].name); + kfree(vmd->resources[VMD_RES_BUS1_MBAR_2].name); + } + vmd_cleanup_srcu(vmd); vmd_detach_resources(vmd); vmd_remove_irq_domain(vmd); @@ -1202,4 +1354,4 @@ module_pci_driver(vmd_drv); MODULE_AUTHOR("Intel Corporation"); MODULE_DESCRIPTION("Volume Management Device driver"); MODULE_LICENSE("GPL v2"); -MODULE_VERSION("0.6"); +MODULE_VERSION("0.7"); From patchwork Fri Nov 22 08:52:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szymon Durawa X-Patchwork-Id: 13882852 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81F39189F48 for ; Fri, 22 Nov 2024 07:51:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261881; cv=none; b=O+IAFZAhHg53ddnqnj49to3Mk+kyQLwGLcTsw3MuQl3rVi/VaO5e3c8XkpM1yuvyvXC+fEuxxomkXz+kpwC4Ffyy2sHreUm7gmEFLONjogJ7HcVxyfR1Hl7/wOnyAJNTRmGH5/KgWRqp1uAi5wZFIvkhLILV7xadlf5mxSkrEPk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732261881; c=relaxed/simple; bh=76KQkcTQFNUF3vyRhFY4U35pGp30sPnRO2R4D5U950g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gLL3aYW1FarAQ5C+1aq+lUAWblnRyj+Xxhm5Mo9gJRALwuxdAMeyiciz6SUqyVlRSDSAaf6qI4rW/9hBBB/mcrn4mAkLwB47onbmcmzh5CogRfGG7inuxpivurWwo39tqTkXv3KDegQVbxkGAFPkm/09b6raxkJo7UrjOr2r4xk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=HzPh83XF; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="HzPh83XF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732261880; x=1763797880; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=76KQkcTQFNUF3vyRhFY4U35pGp30sPnRO2R4D5U950g=; b=HzPh83XFQ20bCm3gkARHLIZi6z9Rk647z/bivicLt02aU+OQn6g6955M 3qKidHlgUS5QYQdWYOukqmHAFpV0kdQ62Qr0EH7dwnu+l7WwrdY7gMi2j /3iG+3jFF7y+l2x18ZLxCWGIWyl3hQvJtrrf2gsOKZpcXq+Bw/pja/KVe XCRuDZpnJyrfIhICxq0P5fks+T9Q1tGFvcu9wl8LYdyVou09zGnWq0I3k IiEtvahVaT2A8nZb4wqoQoUAZWYxhXnUpPsrop+yZKVk5gqgu5Pn2NvWu WH5Ww9a9cXh8zSmUwygv7s/f4zkwI7Wc1j9ZU7KBBIYKLk7xttHK5sn3e w==; X-CSE-ConnectionGUID: 03kmFM5hQn6wqSlPprLjkA== X-CSE-MsgGUID: 6096okw3QMytpaIz0UHTkA== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="32156895" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="32156895" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:19 -0800 X-CSE-ConnectionGUID: sm5clgXkR+OBblrK6F5Kzw== X-CSE-MsgGUID: WV1w4qD4TA+OquSW2wZPdA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="90301716" Received: from arl-s-03.igk.intel.com ([10.91.111.103]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Nov 2024 23:51:17 -0800 From: Szymon Durawa To: helgaas@kernel.org Cc: Szymon Durawa , Bjorn Helgaas , Dan Williams , Lukas Wunner , linux-pci@vger.kernel.org, Nirmal Patel , Mariusz Tkaczyk Subject: [PATCH v3 8/8] PCI: vmd: Add workaround for bus number hardwired to fixed non-zero value Date: Fri, 22 Nov 2024 09:52:15 +0100 Message-Id: <20241122085215.424736-9-szymon.durawa@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241122085215.424736-1-szymon.durawa@linux.intel.com> References: <20241122085215.424736-1-szymon.durawa@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 VMD BUS1 rootbus primary number is 0x80 and pci_scan_bridge_extend() detects that primary bus number doesn't match the bus it's sitting on. As a result primary rootbus number is deconfigured in the first pass of pci_scan_bridge() to be re-assigned to 0x0 in the second pass. To avoid bus number reconfiguration, BUS1 number has to be the same as BUS1 primary number. Suggested-by: Nirmal Patel Reviewed-by: Mariusz Tkaczyk Signed-off-by: Szymon Durawa --- drivers/pci/controller/vmd.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 6cd14c28fd4e..3b74cb8dd023 100755 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -421,8 +421,22 @@ static void vmd_remove_irq_domain(struct vmd_dev *vmd) static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus, unsigned int devfn, int reg, int len) { - unsigned int busnr_ecam = bus->number - vmd->busn_start[VMD_BUS_0]; - u32 offset = PCIE_ECAM_OFFSET(busnr_ecam, devfn, reg); + unsigned char bus_number; + unsigned int busnr_ecam; + u32 offset; + + /* + * VMD workaraund: for BUS1, bus->number is set to VMD_PRIMARY_BUS1 + * (see comment under vmd_create_bus() for BUS1) but original value + * is 225 which is stored in vmd->busn_start[VMD_BUS_1]. + */ + if (vmd->bus1_rootbus && bus->number == VMD_PRIMARY_BUS1) + bus_number = vmd->busn_start[VMD_BUS_1]; + else + bus_number = bus->number; + + busnr_ecam = bus_number - vmd->busn_start[VMD_BUS_0]; + offset = PCIE_ECAM_OFFSET(busnr_ecam, devfn, reg); if (offset + len >= resource_size(&vmd->dev->resource[VMD_CFGBAR])) return NULL; @@ -1170,6 +1184,18 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) */ vmd->bus[VMD_BUS_1]->primary = VMD_PRIMARY_BUS1; + /* + * This is a workaround for pci_scan_bridge_extend() code. + * It detects that non-zero (0x80) primary bus number doesn't + * match the bus it's sitting on. As a result rootbus number is + * deconfigured in the first pass of pci_scan_bridge() to be + * re-assigned to 0x0 in the second pass. + * Update vmd->bus[VMD_BUS_1]->number and + * vmd->bus[VMD_BUS_1]->primary to the same value, which + * bypasses bus number reconfiguration. + */ + vmd->bus[VMD_BUS_1]->number = VMD_PRIMARY_BUS1; + WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus[VMD_BUS_1]->dev.kobj, "domain1"),