| Message ID | 20240627005856.11449-1-zhoushengqing@ttyinfo.com (mailing list archive) |
|---|---|
| State | New |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | [v3] PCI: Enable io space 1k granularity for intel cpu root port | expand |
On Thu, Jun 27, 2024 at 12:58:56AM +0000, Zhou Shengqing wrote: > This patch add 1k granularity for intel root port bridge.Intel latest > server CPU support 1K granularity,And there is an BIOS setup item named > "EN1K",but linux doesn't support it. if an IIO has 5 IOU (SPR has 5 IOUs) > all are bifurcated 2x8.In a 2P server system,There are 20 P2P bridges > present.if keep 4K granularity allocation,it need 20*4=80k io space, > exceeding 64k.I test it in a 16*nvidia 4090s system under intel eaglestrem > platform.There are six 4090s that cannot be allocated I/O resources. > So I applied this patch.And I found a similar implementation in quirks.c, > but it only targets the Intel P64H2 platform. > > Signed-off-by: Zhou Shengqing <zhoushengqing@ttyinfo.com> > --- > drivers/pci/probe.c | 23 +++++++++++++++++++++++ > 1 file changed, 23 insertions(+) > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > index 5fbabb4e3425..909962795311 100644 > --- a/drivers/pci/probe.c > +++ b/drivers/pci/probe.c > @@ -461,6 +461,9 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) > u32 buses; > u16 io; > u32 pmem, tmp; > + u16 ven_id, dev_id; > + u16 en1k = 0; > + struct pci_dev *dev = NULL; > struct resource res; > > pci_read_config_dword(bridge, PCI_PRIMARY_BUS, &buses); > @@ -478,6 +481,26 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) > } > if (io) { > bridge->io_window = 1; > + if (pci_is_root_bus(bridge->bus)) { > + list_for_each_entry(dev, &bridge->bus->devices, bus_list) { > + pci_read_config_word(dev, PCI_VENDOR_ID, &ven_id); > + pci_read_config_word(dev, PCI_DEVICE_ID, &dev_id); > + if (ven_id == PCI_VENDOR_ID_INTEL && dev_id == 0x09a2) { > + /*IIO MISC Control offset 0x1c0*/ > + pci_read_config_word(dev, 0x1c0, &en1k); > + } > + } > + /* > + *Intel ICX SPR EMR GNR > + *IIO MISC Control (IIOMISCCTRL_1_5_0_CFG) — Offset 1C0h > + *bit 2:Enable 1K (EN1K) > + *This bit when set, enables 1K granularity for I/O space decode > + *in each of the virtual P2P bridges > + *corresponding to root ports, and DMI ports. > + */ > + if (en1k & 0x4) > + bridge->io_window_1k = 1; > + } I still think this is not going to work because I don't want this kind of device-specific clutter in this generic path. The pcibios_* interfaces are history that we'd like to get rid of also, but it would be better than putting it here. Please follow english conventions as much as you can, e.g., space after "*" in comments, space after period at end of sentence, capitalize first word of sentence, blank line between paragraphs. Most of this you can see by looking at other comments. > pci_read_bridge_io(bridge, &res, true); > } > > -- > 2.39.2 >
> On Thu, Jun 27, 2024 at 12:58:56AM +0000, Zhou Shengqing wrote: > > This patch add 1k granularity for intel root port bridge.Intel latest > > server CPU support 1K granularity,And there is an BIOS setup item named > > "EN1K",but linux doesn't support it. if an IIO has 5 IOU (SPR has 5 IOUs) > > all are bifurcated 2x8.In a 2P server system,There are 20 P2P bridges > > present.if keep 4K granularity allocation,it need 20*4=80k io space, > > exceeding 64k.I test it in a 16*nvidia 4090s system under intel eaglestrem > > platform.There are six 4090s that cannot be allocated I/O resources. > > So I applied this patch.And I found a similar implementation in quirks.c, > > but it only targets the Intel P64H2 platform. > > > > Signed-off-by: Zhou Shengqing <zhoushengqing@ttyinfo.com> > > --- > > drivers/pci/probe.c | 23 +++++++++++++++++++++++ > > 1 file changed, 23 insertions(+) > > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > > index 5fbabb4e3425..909962795311 100644 > > --- a/drivers/pci/probe.c > > +++ b/drivers/pci/probe.c > > @@ -461,6 +461,9 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) > > u32 buses; > > u16 io; > > u32 pmem, tmp; > > + u16 ven_id, dev_id; > > + u16 en1k = 0; > > + struct pci_dev *dev = NULL; > > struct resource res; > > > > pci_read_config_dword(bridge, PCI_PRIMARY_BUS, &buses); > > @@ -478,6 +481,26 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) > > } > > if (io) { > > bridge->io_window = 1; > > + if (pci_is_root_bus(bridge->bus)) { > > + list_for_each_entry(dev, &bridge->bus->devices, bus_list) { > > + pci_read_config_word(dev, PCI_VENDOR_ID, &ven_id); > > + pci_read_config_word(dev, PCI_DEVICE_ID, &dev_id); > > + if (ven_id == PCI_VENDOR_ID_INTEL && dev_id == 0x09a2) { > > + /*IIO MISC Control offset 0x1c0*/ > > + pci_read_config_word(dev, 0x1c0, &en1k); > > + } > > + } > > + /* > > + *Intel ICX SPR EMR GNR > > + *IIO MISC Control (IIOMISCCTRL_1_5_0_CFG) — Offset 1C0h > > + *bit 2:Enable 1K (EN1K) > > + *This bit when set, enables 1K granularity for I/O space decode > > + *in each of the virtual P2P bridges > > + *corresponding to root ports, and DMI ports. > > + */ > > + if (en1k & 0x4) > > + bridge->io_window_1k = 1; > > + } > > I still think this is not going to work because I don't want this kind > of device-specific clutter in this generic path. The pcibios_* > interfaces are history that we'd like to get rid of also, but it would > be better than putting it here. Do you think it should be putted to the pci_bios* interface? And if there is no suitable place to apply this patch, then let's just ignore this issue. > > Please follow english conventions as much as you can, e.g., space > after "*" in comments, space after period at end of sentence, > capitalize first word of sentence, blank line between paragraphs. > Most of this you can see by looking at other comments. > Thank you sincerely for your help with my first patch commitment. > > pci_read_bridge_io(bridge, &res, true); > > } > > > > -- > > 2.39.2 > >
On Sun, Jun 30, 2024 at 02:52:25AM +0000, Zhou Shengqing wrote: > > On Thu, Jun 27, 2024 at 12:58:56AM +0000, Zhou Shengqing wrote: > > > This patch add 1k granularity for intel root port bridge.Intel latest > > > server CPU support 1K granularity,And there is an BIOS setup item named > > > "EN1K",but linux doesn't support it. if an IIO has 5 IOU (SPR has 5 IOUs) > > > all are bifurcated 2x8.In a 2P server system,There are 20 P2P bridges > > > present.if keep 4K granularity allocation,it need 20*4=80k io space, > > > exceeding 64k.I test it in a 16*nvidia 4090s system under intel eaglestrem > > > platform.There are six 4090s that cannot be allocated I/O resources. > > > So I applied this patch.And I found a similar implementation in quirks.c, > > > but it only targets the Intel P64H2 platform. > > > > > > Signed-off-by: Zhou Shengqing <zhoushengqing@ttyinfo.com> > > > --- > > > drivers/pci/probe.c | 23 +++++++++++++++++++++++ > > > 1 file changed, 23 insertions(+) > > > > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > > > index 5fbabb4e3425..909962795311 100644 > > > --- a/drivers/pci/probe.c > > > +++ b/drivers/pci/probe.c > > > @@ -461,6 +461,9 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) > > > u32 buses; > > > u16 io; > > > u32 pmem, tmp; > > > + u16 ven_id, dev_id; > > > + u16 en1k = 0; > > > + struct pci_dev *dev = NULL; > > > struct resource res; > > > > > > pci_read_config_dword(bridge, PCI_PRIMARY_BUS, &buses); > > > @@ -478,6 +481,26 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) > > > } > > > if (io) { > > > bridge->io_window = 1; > > > + if (pci_is_root_bus(bridge->bus)) { > > > + list_for_each_entry(dev, &bridge->bus->devices, bus_list) { > > > + pci_read_config_word(dev, PCI_VENDOR_ID, &ven_id); > > > + pci_read_config_word(dev, PCI_DEVICE_ID, &dev_id); > > > + if (ven_id == PCI_VENDOR_ID_INTEL && dev_id == 0x09a2) { > > > + /*IIO MISC Control offset 0x1c0*/ > > > + pci_read_config_word(dev, 0x1c0, &en1k); > > > + } > > > + } > > > + /* > > > + *Intel ICX SPR EMR GNR > > > + *IIO MISC Control (IIOMISCCTRL_1_5_0_CFG) — Offset 1C0h > > > + *bit 2:Enable 1K (EN1K) > > > + *This bit when set, enables 1K granularity for I/O space decode > > > + *in each of the virtual P2P bridges > > > + *corresponding to root ports, and DMI ports. > > > + */ > > > + if (en1k & 0x4) > > > + bridge->io_window_1k = 1; > > > + } > > > > I still think this is not going to work because I don't want this kind > > of device-specific clutter in this generic path. The pcibios_* > > interfaces are history that we'd like to get rid of also, but it would > > be better than putting it here. > > Do you think it should be putted to the pci_bios* interface? > > And if there is no suitable place to apply this patch, > then let's just ignore this issue. What are the device/function addresses of the IIO and the Root Ports? They must all be on the root bus, and if the IIO is enumerated first (i.e., if it has a smaller device/function number), you may be able to make a quirk that applies to the Root Port vendor/device ID but reads the config bit from the IIO. quirk_passive_release(), quirk_vialatency(), quirk_amd_780_apc_msi(), etc. do similar things. Bjorn
> > I still think this is not going to work because I don't want this kind > > of device-specific clutter in this generic path. The pcibios_* > > interfaces are history that we'd like to get rid of also, but it would > > be better than putting it here. > > Do you think it should be putted to the pci_bios* interface? > > And if there is no suitable place to apply this patch, > then let's just ignore this issue. > What are the device/function addresses of the IIO and the Root Ports? IIO device = 0,function = 0. Root Ports device = 1/2/3/4/5/6/7/8,function = 0, And they are on the same bus. > > They must all be on the root bus, and if the IIO is enumerated first > (i.e., if it has a smaller device/function number), you may be able to > make a quirk that applies to the Root Port vendor/device ID but reads > the config bit from the IIO. > > quirk_passive_release(), quirk_vialatency(), quirk_amd_780_apc_msi(), > etc. do similar things. I have commited a new patch referred to quirk_passive_release(). https://lore.kernel.org/all/20240702035649.26039-1-zhoushengqing@ttyinfo.com/ And I validate it on my system which previously some devices couldn't have IO space. After applying this patch, all devices were able to have IO space. Thank you again for your assistance. > > Bjorn
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 5fbabb4e3425..909962795311 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -461,6 +461,9 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) u32 buses; u16 io; u32 pmem, tmp; + u16 ven_id, dev_id; + u16 en1k = 0; + struct pci_dev *dev = NULL; struct resource res; pci_read_config_dword(bridge, PCI_PRIMARY_BUS, &buses); @@ -478,6 +481,26 @@ static void pci_read_bridge_windows(struct pci_dev *bridge) } if (io) { bridge->io_window = 1; + if (pci_is_root_bus(bridge->bus)) { + list_for_each_entry(dev, &bridge->bus->devices, bus_list) { + pci_read_config_word(dev, PCI_VENDOR_ID, &ven_id); + pci_read_config_word(dev, PCI_DEVICE_ID, &dev_id); + if (ven_id == PCI_VENDOR_ID_INTEL && dev_id == 0x09a2) { + /*IIO MISC Control offset 0x1c0*/ + pci_read_config_word(dev, 0x1c0, &en1k); + } + } + /* + *Intel ICX SPR EMR GNR + *IIO MISC Control (IIOMISCCTRL_1_5_0_CFG) — Offset 1C0h + *bit 2:Enable 1K (EN1K) + *This bit when set, enables 1K granularity for I/O space decode + *in each of the virtual P2P bridges + *corresponding to root ports, and DMI ports. + */ + if (en1k & 0x4) + bridge->io_window_1k = 1; + } pci_read_bridge_io(bridge, &res, true); }
This patch add 1k granularity for intel root port bridge.Intel latest server CPU support 1K granularity,And there is an BIOS setup item named "EN1K",but linux doesn't support it. if an IIO has 5 IOU (SPR has 5 IOUs) all are bifurcated 2x8.In a 2P server system,There are 20 P2P bridges present.if keep 4K granularity allocation,it need 20*4=80k io space, exceeding 64k.I test it in a 16*nvidia 4090s system under intel eaglestrem platform.There are six 4090s that cannot be allocated I/O resources. So I applied this patch.And I found a similar implementation in quirks.c, but it only targets the Intel P64H2 platform. Signed-off-by: Zhou Shengqing <zhoushengqing@ttyinfo.com> --- drivers/pci/probe.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)