| Message ID | 20210322184614.802565-13-maz@kernel.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | PCI/MSI: Getting rid of msi_controller, and other cleanups | expand |
On 2021-03-22 18:46, Marc Zyngier wrote: > The new 'no_msi' attribute solves the problem of advertising the lack > of MSI capability for host bridges that know for sure that there will > be no MSI for their end-points. > > However, there is a whole class of host bridges that cannot know > whether MSIs will be provided or not, as they rely on other blocks > to provide the MSI functionnality, using MSI domains. This is > the case for example on systems that use the ARM GIC architecture. > > Introduce a new attribute ('msi_domain') indicating that implicit > dependency, and use this property to set the NO_MSI flag when > no MSI domain is found at probe time. > > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > Signed-off-by: Marc Zyngier <maz@kernel.org> > --- > drivers/pci/probe.c | 2 +- > include/linux/pci.h | 1 + > 2 files changed, 2 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > index 146bd85c037e..bac9f69a06a8 100644 > --- a/drivers/pci/probe.c > +++ b/drivers/pci/probe.c > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) > device_enable_async_suspend(bus->bridge); > pci_set_bus_of_node(bus); > pci_set_bus_msi_domain(bus); > - if (bridge->no_msi) > + if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain)) > bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; > > if (!parent) > diff --git a/include/linux/pci.h b/include/linux/pci.h > index 48605cca82ae..d322d00db432 100644 > --- a/include/linux/pci.h > +++ b/include/linux/pci.h > @@ -551,6 +551,7 @@ struct pci_host_bridge { > unsigned int preserve_config:1; /* Preserve FW resource setup */ > unsigned int size_windows:1; /* Enable root bus sizing */ > unsigned int no_msi:1; /* Bridge has no MSI support */ > + unsigned int msi_domain:1; /* Bridge wants MSI domain */ Aren't these really the same thing? Either way we're saying the bridge itself doesn't handle MSIs, it's just in one case we're effectively encoding a platform-specific assumption that an external domain won't be provided. I can't help wondering whether that distinction is really necessary... Robin. > > /* Resource alignment requirements */ > resource_size_t (*align_resource)(struct pci_dev *dev, >
Hi Robin, On Tue, 23 Mar 2021 11:45:02 +0000, Robin Murphy <robin.murphy@arm.com> wrote: > > On 2021-03-22 18:46, Marc Zyngier wrote: > > The new 'no_msi' attribute solves the problem of advertising the lack > > of MSI capability for host bridges that know for sure that there will > > be no MSI for their end-points. > > > > However, there is a whole class of host bridges that cannot know > > whether MSIs will be provided or not, as they rely on other blocks > > to provide the MSI functionnality, using MSI domains. This is > > the case for example on systems that use the ARM GIC architecture. > > > > Introduce a new attribute ('msi_domain') indicating that implicit > > dependency, and use this property to set the NO_MSI flag when > > no MSI domain is found at probe time. > > > > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > > Signed-off-by: Marc Zyngier <maz@kernel.org> > > --- > > drivers/pci/probe.c | 2 +- > > include/linux/pci.h | 1 + > > 2 files changed, 2 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > > index 146bd85c037e..bac9f69a06a8 100644 > > --- a/drivers/pci/probe.c > > +++ b/drivers/pci/probe.c > > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) > > device_enable_async_suspend(bus->bridge); > > pci_set_bus_of_node(bus); > > pci_set_bus_msi_domain(bus); > > - if (bridge->no_msi) > > + if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain)) > > bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; > > if (!parent) > > diff --git a/include/linux/pci.h b/include/linux/pci.h > > index 48605cca82ae..d322d00db432 100644 > > --- a/include/linux/pci.h > > +++ b/include/linux/pci.h > > @@ -551,6 +551,7 @@ struct pci_host_bridge { > > unsigned int preserve_config:1; /* Preserve FW resource setup */ > > unsigned int size_windows:1; /* Enable root bus sizing */ > > unsigned int no_msi:1; /* Bridge has no MSI support */ > > + unsigned int msi_domain:1; /* Bridge wants MSI domain */ > > Aren't these really the same thing? Either way we're saying the bridge > itself doesn't handle MSIs, it's just in one case we're effectively > encoding a platform-specific assumption that an external domain won't > be provided. I can't help wondering whether that distinction is really > necessary... There is a subtle difference: no_msi indicates that there is no way *any* MSI can be dealt with whatsoever (maybe because the RC doesn't forward the corresponding TLPs?). msi_domain says "no MSI unless...". We could implement the former with the latter, but I have the feeling that's not totally bullet proof. Happy to revisit this if you think it really matters. Thanks, M.
On 2021-03-23 18:09, Marc Zyngier wrote: > Hi Robin, > > On Tue, 23 Mar 2021 11:45:02 +0000, > Robin Murphy <robin.murphy@arm.com> wrote: >> >> On 2021-03-22 18:46, Marc Zyngier wrote: >>> The new 'no_msi' attribute solves the problem of advertising the lack >>> of MSI capability for host bridges that know for sure that there will >>> be no MSI for their end-points. >>> >>> However, there is a whole class of host bridges that cannot know >>> whether MSIs will be provided or not, as they rely on other blocks >>> to provide the MSI functionnality, using MSI domains. This is >>> the case for example on systems that use the ARM GIC architecture. >>> >>> Introduce a new attribute ('msi_domain') indicating that implicit >>> dependency, and use this property to set the NO_MSI flag when >>> no MSI domain is found at probe time. >>> >>> Acked-by: Bjorn Helgaas <bhelgaas@google.com> >>> Signed-off-by: Marc Zyngier <maz@kernel.org> >>> --- >>> drivers/pci/probe.c | 2 +- >>> include/linux/pci.h | 1 + >>> 2 files changed, 2 insertions(+), 1 deletion(-) >>> >>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c >>> index 146bd85c037e..bac9f69a06a8 100644 >>> --- a/drivers/pci/probe.c >>> +++ b/drivers/pci/probe.c >>> @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) >>> device_enable_async_suspend(bus->bridge); >>> pci_set_bus_of_node(bus); >>> pci_set_bus_msi_domain(bus); >>> - if (bridge->no_msi) >>> + if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain)) >>> bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; >>> if (!parent) >>> diff --git a/include/linux/pci.h b/include/linux/pci.h >>> index 48605cca82ae..d322d00db432 100644 >>> --- a/include/linux/pci.h >>> +++ b/include/linux/pci.h >>> @@ -551,6 +551,7 @@ struct pci_host_bridge { >>> unsigned int preserve_config:1; /* Preserve FW resource setup */ >>> unsigned int size_windows:1; /* Enable root bus sizing */ >>> unsigned int no_msi:1; /* Bridge has no MSI support */ >>> + unsigned int msi_domain:1; /* Bridge wants MSI domain */ >> >> Aren't these really the same thing? Either way we're saying the bridge >> itself doesn't handle MSIs, it's just in one case we're effectively >> encoding a platform-specific assumption that an external domain won't >> be provided. I can't help wondering whether that distinction is really >> necessary... > > There is a subtle difference: no_msi indicates that there is no way > *any* MSI can be dealt with whatsoever (maybe because the RC doesn't > forward the corresponding TLPs?). msi_domain says "no MSI unless...". PCI says that MSIs are simply memory writes at the transaction level, so AFAIK unless the host bridge can't support DMA at all, it shouldn't be in a position to make any assumptions about what transactions mean what. I suppose there could in theory be an issue in the other direction, where config space somehow didn't allow access to the MSI capability in the first place, but then we'd presumably just never detect any device as being MSI-capable in the first place, and it wouldn't matter either way. > We could implement the former with the latter, but I have the feeling > that's not totally bullet proof. Happy to revisit this if you think it > really matters. I would expect it to be a fairly safe assumption that a host bridge which "doesn't support MSIs" wouldn't have an msi-parent set, but I don't have a strong opinion either way - I just figured we could probably save a little bit of complexity here. Cheers, Robin.
On Tue, Mar 23, 2021 at 06:09:36PM +0000, Marc Zyngier wrote: > Hi Robin, > > On Tue, 23 Mar 2021 11:45:02 +0000, > Robin Murphy <robin.murphy@arm.com> wrote: > > > > On 2021-03-22 18:46, Marc Zyngier wrote: > > > The new 'no_msi' attribute solves the problem of advertising the lack > > > of MSI capability for host bridges that know for sure that there will > > > be no MSI for their end-points. > > > > > > However, there is a whole class of host bridges that cannot know > > > whether MSIs will be provided or not, as they rely on other blocks > > > to provide the MSI functionnality, using MSI domains. This is > > > the case for example on systems that use the ARM GIC architecture. > > > > > > Introduce a new attribute ('msi_domain') indicating that implicit > > > dependency, and use this property to set the NO_MSI flag when > > > no MSI domain is found at probe time. > > > > > > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > > > Signed-off-by: Marc Zyngier <maz@kernel.org> > > > --- > > > drivers/pci/probe.c | 2 +- > > > include/linux/pci.h | 1 + > > > 2 files changed, 2 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > > > index 146bd85c037e..bac9f69a06a8 100644 > > > --- a/drivers/pci/probe.c > > > +++ b/drivers/pci/probe.c > > > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) > > > device_enable_async_suspend(bus->bridge); > > > pci_set_bus_of_node(bus); > > > pci_set_bus_msi_domain(bus); > > > - if (bridge->no_msi) > > > + if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain)) > > > bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; > > > if (!parent) > > > diff --git a/include/linux/pci.h b/include/linux/pci.h > > > index 48605cca82ae..d322d00db432 100644 > > > --- a/include/linux/pci.h > > > +++ b/include/linux/pci.h > > > @@ -551,6 +551,7 @@ struct pci_host_bridge { > > > unsigned int preserve_config:1; /* Preserve FW resource setup */ > > > unsigned int size_windows:1; /* Enable root bus sizing */ > > > unsigned int no_msi:1; /* Bridge has no MSI support */ > > > + unsigned int msi_domain:1; /* Bridge wants MSI domain */ > > > > Aren't these really the same thing? Either way we're saying the bridge > > itself doesn't handle MSIs, it's just in one case we're effectively > > encoding a platform-specific assumption that an external domain won't > > be provided. I can't help wondering whether that distinction is really > > necessary... > > There is a subtle difference: no_msi indicates that there is no way > *any* MSI can be dealt with whatsoever (maybe because the RC doesn't > forward the corresponding TLPs?). msi_domain says "no MSI unless...". > > We could implement the former with the latter, but I have the feeling > that's not totally bullet proof. Happy to revisit this if you think it > really matters. IIUC msi_domain == 1 means: this host bridge needs an msi_domain to enable MSIs, which in turn means that there are bridges that do _not_ require an msi_domain to enable MSIs. I don't know how other arches handle the msi_domain pointer but I am asking whether making: if (bridge->no_msi || !bus->dev.msi_domain)) bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; is a possibility (removing the need for the msi_domain flag). At least this looks more like an arch property than a host bridge specific property (eg patch [13] pci_host_common_probe() may be used on arches other than ARM where it is not necessary true that it requires an msi_domain to enable MSIs). I agree that's complicated to untangle - just asking if there is way to simplify it. Thanks, Lorenzo > Thanks, > > M. > > -- > Without deviation from the norm, progress is not possible.
On Wed, 24 Mar 2021 13:19:38 +0000, Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> wrote: > > On Tue, Mar 23, 2021 at 06:09:36PM +0000, Marc Zyngier wrote: > > Hi Robin, > > > > On Tue, 23 Mar 2021 11:45:02 +0000, > > Robin Murphy <robin.murphy@arm.com> wrote: > > > > > > On 2021-03-22 18:46, Marc Zyngier wrote: > > > > The new 'no_msi' attribute solves the problem of advertising the lack > > > > of MSI capability for host bridges that know for sure that there will > > > > be no MSI for their end-points. > > > > > > > > However, there is a whole class of host bridges that cannot know > > > > whether MSIs will be provided or not, as they rely on other blocks > > > > to provide the MSI functionnality, using MSI domains. This is > > > > the case for example on systems that use the ARM GIC architecture. > > > > > > > > Introduce a new attribute ('msi_domain') indicating that implicit > > > > dependency, and use this property to set the NO_MSI flag when > > > > no MSI domain is found at probe time. > > > > > > > > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > > > > Signed-off-by: Marc Zyngier <maz@kernel.org> > > > > --- > > > > drivers/pci/probe.c | 2 +- > > > > include/linux/pci.h | 1 + > > > > 2 files changed, 2 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > > > > index 146bd85c037e..bac9f69a06a8 100644 > > > > --- a/drivers/pci/probe.c > > > > +++ b/drivers/pci/probe.c > > > > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) > > > > device_enable_async_suspend(bus->bridge); > > > > pci_set_bus_of_node(bus); > > > > pci_set_bus_msi_domain(bus); > > > > - if (bridge->no_msi) > > > > + if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain)) > > > > bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; > > > > if (!parent) > > > > diff --git a/include/linux/pci.h b/include/linux/pci.h > > > > index 48605cca82ae..d322d00db432 100644 > > > > --- a/include/linux/pci.h > > > > +++ b/include/linux/pci.h > > > > @@ -551,6 +551,7 @@ struct pci_host_bridge { > > > > unsigned int preserve_config:1; /* Preserve FW resource setup */ > > > > unsigned int size_windows:1; /* Enable root bus sizing */ > > > > unsigned int no_msi:1; /* Bridge has no MSI support */ > > > > + unsigned int msi_domain:1; /* Bridge wants MSI domain */ > > > > > > Aren't these really the same thing? Either way we're saying the bridge > > > itself doesn't handle MSIs, it's just in one case we're effectively > > > encoding a platform-specific assumption that an external domain won't > > > be provided. I can't help wondering whether that distinction is really > > > necessary... > > > > There is a subtle difference: no_msi indicates that there is no way > > *any* MSI can be dealt with whatsoever (maybe because the RC doesn't > > forward the corresponding TLPs?). msi_domain says "no MSI unless...". > > > > We could implement the former with the latter, but I have the feeling > > that's not totally bullet proof. Happy to revisit this if you think it > > really matters. > > IIUC msi_domain == 1 means: this host bridge needs an msi_domain to enable > MSIs, which in turn means that there are bridges that do _not_ require > an msi_domain to enable MSIs. I don't know how other arches handle the > msi_domain pointer but I am asking whether making: > > if (bridge->no_msi || !bus->dev.msi_domain)) > bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; > > is a possibility (removing the need for the msi_domain flag). > > At least this looks more like an arch property than a host bridge > specific property (eg patch [13] pci_host_common_probe() may be used on > arches other than ARM where it is not necessary true that it requires an > msi_domain to enable MSIs). > > I agree that's complicated to untangle - just asking if there is way > to simplify it. I tried to simplify that in the past (see the original discussion at [1]), and tglx reported some breakages on systems that do not use MSI domains, which is why we ended up with an explicit flag. What I have done for now is to go with Robin's proposal of dropping 'no_msi' and rely on solely on 'msi_domain' to set PCI_BUS_FLAGS_NO_MSI when no domain is found. Note that if we indeed have a host bridge that uses pci_host_common_probe() that doesn't use MSI domains, we may indeed run into problems. I don't have a good way around that, unfortunately. Thanks, M. [1] https://lore.kernel.org/r/20201031140330.83768-1-linux@fw-web.de
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 146bd85c037e..bac9f69a06a8 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) device_enable_async_suspend(bus->bridge); pci_set_bus_of_node(bus); pci_set_bus_msi_domain(bus); - if (bridge->no_msi) + if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain)) bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI; if (!parent) diff --git a/include/linux/pci.h b/include/linux/pci.h index 48605cca82ae..d322d00db432 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -551,6 +551,7 @@ struct pci_host_bridge { unsigned int preserve_config:1; /* Preserve FW resource setup */ unsigned int size_windows:1; /* Enable root bus sizing */ unsigned int no_msi:1; /* Bridge has no MSI support */ + unsigned int msi_domain:1; /* Bridge wants MSI domain */ /* Resource alignment requirements */ resource_size_t (*align_resource)(struct pci_dev *dev,
The new 'no_msi' attribute solves the problem of advertising the lack of MSI capability for host bridges that know for sure that there will be no MSI for their end-points. However, there is a whole class of host bridges that cannot know whether MSIs will be provided or not, as they rely on other blocks to provide the MSI functionnality, using MSI domains. This is the case for example on systems that use the ARM GIC architecture. Introduce a new attribute ('msi_domain') indicating that implicit dependency, and use this property to set the NO_MSI flag when no MSI domain is found at probe time. Acked-by: Bjorn Helgaas <bhelgaas@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org> --- drivers/pci/probe.c | 2 +- include/linux/pci.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)