| Message ID | 20211116170133.724751-2-fbarrat@linux.ibm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Fix irq allocation of PCI host bridge on powernv | expand |
On 11/16/21 18:01, Frederic Barrat wrote: > The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the > Interrupt Pin register in the config space so that the model matches > the hardware. > > If we don't, then we inherit from the default pcie root bridge, which > requests a LSI. And because we don't map it correctly in the device > tree, all PHBs allocate the same bogus hw interrupt. We end up with > inconsistent interrupt controller (xive) data. The problem goes away > if we don't allocate the LSI in the first place. > > Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> Reviewed-by: Cédric Le Goater <clg@kaod.org> Thanks, C. > --- > hw/pci-host/pnv_phb4.c | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c > index 5c375a9f28..1659d55b4f 100644 > --- a/hw/pci-host/pnv_phb4.c > +++ b/hw/pci-host/pnv_phb4.c > @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) > PCIDevice *root_dev = PCI_DEVICE(&phb->root); > > /* > - * Configure PCI device id at reset using a property. > + * Configure the PCI device at reset: > + * - set the Vendor and Device ID to for the root bridge > + * - no LSI > */ > pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); > pci_config_set_device_id(root_dev->config, phb->device_id); > + pci_config_set_interrupt_pin(root_dev->config, 0); > } > > static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, >
On 11/16/21 18:01, Frederic Barrat wrote: > The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the > Interrupt Pin register in the config space so that the model matches > the hardware. > > If we don't, then we inherit from the default pcie root bridge, which > requests a LSI. And because we don't map it correctly in the device > tree, all PHBs allocate the same bogus hw interrupt. We end up with > inconsistent interrupt controller (xive) data. The problem goes away > if we don't allocate the LSI in the first place. > > Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> > --- > hw/pci-host/pnv_phb4.c | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c > index 5c375a9f28..1659d55b4f 100644 > --- a/hw/pci-host/pnv_phb4.c > +++ b/hw/pci-host/pnv_phb4.c > @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) > PCIDevice *root_dev = PCI_DEVICE(&phb->root); > > /* > - * Configure PCI device id at reset using a property. > + * Configure the PCI device at reset: > + * - set the Vendor and Device ID to for the root bridge > + * - no LSI > */ > pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); > pci_config_set_device_id(root_dev->config, phb->device_id); > + pci_config_set_interrupt_pin(root_dev->config, 0); > } > > static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, > FYI, I am seeing an issue with FreeBSD when booting from iso : https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz Thanks, C. SIGTERM received, booting... KDB: debugger backends: ddb KDB: current backend: ddb ---<<BOOT>>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 powerpc FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a82f) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. ofw_initrd: initrd loaded at 0x28000000-0x28c7928c cpu0: IBM POWER9 revision 2.0, 1000.00 MHz cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE> cpu0: Features2 bee00000<ARCH207,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN> real memory = 1014484992 (967 MB) avail memory = 117903360 (112 MB) random: registering fast source PowerISA DARN random number generator random: fast provider: "PowerISA DARN random number generator" arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. random: entropy device external interface kbd0 at kbdmux0 ofwbus0: <Open Firmware Device Tree> on nexus0 opal0: <OPAL Abstraction Firmware> irq 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 on ofwbus0 opal0: registered as a time-of-day clock, resolution 0.002000s simplebus0: <Flattened device tree simple bus> mem 0x6030000000000-0x60300ffffffff on ofwbus0 pcib0: <OPAL Host-PCI bridge> mem 0x600c3c0000000-0x600c3c0000fff,0x600c300000000-0x600c30fffffff on ofwbus0 pci0: <OFW PCI bus> numa-domain 0 on pcib0 qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
[ Adding Alfredo the thread ] On 11/26/21 10:09, Cédric Le Goater wrote: > On 11/16/21 18:01, Frederic Barrat wrote: >> The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the >> Interrupt Pin register in the config space so that the model matches >> the hardware. >> >> If we don't, then we inherit from the default pcie root bridge, which >> requests a LSI. And because we don't map it correctly in the device >> tree, all PHBs allocate the same bogus hw interrupt. We end up with >> inconsistent interrupt controller (xive) data. The problem goes away >> if we don't allocate the LSI in the first place. >> >> Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> >> --- >> hw/pci-host/pnv_phb4.c | 5 ++++- >> 1 file changed, 4 insertions(+), 1 deletion(-) >> >> diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c >> index 5c375a9f28..1659d55b4f 100644 >> --- a/hw/pci-host/pnv_phb4.c >> +++ b/hw/pci-host/pnv_phb4.c >> @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) >> PCIDevice *root_dev = PCI_DEVICE(&phb->root); >> /* >> - * Configure PCI device id at reset using a property. >> + * Configure the PCI device at reset: >> + * - set the Vendor and Device ID to for the root bridge >> + * - no LSI >> */ >> pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); >> pci_config_set_device_id(root_dev->config, phb->device_id); >> + pci_config_set_interrupt_pin(root_dev->config, 0); >> } >> static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, >> > > FYI, I am seeing an issue with FreeBSD when booting from iso : > > https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz > > Thanks, > > C. > > SIGTERM received, booting... > KDB: debugger backends: ddb > KDB: current backend: ddb > ---<<BOOT>>--- > Copyright (c) 1992-2021 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 UTC 2021 > root@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 powerpc > FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a82f) > WARNING: WITNESS option enabled, expect reduced performance. > VT: init without driver. > ofw_initrd: initrd loaded at 0x28000000-0x28c7928c > cpu0: IBM POWER9 revision 2.0, 1000.00 MHz > cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE> > cpu0: Features2 bee00000<ARCH207,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN> > real memory = 1014484992 (967 MB) > avail memory = 117903360 (112 MB) > random: registering fast source PowerISA DARN random number generator > random: fast provider: "PowerISA DARN random number generator" > arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. > random: entropy device external interface > kbd0 at kbdmux0 > ofwbus0: <Open Firmware Device Tree> on nexus0 > opal0: <OPAL Abstraction Firmware> irq 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 on ofwbus0 > opal0: registered as a time-of-day clock, resolution 0.002000s > simplebus0: <Flattened device tree simple bus> mem 0x6030000000000-0x60300ffffffff on ofwbus0 > pcib0: <OPAL Host-PCI bridge> mem 0x600c3c0000000-0x600c3c0000fff,0x600c300000000-0x600c30fffffff on ofwbus0 > pci0: <OFW PCI bus> numa-domain 0 on pcib0 > qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed. > >
On Fri, Nov 26, 2021 at 06:08:30PM +0100, Cédric Le Goater wrote: > [ Adding Alfredo the thread ] > > On 11/26/21 10:09, Cédric Le Goater wrote: > > On 11/16/21 18:01, Frederic Barrat wrote: > > > The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the > > > Interrupt Pin register in the config space so that the model matches > > > the hardware. > > > > > > If we don't, then we inherit from the default pcie root bridge, which > > > requests a LSI. And because we don't map it correctly in the device > > > tree, all PHBs allocate the same bogus hw interrupt. We end up with > > > inconsistent interrupt controller (xive) data. The problem goes away > > > if we don't allocate the LSI in the first place. > > > > > > Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> > > > --- > > > hw/pci-host/pnv_phb4.c | 5 ++++- > > > 1 file changed, 4 insertions(+), 1 deletion(-) > > > > > > diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c > > > index 5c375a9f28..1659d55b4f 100644 > > > --- a/hw/pci-host/pnv_phb4.c > > > +++ b/hw/pci-host/pnv_phb4.c > > > @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) > > > PCIDevice *root_dev = PCI_DEVICE(&phb->root); > > > /* > > > - * Configure PCI device id at reset using a property. > > > + * Configure the PCI device at reset: > > > + * - set the Vendor and Device ID to for the root bridge > > > + * - no LSI > > > */ > > > pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); > > > pci_config_set_device_id(root_dev->config, phb->device_id); > > > + pci_config_set_interrupt_pin(root_dev->config, 0); > > > } > > > static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, > > > > > > > FYI, I am seeing an issue with FreeBSD when booting from iso : > > > > https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz > > > > Thanks, > > > > C. > > > > SIGTERM received, booting... > > KDB: debugger backends: ddb > > KDB: current backend: ddb > > ---<<BOOT>>--- > > Copyright (c) 1992-2021 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 UTC 2021 > > root@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 powerpc > > FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a82f) > > WARNING: WITNESS option enabled, expect reduced performance. > > VT: init without driver. > > ofw_initrd: initrd loaded at 0x28000000-0x28c7928c > > cpu0: IBM POWER9 revision 2.0, 1000.00 MHz > > cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE> > > cpu0: Features2 bee00000<ARCH207,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN> > > real memory = 1014484992 (967 MB) > > avail memory = 117903360 (112 MB) > > random: registering fast source PowerISA DARN random number generator > > random: fast provider: "PowerISA DARN random number generator" > > arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. > > random: entropy device external interface > > kbd0 at kbdmux0 > > ofwbus0: <Open Firmware Device Tree> on nexus0 > > opal0: <OPAL Abstraction Firmware> irq 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 on ofwbus0 > > opal0: registered as a time-of-day clock, resolution 0.002000s > > simplebus0: <Flattened device tree simple bus> mem 0x6030000000000-0x60300ffffffff on ofwbus0 > > pcib0: <OPAL Host-PCI bridge> mem 0x600c3c0000000-0x600c3c0000fff,0x600c300000000-0x600c30fffffff on ofwbus0 > > pci0: <OFW PCI bus> numa-domain 0 on pcib0 > > qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed. > > Frederic?
On 26/11/2021 10:09, Cédric Le Goater wrote: > On 11/16/21 18:01, Frederic Barrat wrote: >> The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the >> Interrupt Pin register in the config space so that the model matches >> the hardware. >> >> If we don't, then we inherit from the default pcie root bridge, which >> requests a LSI. And because we don't map it correctly in the device >> tree, all PHBs allocate the same bogus hw interrupt. We end up with >> inconsistent interrupt controller (xive) data. The problem goes away >> if we don't allocate the LSI in the first place. >> >> Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> >> --- >> hw/pci-host/pnv_phb4.c | 5 ++++- >> 1 file changed, 4 insertions(+), 1 deletion(-) >> >> diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c >> index 5c375a9f28..1659d55b4f 100644 >> --- a/hw/pci-host/pnv_phb4.c >> +++ b/hw/pci-host/pnv_phb4.c >> @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) >> PCIDevice *root_dev = PCI_DEVICE(&phb->root); >> /* >> - * Configure PCI device id at reset using a property. >> + * Configure the PCI device at reset: >> + * - set the Vendor and Device ID to for the root bridge >> + * - no LSI >> */ >> pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); >> pci_config_set_device_id(root_dev->config, phb->device_id); >> + pci_config_set_interrupt_pin(root_dev->config, 0); >> } >> static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, >> > > FYI, I am seeing an issue with FreeBSD when booting from iso : > > > https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz > > I see what's going on... Since the phb4 model borrows most of its code from the pcie_root bridge, there are several instances of code such as: if (msix_enabled(dev)) { do something; } else if (msi_enabled(dev)) { do something else; } else { yet something else which assumes a LSI; } With this series, I removed the LSI from the phb4 root port to match the hardware and fixed one such code pattern in patch 3. But there are others, and we hit one of those when installing from the free bsd iso. So this is going to need more work. Fred > Thanks, > > C. > > SIGTERM received, booting... > KDB: debugger backends: ddb > KDB: current backend: ddb > ---<<BOOT>>--- > Copyright (c) 1992-2021 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 > UTC 2021 > > root@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 > powerpc > FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git > llvmorg-12.0.1-0-gfed41342a82f) > WARNING: WITNESS option enabled, expect reduced performance. > VT: init without driver. > ofw_initrd: initrd loaded at 0x28000000-0x28c7928c > cpu0: IBM POWER9 revision 2.0, 1000.00 MHz > cpu0: Features > dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE> > cpu0: Features2 > bee00000<ARCH207,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN> > real memory = 1014484992 (967 MB) > avail memory = 117903360 (112 MB) > random: registering fast source PowerISA DARN random number generator > random: fast provider: "PowerISA DARN random number generator" > arc4random: WARNING: initial seeding bypassed the cryptographic random > device because it was not yet seeded and the knob > 'bypass_before_seeding' was enabled. > random: entropy device external interface > kbd0 at kbdmux0 > ofwbus0: <Open Firmware Device Tree> on nexus0 > opal0: <OPAL Abstraction Firmware> irq > 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 > on ofwbus0 > opal0: registered as a time-of-day clock, resolution 0.002000s > simplebus0: <Flattened device tree simple bus> mem > 0x6030000000000-0x60300ffffffff on ofwbus0 > pcib0: <OPAL Host-PCI bridge> mem > 0x600c3c0000000-0x600c3c0000fff,0x600c300000000-0x600c30fffffff on ofwbus0 > pci0: <OFW PCI bus> numa-domain 0 on pcib0 > qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 > <= irq_num && irq_num < PCI_NUM_PINS' failed. > > >
diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c index 5c375a9f28..1659d55b4f 100644 --- a/hw/pci-host/pnv_phb4.c +++ b/hw/pci-host/pnv_phb4.c @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) PCIDevice *root_dev = PCI_DEVICE(&phb->root); /* - * Configure PCI device id at reset using a property. + * Configure the PCI device at reset: + * - set the Vendor and Device ID to for the root bridge + * - no LSI */ pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); pci_config_set_device_id(root_dev->config, phb->device_id); + pci_config_set_interrupt_pin(root_dev->config, 0); } static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge,
The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the Interrupt Pin register in the config space so that the model matches the hardware. If we don't, then we inherit from the default pcie root bridge, which requests a LSI. And because we don't map it correctly in the device tree, all PHBs allocate the same bogus hw interrupt. We end up with inconsistent interrupt controller (xive) data. The problem goes away if we don't allocate the LSI in the first place. Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> --- hw/pci-host/pnv_phb4.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)