| Message ID | 1534340781-19194-1-git-send-email-adouglas@cadence.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add MSI-X support for cadence EP driver | expand |
On August 15, 2018 4:46:21 PM GMT+03:00, Alan Douglas <adouglas@cadence.com> wrote: >The patch implements MSI-X support in the cadence endpoint driver. > >This patch depends on on Gustavo Pimentel's patch series adding MSI-X >support for EP ("Add MSI-X support on pcitest tool") > >It also adds fixes for MSI issues discovered during testing of MSI-X > - Use AXI region 0 for interrupt signalling > - Write MSI and MSI-X with 32bit value rather than 16bit > - Check for masking before sending MSI or MSI-X > - Check link is up before sending IRQ > Hi. AFAIK the BIOS allocates physical memory for the bars. Assuming that the MSIx bar is only mapped after kernel boots on the endpoint, could it be too late? Do we need to trigger re-enumeration of the PCI bus from host side when working with this as an endpoint? Thanks, Ramon >Changes since v1: > - Rebased on 4.18-rc1 > - Update commit log to mark first 4 patches as fixes > - Correct formatting issues pointed out by checkpatch --strict > >Alan Douglas (5): > PCI: cadence: Use AXI region 0 to signal interrupts from EP > PCI: cadence: Write MSI data with 32bits > PCI: cadence: Check whether MSI is masked before sending it > PCI: cadence: Check link is up before sending IRQ from EP > PCI: cadence: Add MSI-X capability to EP driver > >drivers/pci/controller/pcie-cadence-ep.c | 131 >+++++++++++++++++++++++++++++-- > drivers/pci/controller/pcie-cadence.h | 1 + > 2 files changed, 125 insertions(+), 7 deletions(-)
Hi Ramon, On 15 August 2018 17:56, Ramon Fried wrote: > On August 15, 2018 4:46:21 PM GMT+03:00, Alan Douglas <adouglas@cadence.com> wrote: > >The patch implements MSI-X support in the cadence endpoint driver. > > > >This patch depends on on Gustavo Pimentel's patch series adding MSI-X > >support for EP ("Add MSI-X support on pcitest tool") > > > >It also adds fixes for MSI issues discovered during testing of MSI-X > > - Use AXI region 0 for interrupt signalling > > - Write MSI and MSI-X with 32bit value rather than 16bit > > - Check for masking before sending MSI or MSI-X > > - Check link is up before sending IRQ > > > Hi. > AFAIK the BIOS allocates physical memory for the bars. Assuming that the MSIx bar is only mapped after kernel boots on the endpoint, > could it be too late? > > Do we need to trigger re-enumeration of the PCI bus from host side when working with this as an endpoint? It depends on how you are using it. PF0 is always enabled in the cadence HW, so will be enumerated at boot, as long as the EP HW is out of reset and PHY is enabled. The PCIe EP hardware can be initialized so that BARs are enabled by default, before the kernel boots on the endpoint, and so they will be found and mapped during the initial enumeration and you don't need to re-enumerate. The MSI-X vectors can't be written to the BAR until the EP kernel has booted and the EP driver has mapped the BAR to local EP memory though (unless you also configure this in the PCIe EP hardware, or in EP pre-boot, but in that case you are probably not using the EP driver framework.) The EP driver framework does, in my understanding, generally expect re-enumeration after the EP kernel has booted and the driver has been initialized, since it allows configuration of device ID, BAR sizes etc., and if you change any of these from the HW defaults at boot you will need to re-enumerate. All tests I have done for the EP driver have been without BIOS enumeration, and triggering re-enumeration after initializing the BAR sizes etc. via the EP driver Regards, Alan
On Fri, Aug 17, 2018 at 7:05 AM Ramon Fried <ramon.fried@gmail.com> wrote: > > > > On Thu, Aug 16, 2018 at 5:28 PM Alan Douglas <adouglas@cadence.com> wrote: >> >> Hi Ramon, >> >> On 15 August 2018 17:56, Ramon Fried wrote: >> > On August 15, 2018 4:46:21 PM GMT+03:00, Alan Douglas <adouglas@cadence.com> wrote: >> > >The patch implements MSI-X support in the cadence endpoint driver. >> > > >> > >This patch depends on on Gustavo Pimentel's patch series adding MSI-X >> > >support for EP ("Add MSI-X support on pcitest tool") >> > > >> > >It also adds fixes for MSI issues discovered during testing of MSI-X >> > > - Use AXI region 0 for interrupt signalling >> > > - Write MSI and MSI-X with 32bit value rather than 16bit >> > > - Check for masking before sending MSI or MSI-X >> > > - Check link is up before sending IRQ >> > > >> > Hi. >> > AFAIK the BIOS allocates physical memory for the bars. Assuming that the MSIx bar is only mapped after kernel boots on the endpoint, >> > could it be too late? >> > >> > Do we need to trigger re-enumeration of the PCI bus from host side when working with this as an endpoint? >> It depends on how you are using it. PF0 is always enabled in the cadence HW, so will be enumerated at boot, >> as long as the EP HW is out of reset and PHY is enabled. >> The PCIe EP hardware can be initialized so that BARs are enabled by default, before the kernel boots on the >> endpoint, and so they will be found and mapped during the initial enumeration and you don't need to >> re-enumerate. The MSI-X vectors can't be written to the BAR until the EP kernel has booted and the EP driver >> has mapped the BAR to local EP memory though (unless you also configure this in the PCIe EP hardware, or in >> EP pre-boot, but in that case you are probably not using the EP driver framework.) >> >> The EP driver framework does, in my understanding, generally expect re-enumeration after the >> EP kernel has booted and the driver has been initialized, since it allows configuration of device ID, BAR >> sizes etc., and if you change any of these from the HW defaults at boot you will need to re-enumerate. >> This basically means that this driver is for hobbyist / enthusiast, you can't base a real product expecting re-enumeration of the bus. right ? > >> All tests I have done for the EP driver have been without BIOS enumeration, and triggering re-enumeration >> after initializing the BAR sizes etc. via the EP driver >> >> Regards, >> Alan >>
Hi Ramon, On 17 August 2018 05:09, Ramon Fried wrote: > On Fri, Aug 17, 2018 at 7:05 AM Ramon Fried <ramon.fried@gmail.com> wrote: > > On Thu, Aug 16, 2018 at 5:28 PM Alan Douglas <adouglas@cadence.com> wrote: > >> On 15 August 2018 17:56, Ramon Fried wrote: > >> > On August 15, 2018 4:46:21 PM GMT+03:00, Alan Douglas <adouglas@cadence.com> wrote: > >> > >The patch implements MSI-X support in the cadence endpoint driver. > >> > > > >> > >This patch depends on on Gustavo Pimentel's patch series adding MSI-X > >> > >support for EP ("Add MSI-X support on pcitest tool") > >> > > > >> > >It also adds fixes for MSI issues discovered during testing of MSI-X > >> > > - Use AXI region 0 for interrupt signalling > >> > > - Write MSI and MSI-X with 32bit value rather than 16bit > >> > > - Check for masking before sending MSI or MSI-X > >> > > - Check link is up before sending IRQ > >> > > > >> > Hi. > >> > AFAIK the BIOS allocates physical memory for the bars. Assuming that the MSIx bar is only mapped after kernel boots on the > endpoint, > >> > could it be too late? > >> > > >> > Do we need to trigger re-enumeration of the PCI bus from host side when working with this as an endpoint? > >> It depends on how you are using it. PF0 is always enabled in the cadence HW, so will be enumerated at boot, > >> as long as the EP HW is out of reset and PHY is enabled. > >> The PCIe EP hardware can be initialized so that BARs are enabled by default, before the kernel boots on the > >> endpoint, and so they will be found and mapped during the initial enumeration and you don't need to > >> re-enumerate. The MSI-X vectors can't be written to the BAR until the EP kernel has booted and the EP driver > >> has mapped the BAR to local EP memory though (unless you also configure this in the PCIe EP hardware, or in > >> EP pre-boot, but in that case you are probably not using the EP driver framework.) > >> > >> The EP driver framework does, in my understanding, generally expect re-enumeration after the > >> EP kernel has booted and the driver has been initialized, since it allows configuration of device ID, BAR > >> sizes etc., and if you change any of these from the HW defaults at boot you will need to re-enumerate. > >> > This basically means that this driver is for hobbyist / enthusiast, > you can't base a real product expecting > re-enumeration of the bus. right ? It depends what you mean by a real product, but yes it's not intended for use in a typical PCIe device. There are a wide variety of use cases for the EP driver framework, not necessarily hobbyist/enthusiast. The documentation here: https://www.kernel.org/doc/Documentation/PCI/endpoint/pci-endpoint.txt mentions testing or validation, co-processor accelerator, etc. as possible use cases. Regards, Alan
On August 17, 2018 11:32:03 AM GMT+03:00, Alan Douglas <adouglas@cadence.com> wrote: >Hi Ramon, > >On 17 August 2018 05:09, Ramon Fried wrote: >> On Fri, Aug 17, 2018 at 7:05 AM Ramon Fried <ramon.fried@gmail.com> >wrote: >> > On Thu, Aug 16, 2018 at 5:28 PM Alan Douglas <adouglas@cadence.com> >wrote: >> >> On 15 August 2018 17:56, Ramon Fried wrote: >> >> > On August 15, 2018 4:46:21 PM GMT+03:00, Alan Douglas ><adouglas@cadence.com> wrote: >> >> > >The patch implements MSI-X support in the cadence endpoint >driver. >> >> > > >> >> > >This patch depends on on Gustavo Pimentel's patch series adding >MSI-X >> >> > >support for EP ("Add MSI-X support on pcitest tool") >> >> > > >> >> > >It also adds fixes for MSI issues discovered during testing of >MSI-X >> >> > > - Use AXI region 0 for interrupt signalling >> >> > > - Write MSI and MSI-X with 32bit value rather than 16bit >> >> > > - Check for masking before sending MSI or MSI-X >> >> > > - Check link is up before sending IRQ >> >> > > >> >> > Hi. >> >> > AFAIK the BIOS allocates physical memory for the bars. Assuming >that the MSIx bar is only mapped after kernel boots on the >> endpoint, >> >> > could it be too late? >> >> > >> >> > Do we need to trigger re-enumeration of the PCI bus from host >side when working with this as an endpoint? >> >> It depends on how you are using it. PF0 is always enabled in the >cadence HW, so will be enumerated at boot, >> >> as long as the EP HW is out of reset and PHY is enabled. >> >> The PCIe EP hardware can be initialized so that BARs are enabled >by default, before the kernel boots on the >> >> endpoint, and so they will be found and mapped during the initial >enumeration and you don't need to >> >> re-enumerate. The MSI-X vectors can't be written to the BAR until >the EP kernel has booted and the EP driver >> >> has mapped the BAR to local EP memory though (unless you also >configure this in the PCIe EP hardware, or in >> >> EP pre-boot, but in that case you are probably not using the EP >driver framework.) >> >> >> >> The EP driver framework does, in my understanding, generally >expect re-enumeration after the >> >> EP kernel has booted and the driver has been initialized, since it >allows configuration of device ID, BAR >> >> sizes etc., and if you change any of these from the HW defaults at >boot you will need to re-enumerate. >> >> >> This basically means that this driver is for hobbyist / enthusiast, >> you can't base a real product expecting >> re-enumeration of the bus. right ? >It depends what you mean by a real product, but yes it's not intended >for use in a typical PCIe device. >There are a wide variety of use cases for the EP driver framework, not >necessarily hobbyist/enthusiast. >The documentation here: >https://www.kernel.org/doc/Documentation/PCI/endpoint/pci-endpoint.txt >mentions testing or validation, co-processor accelerator, etc. as >possible use cases. > Thanks Alan. >Regards, >Alan
The patch implements MSI-X support in the cadence endpoint driver. This patch depends on on Gustavo Pimentel's patch series adding MSI-X support for EP ("Add MSI-X support on pcitest tool") It also adds fixes for MSI issues discovered during testing of MSI-X - Use AXI region 0 for interrupt signalling - Write MSI and MSI-X with 32bit value rather than 16bit - Check for masking before sending MSI or MSI-X - Check link is up before sending IRQ Changes since v1: - Rebased on 4.18-rc1 - Update commit log to mark first 4 patches as fixes - Correct formatting issues pointed out by checkpatch --strict Alan Douglas (5): PCI: cadence: Use AXI region 0 to signal interrupts from EP PCI: cadence: Write MSI data with 32bits PCI: cadence: Check whether MSI is masked before sending it PCI: cadence: Check link is up before sending IRQ from EP PCI: cadence: Add MSI-X capability to EP driver drivers/pci/controller/pcie-cadence-ep.c | 131 +++++++++++++++++++++++++++++-- drivers/pci/controller/pcie-cadence.h | 1 + 2 files changed, 125 insertions(+), 7 deletions(-)