| Message ID | 20250211-ep-msi-v15-0-bcacc1f2b1a9@nxp.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | PCI: EP: Add RC-to-EP doorbell with platform MSI controller | expand |
On Tue, Feb 11, 2025 at 02:21:53PM -0500, Frank Li wrote: Thomas Gleixner and Marc Zyngier: Do you have any comments about irq/msi part? Frank > ┌────────────┐ ┌───────────────────────────────────┐ ┌────────────────┐ > │ │ │ │ │ │ > │ │ │ PCI Endpoint │ │ PCI Host │ > │ │ │ │ │ │ > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ > │ │ │ │ │ │ > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ > │ Controller │ │ update doorbell register address│ │ │ > │ │ │ for BAR │ │ │ > │ │ │ │ │ 3. Write BAR<n>│ > │ │◄──┼───────────────────────────────────┼───┤ │ > │ │ │ │ │ │ > │ ├──►│ 4.Irq Handle │ │ │ > │ │ │ │ │ │ > │ │ │ │ │ │ > └────────────┘ └───────────────────────────────────┘ └────────────────┘ > > This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/ > > Original patch only target to vntb driver. But actually it is common > method. > > This patches add new API to pci-epf-core, so any EP driver can use it. > > Previous v2 discussion here. > https://lore.kernel.org/imx/20230911220920.1817033-1-Frank.Li@nxp.com/ > > Changes in v15: > - rebase to v6.14-rc1 > - fix build issue find by kernel test robot > - Link to v14: https://lore.kernel.org/r/20250207-ep-msi-v14-0-9671b136f2b8@nxp.com > > Changes in v14: > Marc Zyngier raised concerns about adding DOMAIN_BUS_DEVICE_PCI_EP_MSI. As > a result, the approach has been reverted to the v9 method. However, there > are several improvements: > > MSI now supports msi-map in addition to msi-parent. > - The struct device: id is used as the endpoint function (EPF) device > identity to map to the stream ID (sideband information). > - The EPC device tree source (DTS) utilizes msi-map to provide such > information. > - The EPF device's of_node is set to the EPC controller’s node. This > approach is commonly used for multi-function device (MFD) platform child > devices, allowing them to inherit properties from the MFD device’s DTS, > such as reset-cells and gpio-cells. This method is well-suited for the > current case, as the EPF is inherently created/binded to the EPC and > should inherit the EPC’s DTS node properties. > > Additionally: > > Since the basic IMX95 LUT support has already been merged into the > mainline, a DTS and driver increment patch is added to complete the > solution. The patch is rebased onto the latest linux-next tree and > aligned with the new pcitest framework. > > - Link to v13: https://lore.kernel.org/r/20241218-ep-msi-v13-0-646e2192dc24@nxp.com > > Changes in v13: > - Change to use DOMAIN_BUS_PCI_DEVICE_EP_MSI > - Change request id as func | vfunc << 3 > - Remove IRQ_DOMAIN_MSI_IMMUTABLE > > Thomas Gleixner: > > I hope capture all your points in review comments. If missed, let me know. > > - Link to v12: https://lore.kernel.org/r/20241211-ep-msi-v12-0-33d4532fa520@nxp.com > > Changes in v12: > - Change to use IRQ_DOMAIN_MSI_IMMUTABLE and add help function > irq_domain_msi_is_immuatble(). > - split PCI: endpoint: pci-ep-msi: Add MSI address/data pair mutable check to 3 patches > - Link to v11: https://lore.kernel.org/r/20241209-ep-msi-v11-0-7434fa8397bd@nxp.com > > Changes in v11: > - Change to use MSI_FLAG_MSG_IMMUTABLE > - Link to v10: https://lore.kernel.org/r/20241204-ep-msi-v10-0-87c378dbcd6d@nxp.com > > Changes in v10: > > Thomas Gleixner: > There are big change in pci-ep-msi.c. I am sure if go on the > corrent path. The key improvement is remove only 1 function devices's > limitation. > > I use new patch for imutable check, which relative additional > feature compared to base enablement patch. > > - Remove patch Add msi_remove_device_irq_domain() in platform_device_msi_free_irqs_all() > - Add new patch irqchip/gic-v3-its: Avoid overwriting msi_prepare callback if provided by msi_domain_info > - Remove only support 1 endpoint function limiation. > - Create one MSI domain for each endpoint function devices. > - Use "msi-map" in pci ep controler node, instead of of msi-parent. first > argument is > (func_no << 8 | vfunc_no) > > - Link to v9: https://lore.kernel.org/r/20241203-ep-msi-v9-0-a60dbc3f15dd@nxp.com > > Changes in v9 > - Add patch platform-msi: Add msi_remove_device_irq_domain() in platform_device_msi_free_irqs_all() > - Remove patch PCI: endpoint: Add pci_epc_get_fn() API for customizable filtering > - Remove API pci_epf_align_inbound_addr_lo_hi > - Move doorbell_alloc in to doorbell_enable function. > - Link to v8: https://lore.kernel.org/r/20241116-ep-msi-v8-0-6f1f68ffd1bb@nxp.com > > Changes in v8: > - update helper function name to pci_epf_align_inbound_addr() > - Link to v7: https://lore.kernel.org/r/20241114-ep-msi-v7-0-d4ac7aafbd2c@nxp.com > > Changes in v7: > - Add helper function pci_epf_align_addr(); > - Link to v6: https://lore.kernel.org/r/20241112-ep-msi-v6-0-45f9722e3c2a@nxp.com > > Changes in v6: > - change doorbell_addr to doorbell_offset > - use round_down() > - add Niklas's test by tag > - rebase to pci/endpoint > - Link to v5: https://lore.kernel.org/r/20241108-ep-msi-v5-0-a14951c0d007@nxp.com > > Changes in v5: > - Move request_irq to epf test function driver for more flexiable user case > - Add fixed size bar handler > - Some minor improvememtn to see each patches's changelog. > - Link to v4: https://lore.kernel.org/r/20241031-ep-msi-v4-0-717da2d99b28@nxp.com > > Changes in v4: > - Remove patch genirq/msi: Add cleanup guard define for msi_lock_descs()/msi_unlock_descs() > - Use new method to avoid compatible problem. > Add new command DOORBELL_ENABLE and DOORBELL_DISABLE. > pcitest -B send DOORBELL_ENABLE first, EP test function driver try to > remap one of BAR_N (except test register bar) to ITS MSI MMIO space. Old > driver don't support new command, so failure return, not side effect. > After test, DOORBELL_DISABLE command send out to recover original map, so > pcitest bar test can pass as normal. > - Other detail change see each patches's change log > - Link to v3: https://lore.kernel.org/r/20241015-ep-msi-v3-0-cedc89a16c1a@nxp.com > > Change from v2 to v3 > - Fixed manivannan's comments > - Move common part to pci-ep-msi.c and pci-ep-msi.h > - rebase to 6.12-rc1 > - use RevID to distingiush old version > > mkdir /sys/kernel/config/pci_ep/functions/pci_epf_test/func1 > echo 16 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/msi_interrupts > echo 0x080c > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/deviceid > echo 0x1957 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/vendorid > echo 1 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/revid > ^^^^^^ to enable platform msi support. > ln -s /sys/kernel/config/pci_ep/functions/pci_epf_test/func1 /sys/kernel/config/pci_ep/controllers/4c380000.pcie-ep > > - use new device ID, which identify support doorbell to avoid broken > compatility. > > Enable doorbell support only for PCI_DEVICE_ID_IMX8_DB, while other devices > keep the same behavior as before. > > EP side RC with old driver RC with new driver > PCI_DEVICE_ID_IMX8_DB no probe doorbell enabled > Other device ID doorbell disabled* doorbell disabled* > > * Behavior remains unchanged. > > Change from v1 to v2 > - Add missed patch for endpont/pci-epf-test.c > - Move alloc and free to epc driver from epf. > - Provide general help function for EPC driver to alloc platform msi irq. > - Fixed manivannan's comments. > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > --- > Frank Li (15): > platform-msi: Add msi_remove_device_irq_domain() in platform_device_msi_free_irqs_all() > irqdomain: Add IRQ_DOMAIN_FLAG_MSI_IMMUTABLE and irq_domain_is_msi_immutable() > irqchip/gic-v3-its: Set IRQ_DOMAIN_FLAG_MSI_IMMUTABLE for ITS > irqchip/gic-v3-its: Add support for device tree msi-map and msi-mask > PCI: endpoint: Set ID and of_node for function driver > PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller > PCI: endpoint: pci-ep-msi: Add MSI address/data pair mutable check > PCI: endpoint: Add pci_epf_align_inbound_addr() helper for address alignment > PCI: endpoint: pci-epf-test: Add doorbell test support > misc: pci_endpoint_test: Add doorbell test case > selftests: pci_endpoint: Add doorbell test case > pci: imx6: Add helper function imx_pcie_add_lut_by_rid() > pci: imx6: Add LUT setting for MSI/IOMMU in Endpoint mode > arm64: dts: imx95: Add msi-map for pci-ep device > arm64: dts: imx95-19x19-evk: Add PCIe1 endpoint function overlay file > > arch/arm64/boot/dts/freescale/Makefile | 3 + > .../dts/freescale/imx95-19x19-evk-pcie1-ep.dtso | 21 ++++ > arch/arm64/boot/dts/freescale/imx95.dtsi | 1 + > drivers/base/platform-msi.c | 1 + > drivers/irqchip/irq-gic-v3-its-msi-parent.c | 8 ++ > drivers/irqchip/irq-gic-v3-its.c | 2 +- > drivers/misc/pci_endpoint_test.c | 81 +++++++++++++ > drivers/pci/controller/dwc/pci-imx6.c | 25 ++-- > drivers/pci/endpoint/Makefile | 1 + > drivers/pci/endpoint/functions/pci-epf-test.c | 132 +++++++++++++++++++++ > drivers/pci/endpoint/pci-ep-msi.c | 90 ++++++++++++++ > drivers/pci/endpoint/pci-epf-core.c | 48 ++++++++ > include/linux/irqdomain.h | 7 ++ > include/linux/pci-ep-msi.h | 28 +++++ > include/linux/pci-epf.h | 21 ++++ > include/uapi/linux/pcitest.h | 1 + > .../selftests/pci_endpoint/pci_endpoint_test.c | 25 ++++ > 17 files changed, 486 insertions(+), 9 deletions(-) > --- > base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b > change-id: 20241010-ep-msi-8b4cab33b1be > > Best regards, > --- > Frank Li <Frank.Li@nxp.com> >
On Thu, Feb 20 2025 at 15:01, Frank Li wrote: > On Tue, Feb 11, 2025 at 02:21:53PM -0500, Frank Li wrote: > > Thomas Gleixner and Marc Zyngier: > > Do you have any comments about irq/msi part? I'm not having objections, but this needs to be acked by Marc as he had pretty strong opinions on the overall approach. Thanks, tglx
┌────────────┐ ┌───────────────────────────────────┐ ┌────────────────┐ │ │ │ │ │ │ │ │ │ PCI Endpoint │ │ PCI Host │ │ │ │ │ │ │ │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ │ │ │ │ │ │ │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ │ Controller │ │ update doorbell register address│ │ │ │ │ │ for BAR │ │ │ │ │ │ │ │ 3. Write BAR<n>│ │ │◄──┼───────────────────────────────────┼───┤ │ │ │ │ │ │ │ │ ├──►│ 4.Irq Handle │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────┘ └───────────────────────────────────┘ └────────────────┘ This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/ Original patch only target to vntb driver. But actually it is common method. This patches add new API to pci-epf-core, so any EP driver can use it. Previous v2 discussion here. https://lore.kernel.org/imx/20230911220920.1817033-1-Frank.Li@nxp.com/ Changes in v15: - rebase to v6.14-rc1 - fix build issue find by kernel test robot - Link to v14: https://lore.kernel.org/r/20250207-ep-msi-v14-0-9671b136f2b8@nxp.com Changes in v14: Marc Zyngier raised concerns about adding DOMAIN_BUS_DEVICE_PCI_EP_MSI. As a result, the approach has been reverted to the v9 method. However, there are several improvements: MSI now supports msi-map in addition to msi-parent. - The struct device: id is used as the endpoint function (EPF) device identity to map to the stream ID (sideband information). - The EPC device tree source (DTS) utilizes msi-map to provide such information. - The EPF device's of_node is set to the EPC controller’s node. This approach is commonly used for multi-function device (MFD) platform child devices, allowing them to inherit properties from the MFD device’s DTS, such as reset-cells and gpio-cells. This method is well-suited for the current case, as the EPF is inherently created/binded to the EPC and should inherit the EPC’s DTS node properties. Additionally: Since the basic IMX95 LUT support has already been merged into the mainline, a DTS and driver increment patch is added to complete the solution. The patch is rebased onto the latest linux-next tree and aligned with the new pcitest framework. - Link to v13: https://lore.kernel.org/r/20241218-ep-msi-v13-0-646e2192dc24@nxp.com Changes in v13: - Change to use DOMAIN_BUS_PCI_DEVICE_EP_MSI - Change request id as func | vfunc << 3 - Remove IRQ_DOMAIN_MSI_IMMUTABLE Thomas Gleixner: I hope capture all your points in review comments. If missed, let me know. - Link to v12: https://lore.kernel.org/r/20241211-ep-msi-v12-0-33d4532fa520@nxp.com Changes in v12: - Change to use IRQ_DOMAIN_MSI_IMMUTABLE and add help function irq_domain_msi_is_immuatble(). - split PCI: endpoint: pci-ep-msi: Add MSI address/data pair mutable check to 3 patches - Link to v11: https://lore.kernel.org/r/20241209-ep-msi-v11-0-7434fa8397bd@nxp.com Changes in v11: - Change to use MSI_FLAG_MSG_IMMUTABLE - Link to v10: https://lore.kernel.org/r/20241204-ep-msi-v10-0-87c378dbcd6d@nxp.com Changes in v10: Thomas Gleixner: There are big change in pci-ep-msi.c. I am sure if go on the corrent path. The key improvement is remove only 1 function devices's limitation. I use new patch for imutable check, which relative additional feature compared to base enablement patch. - Remove patch Add msi_remove_device_irq_domain() in platform_device_msi_free_irqs_all() - Add new patch irqchip/gic-v3-its: Avoid overwriting msi_prepare callback if provided by msi_domain_info - Remove only support 1 endpoint function limiation. - Create one MSI domain for each endpoint function devices. - Use "msi-map" in pci ep controler node, instead of of msi-parent. first argument is (func_no << 8 | vfunc_no) - Link to v9: https://lore.kernel.org/r/20241203-ep-msi-v9-0-a60dbc3f15dd@nxp.com Changes in v9 - Add patch platform-msi: Add msi_remove_device_irq_domain() in platform_device_msi_free_irqs_all() - Remove patch PCI: endpoint: Add pci_epc_get_fn() API for customizable filtering - Remove API pci_epf_align_inbound_addr_lo_hi - Move doorbell_alloc in to doorbell_enable function. - Link to v8: https://lore.kernel.org/r/20241116-ep-msi-v8-0-6f1f68ffd1bb@nxp.com Changes in v8: - update helper function name to pci_epf_align_inbound_addr() - Link to v7: https://lore.kernel.org/r/20241114-ep-msi-v7-0-d4ac7aafbd2c@nxp.com Changes in v7: - Add helper function pci_epf_align_addr(); - Link to v6: https://lore.kernel.org/r/20241112-ep-msi-v6-0-45f9722e3c2a@nxp.com Changes in v6: - change doorbell_addr to doorbell_offset - use round_down() - add Niklas's test by tag - rebase to pci/endpoint - Link to v5: https://lore.kernel.org/r/20241108-ep-msi-v5-0-a14951c0d007@nxp.com Changes in v5: - Move request_irq to epf test function driver for more flexiable user case - Add fixed size bar handler - Some minor improvememtn to see each patches's changelog. - Link to v4: https://lore.kernel.org/r/20241031-ep-msi-v4-0-717da2d99b28@nxp.com Changes in v4: - Remove patch genirq/msi: Add cleanup guard define for msi_lock_descs()/msi_unlock_descs() - Use new method to avoid compatible problem. Add new command DOORBELL_ENABLE and DOORBELL_DISABLE. pcitest -B send DOORBELL_ENABLE first, EP test function driver try to remap one of BAR_N (except test register bar) to ITS MSI MMIO space. Old driver don't support new command, so failure return, not side effect. After test, DOORBELL_DISABLE command send out to recover original map, so pcitest bar test can pass as normal. - Other detail change see each patches's change log - Link to v3: https://lore.kernel.org/r/20241015-ep-msi-v3-0-cedc89a16c1a@nxp.com Change from v2 to v3 - Fixed manivannan's comments - Move common part to pci-ep-msi.c and pci-ep-msi.h - rebase to 6.12-rc1 - use RevID to distingiush old version mkdir /sys/kernel/config/pci_ep/functions/pci_epf_test/func1 echo 16 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/msi_interrupts echo 0x080c > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/deviceid echo 0x1957 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/vendorid echo 1 > /sys/kernel/config/pci_ep/functions/pci_epf_test/func1/revid ^^^^^^ to enable platform msi support. ln -s /sys/kernel/config/pci_ep/functions/pci_epf_test/func1 /sys/kernel/config/pci_ep/controllers/4c380000.pcie-ep - use new device ID, which identify support doorbell to avoid broken compatility. Enable doorbell support only for PCI_DEVICE_ID_IMX8_DB, while other devices keep the same behavior as before. EP side RC with old driver RC with new driver PCI_DEVICE_ID_IMX8_DB no probe doorbell enabled Other device ID doorbell disabled* doorbell disabled* * Behavior remains unchanged. Change from v1 to v2 - Add missed patch for endpont/pci-epf-test.c - Move alloc and free to epc driver from epf. - Provide general help function for EPC driver to alloc platform msi irq. - Fixed manivannan's comments. Signed-off-by: Frank Li <Frank.Li@nxp.com> --- Frank Li (15): platform-msi: Add msi_remove_device_irq_domain() in platform_device_msi_free_irqs_all() irqdomain: Add IRQ_DOMAIN_FLAG_MSI_IMMUTABLE and irq_domain_is_msi_immutable() irqchip/gic-v3-its: Set IRQ_DOMAIN_FLAG_MSI_IMMUTABLE for ITS irqchip/gic-v3-its: Add support for device tree msi-map and msi-mask PCI: endpoint: Set ID and of_node for function driver PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller PCI: endpoint: pci-ep-msi: Add MSI address/data pair mutable check PCI: endpoint: Add pci_epf_align_inbound_addr() helper for address alignment PCI: endpoint: pci-epf-test: Add doorbell test support misc: pci_endpoint_test: Add doorbell test case selftests: pci_endpoint: Add doorbell test case pci: imx6: Add helper function imx_pcie_add_lut_by_rid() pci: imx6: Add LUT setting for MSI/IOMMU in Endpoint mode arm64: dts: imx95: Add msi-map for pci-ep device arm64: dts: imx95-19x19-evk: Add PCIe1 endpoint function overlay file arch/arm64/boot/dts/freescale/Makefile | 3 + .../dts/freescale/imx95-19x19-evk-pcie1-ep.dtso | 21 ++++ arch/arm64/boot/dts/freescale/imx95.dtsi | 1 + drivers/base/platform-msi.c | 1 + drivers/irqchip/irq-gic-v3-its-msi-parent.c | 8 ++ drivers/irqchip/irq-gic-v3-its.c | 2 +- drivers/misc/pci_endpoint_test.c | 81 +++++++++++++ drivers/pci/controller/dwc/pci-imx6.c | 25 ++-- drivers/pci/endpoint/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-test.c | 132 +++++++++++++++++++++ drivers/pci/endpoint/pci-ep-msi.c | 90 ++++++++++++++ drivers/pci/endpoint/pci-epf-core.c | 48 ++++++++ include/linux/irqdomain.h | 7 ++ include/linux/pci-ep-msi.h | 28 +++++ include/linux/pci-epf.h | 21 ++++ include/uapi/linux/pcitest.h | 1 + .../selftests/pci_endpoint/pci_endpoint_test.c | 25 ++++ 17 files changed, 486 insertions(+), 9 deletions(-) --- base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b change-id: 20241010-ep-msi-8b4cab33b1be Best regards, --- Frank Li <Frank.Li@nxp.com>