| Message ID | 20241017165225.21206-1-alejandro.lucero-palau@amd.com |
|---|---|
| Headers | show |
| Series | cxl: add Type2 device support | expand |
Hi, 10/17/24 18:51, alejandro.lucero-palau@amd.com wrote: > 2) The driver for using the added functionality is not a test driver but a real > one: the SFC ethernet network driver. It uses the CXL region mapped for PIO > buffers instead of regions inside PCIe BARs. I'm sorry for the late feedback, but which is the merge plan here? The series spawns across 2 different subsystems and could cause conflicts. Could the network device change be separated and send (to netdev) after the clx ones land into Linus' tree? Thanks, Paolo
On 10/23/24 09:46, Paolo Abeni wrote: > Hi, > > 10/17/24 18:51, alejandro.lucero-palau@amd.com wrote: >> 2) The driver for using the added functionality is not a test driver but a real >> one: the SFC ethernet network driver. It uses the CXL region mapped for PIO >> buffers instead of regions inside PCIe BARs. > I'm sorry for the late feedback, but which is the merge plan here? > > The series spawns across 2 different subsystems and could cause conflicts. > > Could the network device change be separated and send (to netdev) after > the clx ones land into Linus' tree? Hi Paolo, With v4 all sfc changes are different patches than those modifying CXL core, so I guess this is good for what you suggest. Not sure the implications for merging only some patches into the CXL tree. Thanks, Alejandro > Thanks, > > Paolo >
Hi all, Facing Paolo's question again trying to involve CXL and (more) netdev maintainers. Next v6 could have two different patchsets, one for cxl, one for netdev. The current patchset has already cleanly isolated sfc netdev patches, so it is trivial. The main question is if CXL maintainers will be happy with this change as the sfc is the client justifying the CXL core changes. Also, the split could be delayed until all the patches get the Reviewed-by tag what is now only ~75% of them (sfc related patches without the public approval yet but internally obtained). Thanks, Alejandro On 10/23/24 10:38, Alejandro Lucero Palau wrote: > > On 10/23/24 09:46, Paolo Abeni wrote: >> I'm sorry for the late feedback, but which is the merge plan here? >> >> The series spawns across 2 different subsystems and could cause >> conflicts. >> >> Could the network device change be separated and send (to netdev) after >> the clx ones land into Linus' tree? > > > Hi Paolo, > > > With v4 all sfc changes are different patches than those modifying CXL > core, so I guess this is good for what you suggest. > > > Not sure the implications for merging only some patches into the CXL > tree. > > > Thanks, > > Alejandro > > >> Thanks, >> >> Paolo >>
On 11/20/24 9:50 AM, Alejandro Lucero Palau wrote: > Hi all, > > > Facing Paolo's question again trying to involve CXL and (more) netdev maintainers. > > > Next v6 could have two different patchsets, one for cxl, one for netdev. The current patchset has already cleanly isolated sfc netdev patches, so it is trivial. > > The main question is if CXL maintainers will be happy with this change as the sfc is the client justifying the CXL core changes. Also, the split could be delayed until all the patches get the Reviewed-by tag what is now only ~75% of them (sfc related patches without the public approval yet but internally obtained). Given that the series is dominantly CXL patches, my suggestion would be get the acks from netdev side and CXL can take the whole series without doing any splitting. That's been typically how it has been done with cross subsystem changes. i.e. ACPI+CXL etc. DJ > > Thanks, > > Alejandro > > > On 10/23/24 10:38, Alejandro Lucero Palau wrote: >> >> On 10/23/24 09:46, Paolo Abeni wrote: >>> I'm sorry for the late feedback, but which is the merge plan here? >>> >>> The series spawns across 2 different subsystems and could cause conflicts. >>> >>> Could the network device change be separated and send (to netdev) after >>> the clx ones land into Linus' tree? >> >> >> Hi Paolo, >> >> >> With v4 all sfc changes are different patches than those modifying CXL core, so I guess this is good for what you suggest. >> >> >> Not sure the implications for merging only some patches into the CXL tree. >> >> >> Thanks, >> >> Alejandro >> >> >>> Thanks, >>> >>> Paolo >>>
From: Alejandro Lucero <alejandro.lucero-palau@amd.com> v4 changes: - Use bitmap for capabilities new field (Jonathan Cameron) - Use cxl_mem attributes for sysfs based on device type (Dave Jian) - Add conditional cxl sfc compilation relying on kernel CXL config (kernel test robot) - Add sfc changes in different patches for facilitating backport (Jonathan Cameron) - Remove patch for dealing with cxl modules dependencies and using sfc kconfig plus MODULE_SOFTDEP instead. v3 changes: - cxl_dev_state not defined as opaque but only manipulated by accel drivers through accessors. - accessors names not identified as only for accel drivers. - move pci code from pci driver (drivers/cxl/pci.c) to generic pci code (drivers/cxl/core/pci.c). - capabilities field from u8 to u32 and initialised by CXL regs discovering code. - add capabilities check and removing current check by CXL regs discovering code. - Not fail if CXL Device Registers not found. Not mandatory for Type2. - add timeout in acquire_endpoint for solving a race with the endpoint port creation. - handle EPROBE_DEFER by sfc driver. - Limiting interleave ways to 1 for accel driver HPA/DPA requests. - factoring out interleave ways and granularity helpers from type2 region creation patch. - restricting region_creation for type2 to one endpoint decoder. - add accessor for release_resource. - handle errors and errors messages properly. v2 changes: I have removed the introduction about the concerns with BIOS/UEFI after the discussion leading to confirm the need of the functionality implemented, at least is some scenarios. There are two main changes from the RFC: 1) Following concerns about drivers using CXL core without restrictions, the CXL struct to work with is opaque to those drivers, therefore functions are implemented for modifying or reading those structs indirectly. 2) The driver for using the added functionality is not a test driver but a real one: the SFC ethernet network driver. It uses the CXL region mapped for PIO buffers instead of regions inside PCIe BARs. RFC: Current CXL kernel code is focused on supporting Type3 CXL devices, aka memory expanders. Type2 CXL devices, aka device accelerators, share some functionalities but require some special handling. First of all, Type2 are by definition specific to drivers doing something and not just a memory expander, so it is expected to work with the CXL specifics. This implies the CXL setup needs to be done by such a driver instead of by a generic CXL PCI driver as for memory expanders. Most of such setup needs to use current CXL core code and therefore needs to be accessible to those vendor drivers. This is accomplished exporting opaque CXL structs and adding and exporting functions for working with those structs indirectly. Some of the patches are based on a patchset sent by Dan Williams [1] which was just partially integrated, most related to making things ready for Type2 but none related to specific Type2 support. Those patches based on Dan´s work have Dan´s signing as co-developer, and a link to the original patch. A final note about CXL.cache is needed. This patchset does not cover it at all, although the emulated Type2 device advertises it. From the kernel point of view supporting CXL.cache will imply to be sure the CXL path supports what the Type2 device needs. A device accelerator will likely be connected to a Root Switch, but other configurations can not be discarded. Therefore the kernel will need to check not just HPA, DPA, interleave and granularity, but also the available CXL.cache support and resources in each switch in the CXL path to the Type2 device. I expect to contribute to this support in the following months, and it would be good to discuss about it when possible. [1] https://lore.kernel.org/linux-cxl/98b1f61a-e6c2-71d4-c368-50d958501b0c@intel.com/T/ Alejandro Lucero (26): cxl: add type2 device basic support sfc: add cxl support using new CXL API cxl: add capabilities field to cxl_dev_state and cxl_port cxl/pci: add check for validating capabilities cxl: move pci generic code cxl: add function for type2 cxl regs setup sfc: use cxl api for regs setup and checking cxl: add functions for resource request/release by a driver sfc: request cxl ram resource cxl: harden resource_contains checks to handle zero size resources cxl: add function for setting media ready by a driver sfc: set cxl media ready cxl: prepare memdev creation for type2 sfc: create type2 cxl memdev cxl: define a driver interface for HPA free space enumeration sfc: obtain root decoder with enough HPA free space cxl: define a driver interface for DPA allocation sfc: get endpoint decoder cxl: make region type based on endpoint type cxl/region: factor out interleave ways setup cxl/region: factor out interleave granularity setup cxl: allow region creation by type2 drivers sfc: create cxl region cxl: preclude device memory to be used for dax cxl: add function for obtaining params from a region sfc: support pio mapping based on cxl drivers/cxl/core/hdm.c | 160 ++++++++-- drivers/cxl/core/memdev.c | 124 +++++++- drivers/cxl/core/pci.c | 124 ++++++++ drivers/cxl/core/port.c | 11 +- drivers/cxl/core/region.c | 409 ++++++++++++++++++++++---- drivers/cxl/core/regs.c | 30 +- drivers/cxl/cxl.h | 14 +- drivers/cxl/cxlmem.h | 4 + drivers/cxl/cxlpci.h | 19 +- drivers/cxl/mem.c | 25 +- drivers/cxl/pci.c | 95 ++---- drivers/net/ethernet/sfc/Kconfig | 1 + drivers/net/ethernet/sfc/Makefile | 2 +- drivers/net/ethernet/sfc/ef10.c | 34 ++- drivers/net/ethernet/sfc/efx.c | 16 + drivers/net/ethernet/sfc/efx_cxl.c | 186 ++++++++++++ drivers/net/ethernet/sfc/efx_cxl.h | 29 ++ drivers/net/ethernet/sfc/mcdi_pcol.h | 12 + drivers/net/ethernet/sfc/net_driver.h | 8 + drivers/net/ethernet/sfc/nic.h | 2 + include/linux/cxl/cxl.h | 81 +++++ include/linux/cxl/pci.h | 23 ++ 22 files changed, 1210 insertions(+), 199 deletions(-) create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h create mode 100644 include/linux/cxl/cxl.h create mode 100644 include/linux/cxl/pci.h