mbox series

[iwl-next,rdma,v3,00/24] Add RDMA support for Intel IPU E2000 (GEN3)

Message ID 20250207194931.1569-1-tatyana.e.nikolova@intel.com (mailing list archive)
Headers show
Series Add RDMA support for Intel IPU E2000 (GEN3) | expand

Message

Tatyana Nikolova Feb. 7, 2025, 7:49 p.m. UTC
This patch series is based on 6.14-rc1 and includes both netdev and RDMA
patches for ease of review. It can also be viewed here [1]. A shared pull
request will be sent for patches 1-7 following review.

The patch series introduces RDMA RoCEv2 support for the Intel Infrastructure
Processing Unit (IPU) E2000 line of products, referred to as GEN3 in the
irdma provider.  It supports both physical and virtual functions.
The irdma driver communicates with the device Control Plane (CP) to
discover capabilities and perform privileged operations through an
RDMA-specific messaging interface built atop the Infrastructure
Data-Plane Function (IDPF) mailbox and virtchannel protocol [2].

To support RDMA for E2000 product, the idpf driver requires the use of
the Inter-Driver Communication (IDC) interface which is currently already
in use between ice and irdma. With a second consumer, the IDC is
generalized to support multiple consumers and ice, idpf and irdma
are adapted to the IDC definitions.

The IPU model can host one or more logical network endpoints called vPorts
per PCI function that are flexibly associated with a physical port or an
internal communication port. irdma exposes a verbs device per vPort.

Other key highlights of this series as it pertains to GEN3 device include:

MMIO learning, RDMA capability negotiation and RDMA vectors
discovery between idpf and CP
PCI core device level RDMA resource initialization via
a GEN3 core auxiliary driver
Shared Receive Queue (SRQ) Support
Atomic Operations Support (Compare and Swap and Fetch and Add)
Completion Queue Element (CQE) Error and Flush Handling
Push Page Support

Changelog:

V3 series irdma changes:
* Move the call to get RDMA features just after CQP is created,
otherwise the feature flags are not defined before used.
* Move the check for supported atomic operations after reading
the RDMA feature info to correctly enable atomics.
* Round up to power of two the resource size for Read Responses and
Transmit Queue elements.
* Do not use the Work Queue element index passed in the Asynchronous Event
info to get SRQ context, because it is incorrect.
* Fix detection of Completion Queue (CQ) empty when 64-byte CQ elements
are enabled.
* Minor improvements and cleanup.

V3 series idpf changes:
* Reduce required minimum RDMA vectors to 2.

V2 RFC series includes only idpf changes:

* RDMA vector number adjustment
* Fix unplugging vport auxiliary device twice
* General cleanup and minor improvements

V2 RFC series is at https://lwn.net/Articles/987141/.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/linux.git/log/?h=idpf-rdma
[2] https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c

Christopher Bednarz (1):
  RDMA/irdma: Discover and set up GEN3 hardware register layout

Dave Ertman (1):
  iidc/ice/irdma: Update IDC to support multiple consumers

Faisal Latif (2):
  RDMA/irdma: Add SRQ support
  RDMA/irdma: Add Atomic Operations support

Jay Bhat (1):
  RDMA/irdma: Add Push Page Support for GEN3

Joshua Hay (6):
  idpf: use reserved RDMA vectors from control plane
  idpf: implement core RDMA auxiliary dev create, init, and destroy
  idpf: implement RDMA vport auxiliary dev create, init, and destroy
  idpf: implement remaining IDC RDMA core callbacks and handlers
  idpf: implement IDC vport aux driver MTU change handler
  idpf: implement get LAN mmio memory regions

Krzysztof Czurylo (2):
  RDMA/irdma: Add GEN3 CQP support with deferred completions
  RDMA/irdma: Add GEN3 HW statistics support

Mustafa Ismail (3):
  RDMA/irdma: Refactor GEN2 auxiliary driver
  RDMA/irdma: Add GEN3 core driver support
  RDMA/irdma: Introduce GEN3 vPort driver support

Shiraz Saleem (7):
  RDMA/irdma: Add GEN3 support for AEQ and CEQ
  RDMA/irdma: Add GEN3 virtual QP1 support
  RDMA/irdma: Extend QP context programming for GEN3
  RDMA/irdma: Support 64-byte CQEs and GEN3 CQE opcode decoding
  RDMA/irdma: Restrict Memory Window and CQE Timestamping to GEN3
  RDMA/irdma: Extend CQE Error and Flush Handling for GEN3 Devices
  RDMA/irdma: Update Kconfig

Vinoth Kumar Chandra Mohan (1):
  RDMA/irdma: Add support for V2 HMC resource management scheme

 drivers/infiniband/hw/irdma/Kconfig           |    7 +-
 drivers/infiniband/hw/irdma/Makefile          |    4 +
 drivers/infiniband/hw/irdma/ctrl.c            | 1469 +++++++++++++++--
 drivers/infiniband/hw/irdma/defs.h            |  266 +--
 drivers/infiniband/hw/irdma/hmc.c             |   18 +-
 drivers/infiniband/hw/irdma/hmc.h             |   19 +-
 drivers/infiniband/hw/irdma/hw.c              |  357 ++--
 drivers/infiniband/hw/irdma/i40iw_hw.c        |    2 +
 drivers/infiniband/hw/irdma/i40iw_hw.h        |    2 +
 drivers/infiniband/hw/irdma/i40iw_if.c        |    3 +
 drivers/infiniband/hw/irdma/icrdma_hw.c       |    3 +
 drivers/infiniband/hw/irdma/icrdma_hw.h       |    5 +-
 drivers/infiniband/hw/irdma/icrdma_if.c       |  267 +++
 drivers/infiniband/hw/irdma/ig3rdma_hw.c      |  170 ++
 drivers/infiniband/hw/irdma/ig3rdma_hw.h      |   29 +
 drivers/infiniband/hw/irdma/ig3rdma_if.c      |  279 ++++
 drivers/infiniband/hw/irdma/irdma.h           |   23 +-
 drivers/infiniband/hw/irdma/main.c            |  323 +---
 drivers/infiniband/hw/irdma/main.h            |   42 +-
 drivers/infiniband/hw/irdma/osdep.h           |    4 +-
 drivers/infiniband/hw/irdma/pble.c            |   20 +-
 drivers/infiniband/hw/irdma/protos.h          |    1 +
 drivers/infiniband/hw/irdma/puda.h            |    4 +-
 drivers/infiniband/hw/irdma/type.h            |  224 ++-
 drivers/infiniband/hw/irdma/uda_d.h           |    5 +-
 drivers/infiniband/hw/irdma/uk.c              |  303 +++-
 drivers/infiniband/hw/irdma/user.h            |  268 ++-
 drivers/infiniband/hw/irdma/utils.c           |  133 +-
 drivers/infiniband/hw/irdma/verbs.c           |  873 ++++++++--
 drivers/infiniband/hw/irdma/verbs.h           |   47 +
 drivers/infiniband/hw/irdma/virtchnl.c        |  656 ++++++++
 drivers/infiniband/hw/irdma/virtchnl.h        |  189 +++
 .../net/ethernet/intel/ice/devlink/devlink.c  |   40 +-
 drivers/net/ethernet/intel/ice/ice.h          |    6 +-
 drivers/net/ethernet/intel/ice/ice_dcb_lib.c  |   46 +-
 drivers/net/ethernet/intel/ice/ice_dcb_lib.h  |    4 +
 drivers/net/ethernet/intel/ice/ice_ethtool.c  |    8 +-
 drivers/net/ethernet/intel/ice/ice_idc.c      |  255 ++-
 drivers/net/ethernet/intel/ice/ice_idc_int.h  |    5 +-
 drivers/net/ethernet/intel/ice/ice_main.c     |   18 +-
 drivers/net/ethernet/intel/idpf/Makefile      |    1 +
 drivers/net/ethernet/intel/idpf/idpf.h        |  114 +-
 .../net/ethernet/intel/idpf/idpf_controlq.c   |   14 +-
 .../net/ethernet/intel/idpf/idpf_controlq.h   |   15 +-
 drivers/net/ethernet/intel/idpf/idpf_dev.c    |   48 +-
 drivers/net/ethernet/intel/idpf/idpf_idc.c    |  490 ++++++
 drivers/net/ethernet/intel/idpf/idpf_lib.c    |   93 +-
 drivers/net/ethernet/intel/idpf/idpf_main.c   |   32 +-
 drivers/net/ethernet/intel/idpf/idpf_mem.h    |    8 +-
 drivers/net/ethernet/intel/idpf/idpf_txrx.h   |    1 +
 drivers/net/ethernet/intel/idpf/idpf_vf_dev.c |   44 +-
 .../net/ethernet/intel/idpf/idpf_virtchnl.c   |  189 ++-
 .../net/ethernet/intel/idpf/idpf_virtchnl.h   |    3 +
 drivers/net/ethernet/intel/idpf/virtchnl2.h   |   52 +-
 include/linux/net/intel/idc_rdma.h            |  138 ++
 include/linux/net/intel/iidc.h                |  107 --
 include/linux/net/intel/iidc_rdma.h           |   67 +
 include/uapi/rdma/irdma-abi.h                 |   17 +-
 58 files changed, 6693 insertions(+), 1137 deletions(-)
 create mode 100644 drivers/infiniband/hw/irdma/icrdma_if.c
 create mode 100644 drivers/infiniband/hw/irdma/ig3rdma_hw.c
 create mode 100644 drivers/infiniband/hw/irdma/ig3rdma_hw.h
 create mode 100644 drivers/infiniband/hw/irdma/ig3rdma_if.c
 create mode 100644 drivers/infiniband/hw/irdma/virtchnl.c
 create mode 100644 drivers/infiniband/hw/irdma/virtchnl.h
 create mode 100644 drivers/net/ethernet/intel/idpf/idpf_idc.c
 create mode 100644 include/linux/net/intel/idc_rdma.h
 delete mode 100644 include/linux/net/intel/iidc.h
 create mode 100644 include/linux/net/intel/iidc_rdma.h

Comments

Przemek Kitszel Feb. 10, 2025, 10:41 a.m. UTC | #1
On 2/7/25 20:49, Tatyana Nikolova wrote:
> This patch series is based on 6.14-rc1 and includes both netdev and RDMA
> patches for ease of review. It can also be viewed here [1]. A shared pull
> request will be sent for patches 1-7 following review.
> 

[...]
TLDR of my mail: could be take 1st patch prior to the rest?

> V2 RFC series is at https://lwn.net/Articles/987141/.

code there was mostly the same, and noone commented, I bet due
to the sheer size of the series

>   58 files changed, 6693 insertions(+), 1137 deletions(-)


While it is very good to have a common cover letter and all the changes
available for inspection/cross reference, it is just too much for doing
a practical review.

I think it would be much easier to follow and split into multiple series
if we will apply the first patch, is that feasible now? (It was a no-no
back when there were no idpf at all, no it is not a theoretical case
anymore, and the patch makes code better anyway)?

The first patch:
Dave Ertman (1):
   iidc/ice/irdma: Update IDC to support multiple consumers

@Tatyana
The rest of the patches needs to go via our internal e1000 ML,
and get the Reviewed-by tag prior to being ready for wider review
(now the community would have to put substantial effor to get rid
of style distractions, etc, and, as we have seen, there was not
much eagerness to do that for your RFC v2).
Leon Romanovsky Feb. 10, 2025, 11:09 a.m. UTC | #2
On Mon, Feb 10, 2025 at 11:41:31AM +0100, Przemek Kitszel wrote:
> On 2/7/25 20:49, Tatyana Nikolova wrote:
> > This patch series is based on 6.14-rc1 and includes both netdev and RDMA
> > patches for ease of review. It can also be viewed here [1]. A shared pull
> > request will be sent for patches 1-7 following review.
> > 
> 
> [...]
> TLDR of my mail: could be take 1st patch prior to the rest?
> 
> > V2 RFC series is at https://lwn.net/Articles/987141/.
> 
> code there was mostly the same, and noone commented, I bet due
> to the sheer size of the series

It was very optimistic to expect for a review during holiday season
and merge window, especially series of 25 patches which are marked
as RFC.

Thanks
Przemek Kitszel Feb. 13, 2025, 4:12 p.m. UTC | #3
On 2/10/25 12:09, Leon Romanovsky wrote:
> On Mon, Feb 10, 2025 at 11:41:31AM +0100, Przemek Kitszel wrote:
>> On 2/7/25 20:49, Tatyana Nikolova wrote:
>>> This patch series is based on 6.14-rc1 and includes both netdev and RDMA
>>> patches for ease of review. It can also be viewed here [1]. A shared pull
>>> request will be sent for patches 1-7 following review.
>>>
>>
>> [...]
>> TLDR of my mail: could be take 1st patch prior to the rest?
>>
>>> V2 RFC series is at https://lwn.net/Articles/987141/.
>>
>> code there was mostly the same, and noone commented, I bet due
>> to the sheer size of the series
> 
> It was very optimistic to expect for a review during holiday season
> and merge window, especially series of 25 patches which are marked
> as RFC.

that's true

so, given most of the patches will go via your tree, how do you want
to split us the existing ones into series?

a) 1st, idpf, rdma
b) 1st, rest
c) all together

In any case I will do a review too of course

> 
> Thanks
Leon Romanovsky Feb. 16, 2025, 11:18 a.m. UTC | #4
On Thu, Feb 13, 2025 at 05:12:46PM +0100, Przemek Kitszel wrote:
> On 2/10/25 12:09, Leon Romanovsky wrote:
> > On Mon, Feb 10, 2025 at 11:41:31AM +0100, Przemek Kitszel wrote:
> > > On 2/7/25 20:49, Tatyana Nikolova wrote:
> > > > This patch series is based on 6.14-rc1 and includes both netdev and RDMA
> > > > patches for ease of review. It can also be viewed here [1]. A shared pull
> > > > request will be sent for patches 1-7 following review.
> > > > 
> > > 
> > > [...]
> > > TLDR of my mail: could be take 1st patch prior to the rest?
> > > 
> > > > V2 RFC series is at https://lwn.net/Articles/987141/.
> > > 
> > > code there was mostly the same, and noone commented, I bet due
> > > to the sheer size of the series
> > 
> > It was very optimistic to expect for a review during holiday season
> > and merge window, especially series of 25 patches which are marked
> > as RFC.
> 
> that's true
> 
> so, given most of the patches will go via your tree, how do you want
> to split us the existing ones into series?
> 
> a) 1st, idpf, rdma
> b) 1st, rest
> c) all together
> 
> In any case I will do a review too of course

There is a need to get Acks for the netdev part and because that part is
going to be before RDMA patches, the fist option is proffered one.

If you plan to merge everything in this cycle, netdev and RDMA parts need
to be merged into some shared branch based on clean -rcX.

I can do it for you as well, but first need to get Acks for netdev part.

Thanks

> 
> > 
> > Thanks
> 
>