[v7,00/23] nvme-tcp receive offloads

Message ID	20221025135958.6242-1-aaptel@nvidia.com (mailing list archive)
Headers	show Return-Path: <netdev-owner@kernel.org> From: Aurelien Aptel <aaptel@nvidia.com> To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, edumazet@google.com, pabeni@redhat.com, saeedm@nvidia.com, tariqt@nvidia.com, leon@kernel.org, linux-nvme@lists.infradead.org, sagi@grimberg.me, hch@lst.de, kbusch@kernel.org, axboe@fb.com, chaitanyak@nvidia.com Cc: smalin@nvidia.com, aaptel@nvidia.com, ogerlitz@nvidia.com, yorayz@nvidia.com, borisp@nvidia.com, aurelien.aptel@gmail.com, malin1024@gmail.com Subject: [PATCH v7 00/23] nvme-tcp receive offloads Date: Tue, 25 Oct 2022 16:59:35 +0300 Message-Id: <20221025135958.6242-1-aaptel@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Precedence: bulk
Series	nvme-tcp receive offloads \| expand [v7,00/23] nvme-tcp receive offloads [v7,01/23] net: Introduce direct data placement tcp offload [v7,02/23] iov_iter: DDP copy to iter/pages [v7,03/23] net/tls: export get_netdev_for_sock [v7,04/23] Revert "nvme-tcp: remove the unused queue_size member in nvme_tcp_queue" [v7,05/23] nvme-tcp: Add DDP offload control path [v7,06/23] nvme-tcp: Add DDP data-path [v7,07/23] nvme-tcp: RX DDGST offload [v7,08/23] nvme-tcp: Deal with netdevice DOWN events [v7,09/23] nvme-tcp: Add modparam to control the ULP offload enablement [v7,10/23] Documentation: add ULP DDP offload documentation [v7,11/23] net/mlx5e: Rename from tls to transport static params [v7,12/23] net/mlx5e: Refactor ico sq polling to get budget [v7,13/23] net/mlx5e: Have mdev pointer directly on the icosq structure [v7,14/23] net/mlx5e: Refactor doorbell function to allow avoiding a completion [v7,15/23] net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations [v7,16/23] net/mlx5e: NVMEoTCP, offload initialization [v7,17/23] net/mlx5e: TCP flow steering for nvme-tcp acceleration [v7,18/23] net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration [v7,19/23] net/mlx5e: NVMEoTCP, queue init/teardown [v7,20/23] net/mlx5e: NVMEoTCP, ddp setup and resync [v7,21/23] net/mlx5e: NVMEoTCP, async ddp invalidation [v7,22/23] net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload [v7,23/23] net/mlx5e: NVMEoTCP, statistics

Aurelien Aptel Oct. 25, 2022, 1:59 p.m. UTC

Hi,

The nvme-tcp receive offloads series v7 was sent to both net-next and
nvme.  It is the continuation of v5 which was sent on July 2021
https://lore.kernel.org/netdev/20210722110325.371-1-borisp@nvidia.com/ .
V7 is now working on a real HW.

The feature will also be presented in netdev this week
https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-Implementation-and-Performance-Gains

Currently the series is aligned to net-next, please update us if you will prefer otherwise.

Thanks,
Shai, Aurelien

==== COVER LETTER ====
From: Aurelien Aptel <aaptel@nvidia.com>
From: Shai Malin <smalin@nvidia.com>
From: Ben Ben-Ishay <benishay@nvidia.com>
From: Boris Pismenny <borisp@nvidia.com>
From: Or Gerlitz <ogerlitz@nvidia.com>
From: Yoray Zack <yorayz@nvidia.com>

=========================================

This series adds support for NVMe-TCP receive offloads. The method here
does not mandate the offload of the network stack to the device.
Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers.
2. CRC calculation and verification for received PDU.

The series implements these as a generic offload infrastructure for storage
protocols, which calls TCP Direct Data Placement and TCP Offload CRC
respectively. We use this infrastructure to implement NVMe-TCP offload for
copy and CRC.
Future implementations can reuse the same infrastructure for other protocols
such as iSCSI.

Note:
These offloads are similar in nature to the packet-based NIC TLS offloads,
which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html

Queue Level
===========
The offload for IO queues is initialized after the handshake of the
NVMe-TCP protocol is finished by calling `nvme_tcp_offload_socket`
with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usual with no offload.
If it succeeds then `nvme_tcp_setup_ddp` and `nvme_tcp_teardown_ddp` may be
called to perform copy offload, and crc offload will be used.
This initialization does not change the normal operation of NVMe-TCP in any
way besides adding the option to call the above mentioned NDO operations.

For the admin queue, NVMe-TCP does not initialize the offload.
Instead, NVMe-TCP calls the driver to configure limits for the controller,
such as max_hw_sectors and max_segments, these must be limited to accommodate
potential HW resource limits, and to improve performance.

If some error occurs, and the IO queue must be closed or reconnected, then
offload is teardown and initialized again. Additionally, we handle netdev
down events via the existing error recovery flow.

IO Level
========
The NVMe-TCP layer calls the NIC driver to map block layer buffers to CID
using `nvme_tcp_setup_ddp` before sending the read request. When the response
is received, then the NIC HW will write the PDU payload directly into the
designated buffer, and build an SKB such that it points into the destination
buffer. This SKB represents the entire packet received on the wire, but it
points to the block layer buffers. Once NVMe-TCP attempts to copy data from
this SKB to the block layer buffer it can skip the copy by checking in the
copying function: if (src == dst) -> skip copy

Finally, when the PDU has been processed to completion, the NVMe-TCP layer
releases the NIC HW context by calling `nvme_tcp_teardown_ddp` which
asynchronously unmaps the buffers from NIC HW.

The NIC must release its mapping between command IDs and the target buffers.
This mapping is released when NVMe-TCP calls the NIC
driver (`nvme_tcp_offload_socket`).
As completing IOs is performance critical, we introduce asynchronous
completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later
call NVMe-TCP to complete the IO (`nvme_tcp_ddp_teardown_done`).

On the IO level, and in order to use the offload only when a clear
performance improvement is expected, the offload is used only for IOs
which are bigger than io_threshold.

SKB
===
The DDP (zero-copy) and CRC offloads require two additional bits in the SKB.
The ddp bit is useful to prevent condensing of SKBs which are targeted
for zero-copy. The crc bit is useful to prevent GRO coalescing SKBs with
different offload values. This bit is similar in concept to the
"decrypted" bit.

After offload is initialized, we use the SKB's crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's
payload". The bit is set to zero if there was an error, or if HW skipped
offload for some reason. If *any* SKB in a PDU has (crc != 1), then the
calling driver must compute the CRC, and check it. We perform this check, and
accompanying software fallback at the end of the processing of a received PDU.

Resynchronization flow
======================
The resynchronization flow is performed to reset the hardware tracking of
NVMe-TCP PDUs within the TCP stream. The flow consists of a request from
the hardware proxied by the driver, regarding a possible location of a
PDU header. Followed by a response from the NVMe-TCP driver.

This flow is rare, and it should happen only after packet loss or
reordering events that involve NVMe-TCP PDU headers.

CID Mapping
===========
ConnectX-7 assumes linear CID (0...N-1 for queue of size N) where the Linux NVMe
driver uses part of the 16 bit CCID for generation counter.
To address that, we use the existing quirk in the NVMe layer when the HW
driver advertises that they don't support the full 16 bit CCID range.

Enablement on ConnectX-7
========================
By default, NVMeTCP offload is disabled in the mlx driver. In order to enable it:

	# Disable CQE compression (specific for ConnectX)
	ethtool --set-priv-flags <device> rx_cqe_compress off

	# Enable the ULP-DDP
	ethtool -K <device> ulp-ddp-offload on

	# Enable ULP offload in nvme-tcp
	modprobe nvme-tcp ulp_offload=1

Following the device ULP-DDP enablement, all the IO queues/sockets which are
running on the device are offloaded.

Performance
===========
With this implementation, using the ConnectX-7 NIC, we were able to
demonstrate the following CPU utilization improvement:

Without data digest:
For  64K queued read IOs – up to 32% improvement in the BW/IOPS (111 Gbps vs. 84 Gbps).
For 512K queued read IOs – up to 55% improvement in the BW/IOPS (148 Gbps vs. 98 Gbps).

With data digest:
For  64K queued read IOs – up to 107% improvement in the BW/IOPS (111 Gbps vs. 53 Gbps).
For 512K queued read IOs – up to 138% improvement in the BW/IOPS (146 Gbps vs. 61 Gbps).

With small IOs we are not expecting that the offload will show
a performance gain.

The test configuration:
- fio command: qd=128, jobs=8.
- Server: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz, 160 cores.

Patches
=======
Patch 1:  Introduce the infrastructure for all ULP DDP and ULP DDP CRC offloads.
Patch 2:  The iov_iter change to skip copy if (src == dst).
Patch 3:  Export the get_netdev_for_sock function from TLS to generic location.
Patch 4:  Revert nvme_tcp_queue->queue_size removal
Patch 5:  NVMe-TCP changes to call NIC driver on queue init/teardown and resync
Patch 6:  NVMe-TCP changes to call NIC driver on IO operation
          setup/teardown, and support async completions
Patch 7:  NVMe-TCP changes to support CRC offload on receive
          Also, this patch moves CRC calculation to the end of PDU
          in case offload requires software fallback
Patch 8:  NVMe-TCP handling of netdev events: stop the offload if netdev is
          going down.
Patch 9:  Add module parameter to the NVMe-TCP control the enable ULP offload
Patch 10: Documentation of ULP DDP offloads

The rest of the series is the mlx5 implementation of the offload.

Testing
=======
This series was tested on ConnectX-7 HW using various configurations
of IO sizes, queue depths, MTUs, and with both the SPDK and kernel NVMe-TCP
targets.

Future Work
===========
- NVMeTCP transmit offload.
- NVMeTCP target offload.

Changes since v6:
=================
- Moved IS_ULP_{DDP,CRC} macros to skb_is_ulp_{ddp,crc} inline functions (Jakub).
- Fix copyright notice (Leon).
- Added missing ifdef to allow build with MLX5_EN_TLS disabled.
- Fix space alignement, indent and long lines (max 99 columns).
- Add missing field documentation in ulp_ddp.h.

Changes since v5:
=================
- Limit the series to RX offloads.
- Added two separated skb indications to avoid wrong flushing of GRO
  when aggerating offloaded packets.
- Use accessor functions for skb->ddp and skb->crc (Eric D) bits.
- Add kernel-doc for get_netdev_for_sock (Christoph).
- Remove ddp_iter* routines and only modify _copy_to_iter (Al Viro, Christoph).
- Remove consume skb (Sagi).
- Add a knob in the ddp limits struct for the HW driver to advertise
  if they need the nvme-tcp driver to apply the generation counter
  quirk. Use this knob for the mlx5 CX7 offload.
- bugfix: use u8 flags instead of bool in mlx5e_nvmeotcp_queue->dgst.
- bugfix: use sg_dma_len(sgl) instead of sgl->length.
- bugfix: remove sgl leak in nvme_tcp_setup_ddp().
- bugfix: remove sgl leak when only using DDGST_RX offload.
- Add error check for dma_map_sg().
- Reduce #ifdef by using dummy macros/functions.
- Remove redundant netdev null check in nvme_tcp_pdu_last_send().
- Rename ULP_DDP_RESYNC_{REQ -> PENDING}.
- Add per-ulp limits struct (Sagi).
- Add ULP DDP capabilities querying (Sagi).
- Simplify RX DDGST logic (Sagi).
- Document resync flow better.
- Add ulp_offload param to nvme-tcp module to enable ULP offload (Sagi).
- Add a revert commit to reintroduce nvme_tcp_queue->queue_size.

Changes since v4:
=================
- Add transmit offload patches.
- Use one feature bit for both receive and transmit offload.

Changes since v3:
=================
- Use DDP_TCP ifdefs in iov_iter and skb iterators to minimize impact
  when compiled out (Christoph).
- Simplify netdev references and reduce the use of
  get_netdev_for_sock (Sagi).
- Avoid "static" in it's own line, move it one line down (Christoph)
- Pass (queue, skb, *offset) and retrieve the pdu_seq in
  nvme_tcp_resync_response (Sagi).
- Add missing assignment of offloading_netdev to null in offload_limits
  error case (Sagi).
- Set req->offloaded = false once -- the lifetime rules are:
  set to false on cmd_setup / set to true when ddp setup succeeds (Sagi).
- Replace pr_info_ratelimited with dev_info_ratelimited (Sagi).
- Add nvme_tcp_complete_request and invoke it from two similar call
  sites (Sagi).
- Introduce nvme_tcp_req_map_sg earlier in the series (Sagi).
- Add nvme_tcp_consume_skb and put into it a hunk from
  nvme_tcp_recv_data to handle copy with and without offload.

Changes since v2:
=================
- Use skb->ddp_crc for copy offload to avoid skb_condense.
- Default mellanox driver support to no (experimental feature).
- In iov_iter use non-ddp functions for kvec and iovec.
- Remove typecasting in NVMe-TCP.

Changes since v1:
=================
- Rework iov_iter copy skip if src==dst to be less intrusive (David Ahern).
- Add tcp-ddp documentation (David Ahern).
- Refactor mellanox driver patches into more patches (Saeed Mahameed).
- Avoid pointer casting (David Ahern).
- Rename NVMe-TCP offload flags (Shai Malin).
- Update cover-letter according to the above.

Changes since RFC v1:
=====================
- Split mlx5 driver patches to several commits.
- Fix NVMe-TCP handling of recovery flows. In particular, move queue offload.
  init/teardown to the start/stop functions.

Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>

Aurelien Aptel (2):
  Revert "nvme-tcp: remove the unused queue_size member in
    nvme_tcp_queue"
  nvme-tcp: Add modparam to control the ULP offload enablement

Ben Ben-Ishay (10):
  iov_iter: DDP copy to iter/pages
  net/tls: export get_netdev_for_sock
  net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations
  net/mlx5e: NVMEoTCP, offload initialization
  net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration
  net/mlx5e: NVMEoTCP, queue init/teardown
  net/mlx5e: NVMEoTCP, ddp setup and resync
  net/mlx5e: NVMEoTCP, async ddp invalidation
  net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload
  net/mlx5e: NVMEoTCP, statistics

Boris Pismenny (4):
  net: Introduce direct data placement tcp offload
  nvme-tcp: Add DDP offload control path
  nvme-tcp: Add DDP data-path
  net/mlx5e: TCP flow steering for nvme-tcp acceleration

Or Gerlitz (5):
  nvme-tcp: Deal with netdevice DOWN events
  net/mlx5e: Rename from tls to transport static params
  net/mlx5e: Refactor ico sq polling to get budget
  net/mlx5e: Have mdev pointer directly on the icosq structure
  net/mlx5e: Refactor doorbell function to allow avoiding a completion

Yoray Zack (2):
  nvme-tcp: RX DDGST offload
  Documentation: add ULP DDP offload documentation

 Documentation/networking/index.rst            |    1 +
 Documentation/networking/ulp-ddp-offload.rst  |  368 ++++++
 .../net/ethernet/mellanox/mlx5/core/Kconfig   |   11 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |    3 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |   10 +
 .../net/ethernet/mellanox/mlx5/core/en/fs.h   |    4 +-
 .../ethernet/mellanox/mlx5/core/en/params.c   |   12 +-
 .../ethernet/mellanox/mlx5/core/en/params.h   |    3 +
 .../mellanox/mlx5/core/en/reporter_rx.c       |    4 +-
 .../ethernet/mellanox/mlx5/core/en/rx_res.c   |   28 +
 .../ethernet/mellanox/mlx5/core/en/rx_res.h   |    4 +
 .../net/ethernet/mellanox/mlx5/core/en/tir.c  |   15 +
 .../net/ethernet/mellanox/mlx5/core/en/tir.h  |    2 +
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |   28 +-
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   |    1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |    1 +
 .../mlx5/core/en_accel/common_utils.h         |   32 +
 .../mellanox/mlx5/core/en_accel/en_accel.h    |    3 +
 .../mellanox/mlx5/core/en_accel/fs_tcp.c      |   12 +-
 .../mellanox/mlx5/core/en_accel/fs_tcp.h      |    2 +-
 .../mellanox/mlx5/core/en_accel/ktls.c        |    2 +-
 .../mellanox/mlx5/core/en_accel/ktls_rx.c     |    8 +-
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     |    8 +-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c   |   36 +-
 .../mellanox/mlx5/core/en_accel/ktls_utils.h  |   17 +-
 .../mellanox/mlx5/core/en_accel/nvmeotcp.c    | 1068 +++++++++++++++++
 .../mellanox/mlx5/core/en_accel/nvmeotcp.h    |  144 +++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.c        |  325 +++++
 .../mlx5/core/en_accel/nvmeotcp_rxtx.h        |   37 +
 .../mlx5/core/en_accel/nvmeotcp_stats.c       |   61 +
 .../mlx5/core/en_accel/nvmeotcp_utils.h       |   66 +
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |    5 +
 .../net/ethernet/mellanox/mlx5/core/en_fs.c   |    4 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   39 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |   78 +-
 .../ethernet/mellanox/mlx5/core/en_stats.c    |   38 +
 .../ethernet/mellanox/mlx5/core/en_stats.h    |   12 +
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |    4 +-
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |    6 +
 .../net/ethernet/mellanox/mlx5/core/main.c    |    1 +
 drivers/nvme/host/tcp.c                       |  537 ++++++++-
 include/linux/mlx5/device.h                   |   59 +-
 include/linux/mlx5/mlx5_ifc.h                 |   82 +-
 include/linux/mlx5/qp.h                       |    1 +
 include/linux/netdev_features.h               |    3 +-
 include/linux/netdevice.h                     |    5 +
 include/linux/skbuff.h                        |   24 +
 include/net/inet_connection_sock.h            |    4 +
 include/net/sock.h                            |   23 +
 include/net/ulp_ddp.h                         |  182 +++
 lib/iov_iter.c                                |    2 +-
 net/Kconfig                                   |   10 +
 net/core/skbuff.c                             |    3 +-
 net/ethtool/common.c                          |    1 +
 net/ipv4/tcp_input.c                          |    8 +
 net/ipv4/tcp_ipv4.c                           |    3 +
 net/ipv4/tcp_offload.c                        |    3 +
 net/tls/tls_device.c                          |   16 -
 58 files changed, 3331 insertions(+), 138 deletions(-)
 create mode 100644 Documentation/networking/ulp-ddp-offload.rst
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
 create mode 100644 include/net/ulp_ddp.h

Christoph Hellwig Oct. 25, 2022, 4 p.m. UTC | #1

On Tue, Oct 25, 2022 at 04:59:35PM +0300, Aurelien Aptel wrote:
> The feature will also be presented in netdev this week
> https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-Implementation-and-Performance-Gains

That seems to miss slides.

> Currently the series is aligned to net-next, please update us if you will prefer otherwise.

Please also point to a git tree for a huge series with a dependency
on some tree, otherwise there's no good way to review it.

Or Gerlitz Oct. 26, 2022, 8:28 a.m. UTC | #2

On Tue, Oct 25, 2022 at 7:04 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Tue, Oct 25, 2022 at 04:59:35PM +0300, Aurelien Aptel wrote:
> > The feature will also be presented in netdev this week
> > https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-Implementation-and-Performance-Gains
>
> That seems to miss slides.

to be presented on Friday this week.. AFAIK slides are uploaded little later

The design/principles were presented last year
https://netdevconf.info/0x15/session.html?Autonomous-NVMe-TCP-offload

Aurelien Aptel Oct. 26, 2022, 11:52 a.m. UTC | #3

Hi Christoph,

>> Currently the series is aligned to net-next, please update us if you will prefer otherwise.
> Please also point to a git tree for a huge series with a dependency
> on some tree, otherwise there's no good way to review it.

This series is based on top of yesterday's net-next [1] and I created a
github tree if that's easier to use [2].

1: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
   branch 'main' commit d6dd508080a3 ("bnx2: Use kmalloc_size_roundup() to match ksize() usage")

2: Github: https://github.com/aaptel/linux/tree/nvme-rx-offload-v7
   Git repo: https://github.com/aaptel/linux.git branch nvme-rx-offload-v7

Sagi Grimberg Oct. 27, 2022, 10:35 a.m. UTC | #4

> Hi,
> 
> The nvme-tcp receive offloads series v7 was sent to both net-next and
> nvme.  It is the continuation of v5 which was sent on July 2021
> https://lore.kernel.org/netdev/20210722110325.371-1-borisp@nvidia.com/ .
> V7 is now working on a real HW.
> 
> The feature will also be presented in netdev this week
> https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-Implementation-and-Performance-Gains
> 
> Currently the series is aligned to net-next, please update us if you will prefer otherwise.
> 
> Thanks,
> Shai, Aurelien

Hey Shai & Aurelien

Can you please add in the next time documentation of the limitations
that this offload has in terms of compatibility? i.e. for example (from
my own imagination):
1. bonding/teaming/other-stacking?
2. TLS (sw/hw)?
3. any sort of tunneling/overlay?
4. VF/PF?
5. any nvme features?
6. ...

And what are your plans to address each if at all.

Also, does this have a path to userspace? for example almost all
of the nvme-tcp targets live in userspace.

I don't think I see in the code any limits like the maximum
connections that can be offloaded on a single device/port. Can
you share some details on this?

Thanks.

Shai Malin Oct. 27, 2022, 10:45 a.m. UTC | #5

On Thu, 27 Oct 2022 at 11:35, Sagi Grimberg <sagi@grimberg.me> wrote:
> > Hi,
> >
> > The nvme-tcp receive offloads series v7 was sent to both net-next and
> > nvme.  It is the continuation of v5 which was sent on July 2021
> > https://lore.kernel.org/netdev/20210722110325.371-1-borisp@nvidia.com/ .
> > V7 is now working on a real HW.
> >
> > The feature will also be presented in netdev this week
> > https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-
> Implementation-and-Performance-Gains
> >
> > Currently the series is aligned to net-next, please update us if you will prefer
> otherwise.
> >
> > Thanks,
> > Shai, Aurelien
> 
> Hey Shai & Aurelien
> 
> Can you please add in the next time documentation of the limitations
> that this offload has in terms of compatibility? i.e. for example (from
> my own imagination):
> 1. bonding/teaming/other-stacking?
> 2. TLS (sw/hw)?
> 3. any sort of tunneling/overlay?
> 4. VF/PF?
> 5. any nvme features?
> 6. ...
> 
> And what are your plans to address each if at all.
> 
> Also, does this have a path to userspace? for example almost all
> of the nvme-tcp targets live in userspace.
> 
> I don't think I see in the code any limits like the maximum
> connections that can be offloaded on a single device/port. Can
> you share some details on this?
> 
> Thanks.

Sure, we add it.

[v7,00/23] nvme-tcp receive offloads

Message

Comments