| Message ID | 20240208035352.387423-1-saeed@kernel.org (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
On Wed, 7 Feb 2024 19:53:37 -0800 Saeed Mahameed wrote: > This series adds support for combining multiple devices (PFs) of the > same port under one netdev instance. Passing traffic through different > devices belonging to different NUMA sockets saves cross-numa traffic and > allows apps running on the same netdev from different numas to still > feel a sense of proximity to the device and acheive improved > performance. s/acheive/acheive/ throughout
On Fri, 9 Feb 2024 21:54:03 -0800 Jakub Kicinski wrote: > On Wed, 7 Feb 2024 19:53:37 -0800 Saeed Mahameed wrote: > > This series adds support for combining multiple devices (PFs) of the > > same port under one netdev instance. Passing traffic through different > > devices belonging to different NUMA sockets saves cross-numa traffic and > > allows apps running on the same netdev from different numas to still > > feel a sense of proximity to the device and acheive improved > > performance. > > s/acheive/acheive/ throughout Heh, I misspelled :) You know what I mean.
From: Saeed Mahameed <saeedm@nvidia.com> Support Socket-Direct multi-dev netdev. V1: - https://lore.kernel.org/netdev/20231221005721.186607-1-saeed@kernel.org/ V2: - Add documentation in a new patch. - Add debugfs in a new patch. - Add mlx5_ifc bit for MPIR cap check and use it before query. For more information please see tag log below. Please pull and let me know if there is any problem. Thanks, Saeed. The following changes since commit 1701940b1a02addc8fe445538442112e84270b02: Merge branch 'tools-net-ynl-add-features-for-tc-family' (2024-01-31 21:19:22 -0800) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-socket-direct-v2 for you to fetch changes up to 1bf74fd7f25345fa46cd8e7c44c0b39c62bea119: Documentation: net/mlx5: Add description for Socket-Direct netdev combining (2024-02-01 00:08:06 -0800) ---------------------------------------------------------------- mlx5-socket-direct-v2 Support Socket-Direct multi-dev netdev This series adds support for combining multiple devices (PFs) of the same port under one netdev instance. Passing traffic through different devices belonging to different NUMA sockets saves cross-numa traffic and allows apps running on the same netdev from different numas to still feel a sense of proximity to the device and acheive improved performance. We acheive this by grouping PFs together, and creating the netdev only once all group members are probed. Symmetrically, we destroy the netdev once any of the PFs is removed. The channels are distributed between all devices, a proper configuration would utilize the correct close numa when working on a certain app/cpu. We pick one device to be a primary (leader), and it fills a special role. The other devices (secondaries) are disconnected from the network in the chip level (set to silent mode). All RX/TX traffic is steered through the primary to/from the secondaries. Currently, we limit the support to PFs only, and up to two devices (sockets). ---------------------------------------------------------------- Tariq Toukan (15): net/mlx5: Add MPIR bit in mcam_access_reg net/mlx5: SD, Introduce SD lib net/mlx5: SD, Implement basic query and instantiation net/mlx5: SD, Implement devcom communication and primary election net/mlx5: SD, Implement steering for primary and secondaries net/mlx5: SD, Add informative prints in kernel log net/mlx5: SD, Add debugfs net/mlx5e: Create single netdev per SD group net/mlx5e: Create EN core HW resources for all secondary devices net/mlx5e: Let channels be SD-aware net/mlx5e: Support cross-vhca RSS net/mlx5e: Support per-mdev queue counter net/mlx5e: Block TLS device offload on combined SD netdev net/mlx5: Enable SD feature Documentation: net/mlx5: Add description for Socket-Direct netdev combining .../device_drivers/ethernet/mellanox/mlx5/sd.rst | 134 ++++++ drivers/net/ethernet/mellanox/mlx5/core/Makefile | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 9 +- .../net/ethernet/mellanox/mlx5/core/en/channels.c | 10 +- .../net/ethernet/mellanox/mlx5/core/en/channels.h | 6 +- .../ethernet/mellanox/mlx5/core/en/monitor_stats.c | 48 +- .../net/ethernet/mellanox/mlx5/core/en/params.c | 9 +- .../net/ethernet/mellanox/mlx5/core/en/params.h | 3 - drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c | 12 +- drivers/net/ethernet/mellanox/mlx5/core/en/qos.c | 8 +- .../ethernet/mellanox/mlx5/core/en/reporter_rx.c | 4 +- .../ethernet/mellanox/mlx5/core/en/reporter_tx.c | 3 +- drivers/net/ethernet/mellanox/mlx5/core/en/rqt.c | 123 ++++- drivers/net/ethernet/mellanox/mlx5/core/en/rqt.h | 9 +- drivers/net/ethernet/mellanox/mlx5/core/en/rss.c | 17 +- drivers/net/ethernet/mellanox/mlx5/core/en/rss.h | 4 +- .../net/ethernet/mellanox/mlx5/core/en/rx_res.c | 62 ++- .../net/ethernet/mellanox/mlx5/core/en/rx_res.h | 1 + drivers/net/ethernet/mellanox/mlx5/core/en/trap.c | 11 +- .../net/ethernet/mellanox/mlx5/core/en/xsk/pool.c | 6 +- .../net/ethernet/mellanox/mlx5/core/en/xsk/setup.c | 8 +- .../ethernet/mellanox/mlx5/core/en_accel/ktls.c | 2 +- .../ethernet/mellanox/mlx5/core/en_accel/ktls.h | 4 +- .../ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c | 6 +- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 176 +++++-- drivers/net/ethernet/mellanox/mlx5/core/en_stats.c | 39 +- drivers/net/ethernet/mellanox/mlx5/core/en_tc.c | 4 +- .../net/ethernet/mellanox/mlx5/core/lib/devcom.h | 1 + drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 12 + drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c | 524 +++++++++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/lib/sd.h | 38 ++ include/linux/mlx5/driver.h | 1 + include/linux/mlx5/mlx5_ifc.h | 4 +- 33 files changed, 1127 insertions(+), 173 deletions(-) create mode 100644 Documentation/networking/device_drivers/ethernet/mellanox/mlx5/sd.rst create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/sd.h