| Message ID | 20220727094346.10540-1-tariqt@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | mlx5e use TLS TX pool to improve connection rate | expand |
Hello: This series was applied to netdev/net-next.git (master) by Jakub Kicinski <kuba@kernel.org>: On Wed, 27 Jul 2022 12:43:40 +0300 you wrote: > To offload encryption operations, the mlx5 device maintains state and > keeps track of every kTLS device-offloaded connection. Two HW objects > are used per TX context of a kTLS offloaded connection: a. Transport > interface send (TIS) object, to reach the HW context. b. Data Encryption > Key (DEK) to perform the crypto operations. > > These two objects are created and destroyed per TLS TX context, via FW > commands. In total, 4 FW commands are issued per TLS TX context, which > seriously limits the connection rate. > > [...] Here is the summary with links: - [net-next,V3,1/6] net/tls: Perform immediate device ctx cleanup when possible https://git.kernel.org/netdev/net-next/c/113671b255ee - [net-next,V3,2/6] net/tls: Multi-threaded calls to TX tls_dev_del https://git.kernel.org/netdev/net-next/c/7adc91e0c939 - [net-next,V3,3/6] net/mlx5e: kTLS, Introduce TLS-specific create TIS https://git.kernel.org/netdev/net-next/c/da6682faa82f - [net-next,V3,4/6] net/mlx5e: kTLS, Take stats out of OOO handler https://git.kernel.org/netdev/net-next/c/23b1cf1e3fe0 - [net-next,V3,5/6] net/mlx5e: kTLS, Recycle objects of device-offloaded TLS TX connections https://git.kernel.org/netdev/net-next/c/c4dfe704f53f - [net-next,V3,6/6] net/mlx5e: kTLS, Dynamically re-size TX recycling pool https://git.kernel.org/netdev/net-next/c/624bf0992133 You are awesome, thank you!
To offload encryption operations, the mlx5 device maintains state and keeps track of every kTLS device-offloaded connection. Two HW objects are used per TX context of a kTLS offloaded connection: a. Transport interface send (TIS) object, to reach the HW context. b. Data Encryption Key (DEK) to perform the crypto operations. These two objects are created and destroyed per TLS TX context, via FW commands. In total, 4 FW commands are issued per TLS TX context, which seriously limits the connection rate. In this series, we aim to save creation and destroy of TIS objects by recycling them. Upon recycling of a TIS, the HW still needs to be notified for the re-mapping between a TIS and a context. This is done by posting WQEs via an SQ, significantly faster API than the FW command interface. A pool is used for recycling. The pool dynamically interacts to the load and connection rate, growing and shrinking accordingly. Saving the TIS FW commands per context increases connection rate by ~42%, from 11.6K to 16.5K connections per sec. Connection rate is still limited by FW bottleneck due to the remaining per context FW commands (DEK create/destroy). This will soon be addressed in a followup series. By combining the two series, the FW bottleneck will be released, and a significantly higher (about 100K connections per sec) kTLS TX device-offloaded connection rate is reached. Regards, Tariq v3: Rebased on top of relevant fixes in TLS module. Tariq Toukan (6): net/tls: Perform immediate device ctx cleanup when possible net/tls: Multi-threaded calls to TX tls_dev_del net/mlx5e: kTLS, Introduce TLS-specific create TIS net/mlx5e: kTLS, Take stats out of OOO handler net/mlx5e: kTLS, Recycle objects of device-offloaded TLS TX connections net/mlx5e: kTLS, Dynamically re-size TX recycling pool .../mellanox/mlx5/core/en_accel/en_accel.h | 10 + .../mellanox/mlx5/core/en_accel/ktls.h | 14 + .../mellanox/mlx5/core/en_accel/ktls_stats.c | 2 + .../mellanox/mlx5/core/en_accel/ktls_tx.c | 513 +++++++++++++++--- .../net/ethernet/mellanox/mlx5/core/en_main.c | 9 + include/net/tls.h | 2 + net/tls/tls_device.c | 79 +-- 7 files changed, 527 insertions(+), 102 deletions(-)