[net-next,V2,0/6] mlx5e use TLS TX pool to improve connection rate

Message

Tariq Toukan July 13, 2022, 5:15 a.m. UTC

To offload encryption operations, the mlx5 device maintains state and
keeps track of every kTLS device-offloaded connection.  Two HW objects
are used per TX context of a kTLS offloaded connection: a. Transport
interface send (TIS) object, to reach the HW context.  b. Data Encryption
Key (DEK) to perform the crypto operations.

These two objects are created and destroyed per TLS TX context, via FW
commands.  In total, 4 FW commands are issued per TLS TX context, which
seriously limits the connection rate.

In this series, we aim to save creation and destroy of TIS objects by
recycling them.  Upon recycling of a TIS, the HW still needs to be
notified for the re-mapping between a TIS and a context. This is done by
posting WQEs via an SQ, significantly faster API than the FW command
interface.

A pool is used for recycling. The pool dynamically interacts to the load
and connection rate, growing and shrinking accordingly.

Saving the TIS FW commands per context increases connection rate by ~42%,
from 11.6K to 16.5K connections per sec.

Connection rate is still limited by FW bottleneck due to the remaining
per context FW commands (DEK create/destroy). This will soon be addressed
in a followup series.  By combining the two series, the FW bottleneck
will be released, and a significantly higher (about 100K connections per
sec) kTLS TX device-offloaded connection rate is reached.

Regards,
Tariq

Tariq Toukan (6):
  net/tls: Perform immediate device ctx cleanup when possible
  net/tls: Multi-threaded calls to TX tls_dev_del
  net/mlx5e: kTLS, Introduce TLS-specific create TIS
  net/mlx5e: kTLS, Take stats out of OOO handler
  net/mlx5e: kTLS, Recycle objects of device-offloaded TLS TX
    connections
  net/mlx5e: kTLS, Dynamically re-size TX recycling pool

 .../mellanox/mlx5/core/en_accel/en_accel.h    |  10 +
 .../mellanox/mlx5/core/en_accel/ktls.h        |  14 +
 .../mellanox/mlx5/core/en_accel/ktls_stats.c  |   2 +
 .../mellanox/mlx5/core/en_accel/ktls_tx.c     | 513 +++++++++++++++---
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   9 +
 include/net/tls.h                             |   2 +
 net/tls/tls.h                                 |   4 +-
 net/tls/tls_device.c                          |  65 ++-
 net/tls/tls_main.c                            |   7 +-
 9 files changed, 523 insertions(+), 103 deletions(-)