[v7,0/8] RDMA/rxe: Add atomic write operation

Message

Xiao Yang Dec. 1, 2022, 2:37 p.m. UTC

The IB SPEC v1.5[1] defined new atomic write operation. This patchset
makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API
for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
example and test_qp_ex_rc_atomic_write python test to verify
the patchset.

The steps to run the rdma_atomic_write example:
server:
$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.013s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

v6->v7:
1) Rebase on current for-next
2) Define IB_WC_ATOMIC_WRITE in order

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

v4->v5:
1) Rebase on current wip/jgg-for-next
2) Rewrite the implementation on responder

v3->v4:
1) Rebase on current wip/jgg-for-next
2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write
3) Replace 64-bit value with 8-byte array for atomic write

V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device

V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed

Xiao Yang (8):
  RDMA: Extend RDMA user ABI to support atomic write
  RDMA: Extend RDMA kernel ABI to support atomic write
  RDMA/rxe: Extend rxe user ABI to support atomic write
  RDMA/rxe: Extend rxe packet format to support atomic write
  RDMA/rxe: Make requester support atomic write on RC service
  RDMA/rxe: Make responder support atomic write on RC service
  RDMA/rxe: Implement atomic write completion
  RDMA/rxe: Enable atomic write capability for rxe device

 drivers/infiniband/sw/rxe/rxe_comp.c   |  4 ++
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +
 drivers/infiniband/sw/rxe/rxe_param.h  |  5 ++
 drivers/infiniband/sw/rxe/rxe_req.c    | 15 ++++-
 drivers/infiniband/sw/rxe/rxe_resp.c   | 84 ++++++++++++++++++++++++--
 include/rdma/ib_pack.h                 |  2 +
 include/rdma/ib_verbs.h                |  3 +
 include/uapi/rdma/ib_user_verbs.h      |  4 ++
 include/uapi/rdma/rdma_user_rxe.h      |  1 +
 10 files changed, 132 insertions(+), 7 deletions(-)

Comments

Jason Gunthorpe Dec. 1, 2022, 11:55 p.m. UTC | #1

On Thu, Dec 01, 2022 at 02:37:04PM +0000, Xiao Yang wrote:

> Xiao Yang (8):
>   RDMA: Extend RDMA user ABI to support atomic write
>   RDMA: Extend RDMA kernel ABI to support atomic write
>   RDMA/rxe: Extend rxe user ABI to support atomic write
>   RDMA/rxe: Extend rxe packet format to support atomic write
>   RDMA/rxe: Make requester support atomic write on RC service
>   RDMA/rxe: Make responder support atomic write on RC service
>   RDMA/rxe: Implement atomic write completion
>   RDMA/rxe: Enable atomic write capability for rxe device

Applied to for-next, thanks

Jason

Xiao Yang Dec. 2, 2022, 5:37 a.m. UTC | #2

On 2022/12/2 7:55, Jason Gunthorpe wrote:
> On Thu, Dec 01, 2022 at 02:37:04PM +0000, Xiao Yang wrote:
> 
>> Xiao Yang (8):
>>    RDMA: Extend RDMA user ABI to support atomic write
>>    RDMA: Extend RDMA kernel ABI to support atomic write
>>    RDMA/rxe: Extend rxe user ABI to support atomic write
>>    RDMA/rxe: Extend rxe packet format to support atomic write
>>    RDMA/rxe: Make requester support atomic write on RC service
>>    RDMA/rxe: Make responder support atomic write on RC service
>>    RDMA/rxe: Implement atomic write completion
>>    RDMA/rxe: Enable atomic write capability for rxe device
> 
> Applied to for-next, thanks
> 
> Jason

Hi Jason,

Cool, Thank you very much for reviewing and applying the patch set.
Besides, I have removed the rdma_atomic_write example as you suggested 
on rdma-core repo.

Best Regards,
Xiao Yang