| Message ID | 20220917031104.21222-1-rpearsonhpe@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Implement the xrc transport | expand |
On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote: > This patch series implements the xrc transport for the rdma_rxe driver. > It is based on the current for-next branch of rdma-linux. > The first two patches in the series do some cleanup which is helpful > for this effort. The remaining patches implement the xrc functionality. > There is a matching patch set for the user space rxe provider driver. > The communications between these is accomplished without making an > ABI change by taking advantage of the space freed up by a recent > patch called "Remove redundant num_sge fields" which is a reprequisite > for this patch series. > > The two patch sets have been tested with the pyverbs regression test > suite with and without each set installed. This series enables 5 of > the 6 xrc test cases in pyverbs. The ODP case does is currently skipped > but should work once the ODP patch series is accepted. The ODP patch isn't even on patchworks any more, so it needs resending. I can't remember why it needed respin now. I'm inclined to apply this without really looking closely at the rxe code. If someone has other ideas please chime in. It looks like it needs rebasing, yes? Jason
On Tue, Sep 27, 2022 8:14 AM, Jason Gunthorpe wrote: > On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote: > > This patch series implements the xrc transport for the rdma_rxe driver. > > It is based on the current for-next branch of rdma-linux. > > The first two patches in the series do some cleanup which is helpful > > for this effort. The remaining patches implement the xrc functionality. > > There is a matching patch set for the user space rxe provider driver. > > The communications between these is accomplished without making an > > ABI change by taking advantage of the space freed up by a recent > > patch called "Remove redundant num_sge fields" which is a reprequisite > > for this patch series. > > > > The two patch sets have been tested with the pyverbs regression test > > suite with and without each set installed. This series enables 5 of > > the 6 xrc test cases in pyverbs. The ODP case does is currently skipped > > but should work once the ODP patch series is accepted. > > The ODP patch isn't even on patchworks any more, so it needs > resending. I can't remember why it needed respin now. The ODP patch series is the one I posted for rxe this month: [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/ https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both We had an argument about the way to use workqueues instead of tasklets. Some prefer to get rid of tasklets, but others prefer finding a way to switch between the bottom halves for performance reasons. I am currently taking some data to continue the discussion while waiting for Bob to post their(HPE's) implementation that enables the switching. I think I can resend the ODP patches without an RFC tag once we reach an agreement on this point. Thanks, Daisuke Matsuda > > I'm inclined to apply this without really looking closely at the rxe > code. If someone has other ideas please chime in. It looks like it > needs rebasing, yes? > > Jason
On 9/26/22 20:38, matsuda-daisuke@fujitsu.com wrote: > On Tue, Sep 27, 2022 8:14 AM, Jason Gunthorpe wrote: >> On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote: >>> This patch series implements the xrc transport for the rdma_rxe driver. >>> It is based on the current for-next branch of rdma-linux. >>> The first two patches in the series do some cleanup which is helpful >>> for this effort. The remaining patches implement the xrc functionality. >>> There is a matching patch set for the user space rxe provider driver. >>> The communications between these is accomplished without making an >>> ABI change by taking advantage of the space freed up by a recent >>> patch called "Remove redundant num_sge fields" which is a reprequisite >>> for this patch series. >>> >>> The two patch sets have been tested with the pyverbs regression test >>> suite with and without each set installed. This series enables 5 of >>> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped >>> but should work once the ODP patch series is accepted. >> >> The ODP patch isn't even on patchworks any more, so it needs >> resending. I can't remember why it needed respin now. > > The ODP patch series is the one I posted for rxe this month: > [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE > https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/ > https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both > > We had an argument about the way to use workqueues instead of tasklets. > Some prefer to get rid of tasklets, but others prefer finding a way to switch > between the bottom halves for performance reasons. I am currently taking > some data to continue the discussion while waiting for Bob to post their(HPE's) > implementation that enables the switching. I think I can resend the ODP patches > without an RFC tag once we reach an agreement on this point. I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy and finally after a long while he asked me to do that. I have to rebase it from an older kernel to head of tree. I hope to have that done in a day or two. Bob > > Thanks, > Daisuke Matsuda > >> >> I'm inclined to apply this without really looking closely at the rxe >> code. If someone has other ideas please chime in. It looks like it >> needs rebasing, yes? >> >> Jason
On 9/26/22 18:13, Jason Gunthorpe wrote: > On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote: >> This patch series implements the xrc transport for the rdma_rxe driver. >> It is based on the current for-next branch of rdma-linux. >> The first two patches in the series do some cleanup which is helpful >> for this effort. The remaining patches implement the xrc functionality. >> There is a matching patch set for the user space rxe provider driver. >> The communications between these is accomplished without making an >> ABI change by taking advantage of the space freed up by a recent >> patch called "Remove redundant num_sge fields" which is a reprequisite >> for this patch series. >> >> The two patch sets have been tested with the pyverbs regression test >> suite with and without each set installed. This series enables 5 of >> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped >> but should work once the ODP patch series is accepted. > > The ODP patch isn't even on patchworks any more, so it needs > resending. I can't remember why it needed respin now. > > I'm inclined to apply this without really looking closely at the rxe > code. If someone has other ideas please chime in. It looks like it > needs rebasing, yes? I will do that today and resend if it needs to change. Bob > > Jason
On Fri, Sep 30, 2022 12:59 AM Bob Pearson wrote: > On 9/26/22 20:38, matsuda-daisuke@fujitsu.com wrote: > > On Tue, Sep 27, 2022 8:14 AM, Jason Gunthorpe wrote: > >> On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote: > >>> This patch series implements the xrc transport for the rdma_rxe driver. > >>> It is based on the current for-next branch of rdma-linux. > >>> The first two patches in the series do some cleanup which is helpful > >>> for this effort. The remaining patches implement the xrc functionality. > >>> There is a matching patch set for the user space rxe provider driver. > >>> The communications between these is accomplished without making an > >>> ABI change by taking advantage of the space freed up by a recent > >>> patch called "Remove redundant num_sge fields" which is a reprequisite > >>> for this patch series. > >>> > >>> The two patch sets have been tested with the pyverbs regression test > >>> suite with and without each set installed. This series enables 5 of > >>> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped > >>> but should work once the ODP patch series is accepted. > >> > >> The ODP patch isn't even on patchworks any more, so it needs > >> resending. I can't remember why it needed respin now. > > > > The ODP patch series is the one I posted for rxe this month: > > [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE > > https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/ > > https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both > > > > We had an argument about the way to use workqueues instead of tasklets. > > Some prefer to get rid of tasklets, but others prefer finding a way to switch > > between the bottom halves for performance reasons. I am currently taking > > some data to continue the discussion while waiting for Bob to post their(HPE's) > > implementation that enables the switching. I think I can resend the ODP patches > > without an RFC tag once we reach an agreement on this point. > > I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy > and finally after a long while he asked me to do that. I have to rebase it from > an older kernel to head of tree. I hope to have that done in a day or two. > > Bob Thank you for the update. I am glad to hear your work is in progress just now. I am looking forward to seeing your work! Daisuke > > > > Thanks, > > Daisuke Matsuda > > > >> > >> I'm inclined to apply this without really looking closely at the rxe > >> code. If someone has other ideas please chime in. It looks like it > >> needs rebasing, yes? > >> > >> Jason >
Hi Bob, I am ready and willing to review your workqueue implementation. Could you inform me of the current status of the patch series? Are you having trouble rebasing it? Daisuke > > > The ODP patch series is the one I posted for rxe this month: > > > [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE > > > https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/ > > > https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both > > > > > > We had an argument about the way to use workqueues instead of tasklets. > > > Some prefer to get rid of tasklets, but others prefer finding a way to switch > > > between the bottom halves for performance reasons. I am currently taking > > > some data to continue the discussion while waiting for Bob to post their(HPE's) > > > implementation that enables the switching. I think I can resend the ODP patches > > > without an RFC tag once we reach an agreement on this point. > > > > I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy > > and finally after a long while he asked me to do that. I have to rebase it from > > an older kernel to head of tree. I hope to have that done in a day or two. > > > > Bob > > Thank you for the update. > I am glad to hear your work is in progress just now. > I am looking forward to seeing your work! > > Daisuke
On 10/12/22 02:41, matsuda-daisuke@fujitsu.com wrote: > Hi Bob, > > I am ready and willing to review your workqueue implementation. > Could you inform me of the current status of the patch series? > Are you having trouble rebasing it? > > Daisuke > >>>> The ODP patch series is the one I posted for rxe this month: >>>> [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE >>>> https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/ >>>> https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both >>>> >>>> We had an argument about the way to use workqueues instead of tasklets. >>>> Some prefer to get rid of tasklets, but others prefer finding a way to switch >>>> between the bottom halves for performance reasons. I am currently taking >>>> some data to continue the discussion while waiting for Bob to post their(HPE's) >>>> implementation that enables the switching. I think I can resend the ODP patches >>>> without an RFC tag once we reach an agreement on this point. >>> >>> I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy >>> and finally after a long while he asked me to do that. I have to rebase it from >>> an older kernel to head of tree. I hope to have that done in a day or two. >>> >>> Bob >> >> Thank you for the update. >> I am glad to hear your work is in progress just now. >> I am looking forward to seeing your work! >> >> Daisuke > I know. Thanks. It's ready to send in. I am testing the performance and was surprised that the tasklet performance has gotten better in 6.0. Not sure why. I'll send it today. Bob
On Fri, Oct 14, 2022 2:18 AM Bob Pearson wrote: > On 10/12/22 02:41, matsuda-daisuke@fujitsu.com wrote: > > Hi Bob, > > > > I am ready and willing to review your workqueue implementation. > > Could you inform me of the current status of the patch series? > > Are you having trouble rebasing it? > > > > Daisuke > > > >>>> The ODP patch series is the one I posted for rxe this month: > >>>> [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE > >>>> https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/ > >>>> https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both > >>>> > >>>> We had an argument about the way to use workqueues instead of tasklets. > >>>> Some prefer to get rid of tasklets, but others prefer finding a way to switch > >>>> between the bottom halves for performance reasons. I am currently taking > >>>> some data to continue the discussion while waiting for Bob to post their(HPE's) > >>>> implementation that enables the switching. I think I can resend the ODP patches > >>>> without an RFC tag once we reach an agreement on this point. > >>> > >>> I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy > >>> and finally after a long while he asked me to do that. I have to rebase it from > >>> an older kernel to head of tree. I hope to have that done in a day or two. > >>> > >>> Bob > >> > >> Thank you for the update. > >> I am glad to hear your work is in progress just now. > >> I am looking forward to seeing your work! > >> > >> Daisuke > > > > I know. Thanks. It's ready to send in. I am testing the performance and was surprised that the tasklet performance has > gotten better in 6.0. Not sure why. I'll send it today. > > Bob Thank you for the reply. The performance change sounds interesting. BTW, would you please send it in as soon as possible? I would like to see if I can easily rebase my ODP patches on it, and also I will start testing and reviewing your change so that it will be merged safely and quickly. Daisuke
This patch series implements the xrc transport for the rdma_rxe driver. It is based on the current for-next branch of rdma-linux. The first two patches in the series do some cleanup which is helpful for this effort. The remaining patches implement the xrc functionality. There is a matching patch set for the user space rxe provider driver. The communications between these is accomplished without making an ABI change by taking advantage of the space freed up by a recent patch called "Remove redundant num_sge fields" which is a reprequisite for this patch series. The two patch sets have been tested with the pyverbs regression test suite with and without each set installed. This series enables 5 of the 6 xrc test cases in pyverbs. The ODP case does is currently skipped but should work once the ODP patch series is accepted. Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com> Bob Pearson (13): RDMA/rxe: Replace START->FIRST, END->LAST RDMA/rxe: Move next_opcode() to rxe_opcode.c RDMA: Add xrc opcodes to ib_pack.h RDMA/rxe: Extend opcodes and headers to support xrc RDMA/rxe: Add xrc opcodes to next_opcode() RDMA/rxe: Implement open_xrcd and close_xrcd RDMA/rxe: Extend srq verbs to support xrcd RDMA/rxe: Extend rxe_qp.c to support xrc qps RDMA/rxe: Extend rxe_recv.c to support xrc RDMA/rxe: Extend rxe_comp.c to support xrc qps RDMA/rxe: Extend rxe_req.c to support xrc qps RDMA/rxe: Extend rxe_net.c to support xrc qps RDMA/rxe: Extend rxe_resp.c to support xrc qps drivers/infiniband/sw/rxe/rxe.c | 2 + drivers/infiniband/sw/rxe/rxe_av.c | 3 +- drivers/infiniband/sw/rxe/rxe_comp.c | 51 +- drivers/infiniband/sw/rxe/rxe_hdr.h | 41 +- drivers/infiniband/sw/rxe/rxe_loc.h | 13 +- drivers/infiniband/sw/rxe/rxe_mw.c | 14 +- drivers/infiniband/sw/rxe/rxe_net.c | 23 +- drivers/infiniband/sw/rxe/rxe_opcode.c | 766 +++++++++++++++++++++---- drivers/infiniband/sw/rxe/rxe_opcode.h | 9 +- drivers/infiniband/sw/rxe/rxe_param.h | 3 + drivers/infiniband/sw/rxe/rxe_pool.c | 8 + drivers/infiniband/sw/rxe/rxe_pool.h | 1 + drivers/infiniband/sw/rxe/rxe_qp.c | 307 ++++++---- drivers/infiniband/sw/rxe/rxe_recv.c | 79 ++- drivers/infiniband/sw/rxe/rxe_req.c | 204 +------ drivers/infiniband/sw/rxe/rxe_resp.c | 165 ++++-- drivers/infiniband/sw/rxe/rxe_srq.c | 131 +++-- drivers/infiniband/sw/rxe/rxe_verbs.c | 58 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 20 +- include/rdma/ib_pack.h | 32 +- include/uapi/rdma/rdma_user_rxe.h | 4 +- 21 files changed, 1338 insertions(+), 596 deletions(-) base-commit: db77d84cfe3608eac938302f8f7178e44415bcba