| Message ID | 20240429152537.212958-6-cel@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | NFS: Fix another 'check_flush_dependency' splat | expand |
On 29.04.24 17:25, cel@kernel.org wrote: > From: Chuck Lever <chuck.lever@oracle.com> > > Avoid getting work queue splats in the system journal by moving > client-side RPC/RDMA transport tear-down into a background process. > > I've done some testing of this series, now looking for review > comments. How to make tests with nfs && rdma? Can you provide some steps or tools? I am interested in nfs && rdma. Thanks, Zhu Yanjun > > Chuck Lever (4): > xprtrdma: Remove temp allocation of rpcrdma_rep objects > xprtrdma: Clean up synopsis of frwr_mr_unmap() > xprtrdma: Delay releasing connection hardware resources > xprtrdma: Move MRs to struct rpcrdma_ep > > net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- > net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- > net/sunrpc/xprtrdma/transport.c | 20 +++- > net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- > net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- > 5 files changed, 125 insertions(+), 105 deletions(-) > > > base-commit: e67572cd2204894179d89bd7b984072f19313b03
> On Apr 30, 2024, at 3:26 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: > > On 29.04.24 17:25, cel@kernel.org wrote: >> From: Chuck Lever <chuck.lever@oracle.com> >> Avoid getting work queue splats in the system journal by moving >> client-side RPC/RDMA transport tear-down into a background process. >> I've done some testing of this series, now looking for review >> comments. > > How to make tests with nfs && rdma? Can you provide some steps or tools? We are building NFS tests into kdevops: https://github.com/linux-kdevops/kdevops.git and there is a config option to use soft iWARP instead of TCP. kdevops includes workflows for fstests, Mora's nfstest, the git regression suite, and ltp, all of which we use regularly to test the Linux NFS client and server implementations. > I am interested in nfs && rdma. > > Thanks, > Zhu Yanjun > >> Chuck Lever (4): >> xprtrdma: Remove temp allocation of rpcrdma_rep objects >> xprtrdma: Clean up synopsis of frwr_mr_unmap() >> xprtrdma: Delay releasing connection hardware resources >> xprtrdma: Move MRs to struct rpcrdma_ep >> net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- >> net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- >> net/sunrpc/xprtrdma/transport.c | 20 +++- >> net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- >> net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- >> 5 files changed, 125 insertions(+), 105 deletions(-) >> base-commit: e67572cd2204894179d89bd7b984072f19313b03 > -- Chuck Lever
On 30.04.24 15:42, Chuck Lever III wrote: > >> On Apr 30, 2024, at 3:26 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: >> >> On 29.04.24 17:25, cel@kernel.org wrote: >>> From: Chuck Lever <chuck.lever@oracle.com> >>> Avoid getting work queue splats in the system journal by moving >>> client-side RPC/RDMA transport tear-down into a background process. >>> I've done some testing of this series, now looking for review >>> comments. >> How to make tests with nfs && rdma? Can you provide some steps or tools? > We are building NFS tests into kdevops: > > https://github.com/linux-kdevops/kdevops.git > > and there is a config option to use soft iWARP instead of TCP. Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP? If yes, does nfs work well with RXE? I am just curious with nfs && RXE. Normally nfs works with TCP. Now nfs will use RDMA instead of TCP. The popular RDMA implementation is RoCEv2 which is based on UDP protocol. So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not. If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP). Best Regards, Zhu Yanjun > > kdevops includes workflows for fstests, Mora's nfstest, the > git regression suite, and ltp, all of which we use regularly > to test the Linux NFS client and server implementations. > > >> I am interested in nfs && rdma. >> >> Thanks, >> Zhu Yanjun >> >>> Chuck Lever (4): >>> xprtrdma: Remove temp allocation of rpcrdma_rep objects >>> xprtrdma: Clean up synopsis of frwr_mr_unmap() >>> xprtrdma: Delay releasing connection hardware resources >>> xprtrdma: Move MRs to struct rpcrdma_ep >>> net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- >>> net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- >>> net/sunrpc/xprtrdma/transport.c | 20 +++- >>> net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- >>> net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- >>> 5 files changed, 125 insertions(+), 105 deletions(-) >>> base-commit: e67572cd2204894179d89bd7b984072f19313b03 > -- > Chuck Lever > >
> On Apr 30, 2024, at 9:58 AM, Zhu Yanjun <yanjun.zhu@linux.dev> wrote: > > > On 30.04.24 15:42, Chuck Lever III wrote: >> >>> On Apr 30, 2024, at 3:26 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: >>> >>> On 29.04.24 17:25, cel@kernel.org wrote: >>>> From: Chuck Lever <chuck.lever@oracle.com> >>>> Avoid getting work queue splats in the system journal by moving >>>> client-side RPC/RDMA transport tear-down into a background process. >>>> I've done some testing of this series, now looking for review >>>> comments. >>> How to make tests with nfs && rdma? Can you provide some steps or tools? >> We are building NFS tests into kdevops: >> >> https://github.com/linux-kdevops/kdevops.git >> >> and there is a config option to use soft iWARP instead of TCP. > > Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP? > > If yes, does nfs work well with RXE? I am just curious with nfs && RXE. > > Normally nfs works with TCP. Now nfs will use RDMA instead of TCP. > > The popular RDMA implementation is RoCEv2 which is based on UDP protocol. > > So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not. > > If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP). Yes, NFS/RDMA works with rxe and even with rxe mixed with hardware RoCE. Someone else will have to step in and say whether it works "well" since I don't use rxe, only CX-5 and newer on 100GbE. Generally we use siw because our testing environment varies between all systems on a single local network or hypervisor, all the way up to NFS/RDMA on VPN and WAN. The rxe driver doesn't support operation over tunnels, currently. It is possible to add rxe as a second option in kdevops, but siw has worked for our purposes so far, and the NFS test matrix is already enormous. > Best Regards, > > Zhu Yanjun > >> kdevops includes workflows for fstests, Mora's nfstest, the >> git regression suite, and ltp, all of which we use regularly >> to test the Linux NFS client and server implementations. >> >> >>> I am interested in nfs && rdma. >>> >>> Thanks, >>> Zhu Yanjun >>> >>>> Chuck Lever (4): >>>> xprtrdma: Remove temp allocation of rpcrdma_rep objects >>>> xprtrdma: Clean up synopsis of frwr_mr_unmap() >>>> xprtrdma: Delay releasing connection hardware resources >>>> xprtrdma: Move MRs to struct rpcrdma_ep >>>> net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- >>>> net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- >>>> net/sunrpc/xprtrdma/transport.c | 20 +++- >>>> net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- >>>> net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- >>>> 5 files changed, 125 insertions(+), 105 deletions(-) >>>> base-commit: e67572cd2204894179d89bd7b984072f19313b03 >> -- >> Chuck Lever >> >> > -- > Best Regards, > Yanjun.Zhu -- Chuck Lever
On 30.04.24 16:13, Chuck Lever III wrote: > > >> On Apr 30, 2024, at 9:58 AM, Zhu Yanjun <yanjun.zhu@linux.dev> wrote: >> >> >> On 30.04.24 15:42, Chuck Lever III wrote: >>> >>>> On Apr 30, 2024, at 3:26 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: >>>> >>>> On 29.04.24 17:25, cel@kernel.org wrote: >>>>> From: Chuck Lever <chuck.lever@oracle.com> >>>>> Avoid getting work queue splats in the system journal by moving >>>>> client-side RPC/RDMA transport tear-down into a background process. >>>>> I've done some testing of this series, now looking for review >>>>> comments. >>>> How to make tests with nfs && rdma? Can you provide some steps or tools? >>> We are building NFS tests into kdevops: >>> >>> https://github.com/linux-kdevops/kdevops.git >>> >>> and there is a config option to use soft iWARP instead of TCP. >> >> Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP? >> >> If yes, does nfs work well with RXE? I am just curious with nfs && RXE. >> >> Normally nfs works with TCP. Now nfs will use RDMA instead of TCP. >> >> The popular RDMA implementation is RoCEv2 which is based on UDP protocol. >> >> So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not. >> >> If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP). > > Yes, NFS/RDMA works with rxe and even with rxe mixed with > hardware RoCE. Someone else will have to step in and say > whether it works "well" since I don't use rxe, only CX-5 > and newer on 100GbE. > > Generally we use siw because our testing environment varies > between all systems on a single local network or hypervisor, > all the way up to NFS/RDMA on VPN and WAN. The rxe driver > doesn't support operation over tunnels, currently. Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device? > > It is possible to add rxe as a second option in kdevops, > but siw has worked for our purposes so far, and the NFS > test matrix is already enormous. Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version. Best Regards, Zhu Yanjun > > >> Best Regards, >> >> Zhu Yanjun >> >>> kdevops includes workflows for fstests, Mora's nfstest, the >>> git regression suite, and ltp, all of which we use regularly >>> to test the Linux NFS client and server implementations. >>> >>> >>>> I am interested in nfs && rdma. >>>> >>>> Thanks, >>>> Zhu Yanjun >>>> >>>>> Chuck Lever (4): >>>>> xprtrdma: Remove temp allocation of rpcrdma_rep objects >>>>> xprtrdma: Clean up synopsis of frwr_mr_unmap() >>>>> xprtrdma: Delay releasing connection hardware resources >>>>> xprtrdma: Move MRs to struct rpcrdma_ep >>>>> net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- >>>>> net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- >>>>> net/sunrpc/xprtrdma/transport.c | 20 +++- >>>>> net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- >>>>> net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- >>>>> 5 files changed, 125 insertions(+), 105 deletions(-) >>>>> base-commit: e67572cd2204894179d89bd7b984072f19313b03 >>> -- >>> Chuck Lever >>> >>> >> -- >> Best Regards, >> Yanjun.Zhu > > > -- > Chuck Lever > >
> On Apr 30, 2024, at 10:45 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: > > On 30.04.24 16:13, Chuck Lever III wrote: >>> On Apr 30, 2024, at 9:58 AM, Zhu Yanjun <yanjun.zhu@linux.dev> wrote: >>> >>> >>> On 30.04.24 15:42, Chuck Lever III wrote: >>>> >>>>> On Apr 30, 2024, at 3:26 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: >>>>> >>>>> On 29.04.24 17:25, cel@kernel.org wrote: >>>>>> From: Chuck Lever <chuck.lever@oracle.com> >>>>>> Avoid getting work queue splats in the system journal by moving >>>>>> client-side RPC/RDMA transport tear-down into a background process. >>>>>> I've done some testing of this series, now looking for review >>>>>> comments. >>>>> How to make tests with nfs && rdma? Can you provide some steps or tools? >>>> We are building NFS tests into kdevops: >>>> >>>> https://github.com/linux-kdevops/kdevops.git >>>> >>>> and there is a config option to use soft iWARP instead of TCP. >>> >>> Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP? >>> >>> If yes, does nfs work well with RXE? I am just curious with nfs && RXE. >>> >>> Normally nfs works with TCP. Now nfs will use RDMA instead of TCP. >>> >>> The popular RDMA implementation is RoCEv2 which is based on UDP protocol. >>> >>> So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not. >>> >>> If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP). >> Yes, NFS/RDMA works with rxe and even with rxe mixed with >> hardware RoCE. Someone else will have to step in and say >> whether it works "well" since I don't use rxe, only CX-5 >> and newer on 100GbE. >> Generally we use siw because our testing environment varies >> between all systems on a single local network or hypervisor, >> all the way up to NFS/RDMA on VPN and WAN. The rxe driver >> doesn't support operation over tunnels, currently. > > Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device? No, rxe cannot be configured to use tunnel devices, AFAIK. >> It is possible to add rxe as a second option in kdevops, >> but siw has worked for our purposes so far, and the NFS >> test matrix is already enormous. > > Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version. No new tests are necessary. The only thing missing right now is the ability to set up rxe devices on all the test systems. > Best Regards, > Zhu Yanjun > >>> Best Regards, >>> >>> Zhu Yanjun >>> >>>> kdevops includes workflows for fstests, Mora's nfstest, the >>>> git regression suite, and ltp, all of which we use regularly >>>> to test the Linux NFS client and server implementations. >>>> >>>> >>>>> I am interested in nfs && rdma. >>>>> >>>>> Thanks, >>>>> Zhu Yanjun >>>>> >>>>>> Chuck Lever (4): >>>>>> xprtrdma: Remove temp allocation of rpcrdma_rep objects >>>>>> xprtrdma: Clean up synopsis of frwr_mr_unmap() >>>>>> xprtrdma: Delay releasing connection hardware resources >>>>>> xprtrdma: Move MRs to struct rpcrdma_ep >>>>>> net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- >>>>>> net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- >>>>>> net/sunrpc/xprtrdma/transport.c | 20 +++- >>>>>> net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- >>>>>> net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- >>>>>> 5 files changed, 125 insertions(+), 105 deletions(-) >>>>>> base-commit: e67572cd2204894179d89bd7b984072f19313b03 >>>> -- >>>> Chuck Lever >>>> >>>> >>> -- >>> Best Regards, >>> Yanjun.Zhu >> -- >> Chuck Lever -- Chuck Lever
On 30.04.24 16:52, Chuck Lever III wrote: > >> On Apr 30, 2024, at 10:45 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: >> >> On 30.04.24 16:13, Chuck Lever III wrote: >>>> On Apr 30, 2024, at 9:58 AM, Zhu Yanjun <yanjun.zhu@linux.dev> wrote: >>>> >>>> >>>> On 30.04.24 15:42, Chuck Lever III wrote: >>>>>> On Apr 30, 2024, at 3:26 AM, Zhu Yanjun <zyjzyj2000@gmail.com> wrote: >>>>>> >>>>>> On 29.04.24 17:25, cel@kernel.org wrote: >>>>>>> From: Chuck Lever <chuck.lever@oracle.com> >>>>>>> Avoid getting work queue splats in the system journal by moving >>>>>>> client-side RPC/RDMA transport tear-down into a background process. >>>>>>> I've done some testing of this series, now looking for review >>>>>>> comments. >>>>>> How to make tests with nfs && rdma? Can you provide some steps or tools? >>>>> We are building NFS tests into kdevops: >>>>> >>>>> https://github.com/linux-kdevops/kdevops.git >>>>> >>>>> and there is a config option to use soft iWARP instead of TCP. >>>> Thanks a lot. It is interesting. Have you made tests with RXE instead of iWARP? >>>> >>>> If yes, does nfs work well with RXE? I am just curious with nfs && RXE. >>>> >>>> Normally nfs works with TCP. Now nfs will use RDMA instead of TCP. >>>> >>>> The popular RDMA implementation is RoCEv2 which is based on UDP protocol. >>>> >>>> So I am curious if NFS can work well with RXE (RoCEv2 emulation driver) or not. >>>> >>>> If the user wants to use nfs in his production hosts, it is possible that nfs will work with RoCEv2 (UDP). >>> Yes, NFS/RDMA works with rxe and even with rxe mixed with >>> hardware RoCE. Someone else will have to step in and say >>> whether it works "well" since I don't use rxe, only CX-5 >>> and newer on 100GbE. >>> Generally we use siw because our testing environment varies >>> between all systems on a single local network or hypervisor, >>> all the way up to NFS/RDMA on VPN and WAN. The rxe driver >>> doesn't support operation over tunnels, currently. >> Thanks a lot. "The rxe driver doesn't support operation over tunnels, currently." Do you mean that rxe can not work well with tun/tap device? > No, rxe cannot be configured to use tunnel devices, AFAIK. > > >>> It is possible to add rxe as a second option in kdevops, >>> but siw has worked for our purposes so far, and the NFS >>> test matrix is already enormous. >> Thanks. If rxe can be as a second option in kdevops, I will make tests with kdevops to check rxe work well or not in the future kernel version. > No new tests are necessary. The only thing missing right > now is the ability to set up rxe devices on all the test > systems. Got it. Thanks. Zhu Yanjun > > >> Best Regards, >> Zhu Yanjun >> >>>> Best Regards, >>>> >>>> Zhu Yanjun >>>> >>>>> kdevops includes workflows for fstests, Mora's nfstest, the >>>>> git regression suite, and ltp, all of which we use regularly >>>>> to test the Linux NFS client and server implementations. >>>>> >>>>> >>>>>> I am interested in nfs && rdma. >>>>>> >>>>>> Thanks, >>>>>> Zhu Yanjun >>>>>> >>>>>>> Chuck Lever (4): >>>>>>> xprtrdma: Remove temp allocation of rpcrdma_rep objects >>>>>>> xprtrdma: Clean up synopsis of frwr_mr_unmap() >>>>>>> xprtrdma: Delay releasing connection hardware resources >>>>>>> xprtrdma: Move MRs to struct rpcrdma_ep >>>>>>> net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- >>>>>>> net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- >>>>>>> net/sunrpc/xprtrdma/transport.c | 20 +++- >>>>>>> net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- >>>>>>> net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- >>>>>>> 5 files changed, 125 insertions(+), 105 deletions(-) >>>>>>> base-commit: e67572cd2204894179d89bd7b984072f19313b03 >>>>> -- >>>>> Chuck Lever >>>>> >>>>> >>>> -- >>>> Best Regards, >>>> Yanjun.Zhu >>> -- >>> Chuck Lever > > -- > Chuck Lever > >
From: Chuck Lever <chuck.lever@oracle.com> Avoid getting work queue splats in the system journal by moving client-side RPC/RDMA transport tear-down into a background process. I've done some testing of this series, now looking for review comments. Chuck Lever (4): xprtrdma: Remove temp allocation of rpcrdma_rep objects xprtrdma: Clean up synopsis of frwr_mr_unmap() xprtrdma: Delay releasing connection hardware resources xprtrdma: Move MRs to struct rpcrdma_ep net/sunrpc/xprtrdma/frwr_ops.c | 13 ++- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/transport.c | 20 +++- net/sunrpc/xprtrdma/verbs.c | 173 ++++++++++++++++---------------- net/sunrpc/xprtrdma/xprt_rdma.h | 21 ++-- 5 files changed, 125 insertions(+), 105 deletions(-) base-commit: e67572cd2204894179d89bd7b984072f19313b03