| Message ID | f298db4ec5fdf7a2d1d166ca2f66020fd9397e5c.1640079962.git.leonro@nvidia.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Delegated to: | Jason Gunthorpe |
| Headers | show |
| Series | [rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs | expand |
On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote: > From: Maor Gottlieb <maorg@nvidia.com> > > The cited commit moved umem into the union, hence > umem could be accessed only for user MRs. Add udata check > before access umem in the dereg flow. > > Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") > Tested-by: Chuck Lever <chuck.lever@oracle.com> > Signed-off-by: Maor Gottlieb <maorg@nvidia.com> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > --- > drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- > drivers/infiniband/hw/mlx5/mr.c | 4 ++-- > drivers/infiniband/hw/mlx5/odp.c | 4 ++-- > 3 files changed, 5 insertions(+), 5 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h This patch was tested and works for me in our environment for SMC. It wouldn't panic when release link and call ib_dereg_mr. Tested-by: Tony Lu <tonylu@linux.alibaba.com> Thanks, Tony Lu
On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote: > On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote: > > From: Maor Gottlieb <maorg@nvidia.com> > > > > The cited commit moved umem into the union, hence > > umem could be accessed only for user MRs. Add udata check > > before access umem in the dereg flow. > > > > Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") > > Tested-by: Chuck Lever <chuck.lever@oracle.com> > > Signed-off-by: Maor Gottlieb <maorg@nvidia.com> > > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > > --- > > drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- > > drivers/infiniband/hw/mlx5/mr.c | 4 ++-- > > drivers/infiniband/hw/mlx5/odp.c | 4 ++-- > > 3 files changed, 5 insertions(+), 5 deletions(-) > > > > diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h > > This patch was tested and works for me in our environment for SMC. It > wouldn't panic when release link and call ib_dereg_mr. > > Tested-by: Tony Lu <tonylu@linux.alibaba.com> Thanks, unfortunately, this patch is incomplete. > > Thanks, > Tony Lu
Hi, this is your Linux kernel regression tracker speaking. On 03.01.22 10:51, Leon Romanovsky wrote: > On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote: >> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote: >>> From: Maor Gottlieb <maorg@nvidia.com> >>> >>> The cited commit moved umem into the union, hence >>> umem could be accessed only for user MRs. Add udata check >>> before access umem in the dereg flow. >>> >>> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") >>> Tested-by: Chuck Lever <chuck.lever@oracle.com> >>> Signed-off-by: Maor Gottlieb <maorg@nvidia.com> >>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com> >>> --- >>> drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- >>> drivers/infiniband/hw/mlx5/mr.c | 4 ++-- >>> drivers/infiniband/hw/mlx5/odp.c | 4 ++-- >>> 3 files changed, 5 insertions(+), 5 deletions(-) >>> >>> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h >> >> This patch was tested and works for me in our environment for SMC. It >> wouldn't panic when release link and call ib_dereg_mr. >> >> Tested-by: Tony Lu <tonylu@linux.alibaba.com> > > Thanks, unfortunately, this patch is incomplete. Could you be a bit more verbose and give a status update? It's hard to follow from the outside. But according to the "Fixes: f0ae4afe3d35" above this was supposed to fix a regression introduced in v5.16-rc5 that was also reported here: https://lore.kernel.org/linux-rdma/9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com/ Commit f0ae4afe3d35 in fact was also backported to v5.15.y and might cause trouble there as well. Should it maybe simply be reverted (and reapplied with all fixes later) in mainline (5.16 will likely be released in 6 days!) and v5.15.y? Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat) P.S.: As a Linux kernel regression tracker I'm getting a lot of reports on my table. I can only look briefly into most of them. Unfortunately therefore I sometimes will get things wrong or miss something important. I hope that's not the case here; if you think it is, don't hesitate to tell me about it in a public reply, that's in everyone's interest. BTW, I have no personal interest in this issue, which is tracked using regzbot, my Linux kernel regression tracking bot (https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting this mail to get things rolling again and hence don't need to be CC on all further activities wrt to this regression.
On Mon, Jan 03, 2022 at 02:15:59PM +0100, Thorsten Leemhuis wrote: > Hi, this is your Linux kernel regression tracker speaking. > > On 03.01.22 10:51, Leon Romanovsky wrote: > > On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote: > >> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote: > >>> From: Maor Gottlieb <maorg@nvidia.com> > >>> > >>> The cited commit moved umem into the union, hence > >>> umem could be accessed only for user MRs. Add udata check > >>> before access umem in the dereg flow. > >>> > >>> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") > >>> Tested-by: Chuck Lever <chuck.lever@oracle.com> > >>> Signed-off-by: Maor Gottlieb <maorg@nvidia.com> > >>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > >>> --- > >>> drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- > >>> drivers/infiniband/hw/mlx5/mr.c | 4 ++-- > >>> drivers/infiniband/hw/mlx5/odp.c | 4 ++-- > >>> 3 files changed, 5 insertions(+), 5 deletions(-) > >>> > >>> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h > >> > >> This patch was tested and works for me in our environment for SMC. It > >> wouldn't panic when release link and call ib_dereg_mr. > >> > >> Tested-by: Tony Lu <tonylu@linux.alibaba.com> > > > > Thanks, unfortunately, this patch is incomplete. > > Could you be a bit more verbose and give a status update? It's hard to > follow from the outside. But according to the "Fixes: f0ae4afe3d35" > above this was supposed to fix a regression introduced in v5.16-rc5 that > was also reported here: > https://lore.kernel.org/linux-rdma/9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com/ The problematic commit f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") should be reverted https://lore.kernel.org/all/20211222101312.1358616-1-maorg@nvidia.com and rewritten from the beginning. There are three possible ways to rewrite that f0ae4afe3d35 commit, while the nutshell of the problem is: "DM MR is user created verb object that is missing very important property (umem) which is needed to distinguish between kernel and user created objects." The proposals are: 1. Return back to v1, which had dummy umem, so so DM memory regions will behave as regular user created verbs object. 2. Add extra flag to is_user/is_kernel for mlx5 mr struct and update all paths to rely on that flag. 3. Create separate dereg MR function that will treat DM differently. We are waiting for Jason to return from vacation and express his preference as he didn't like my preferred option #1. Thanks > > Commit f0ae4afe3d35 in fact was also backported to v5.15.y and might > cause trouble there as well. > > Should it maybe simply be reverted (and reapplied with all fixes later) > in mainline (5.16 will likely be released in 6 days!) and v5.15.y? > > Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat) > > P.S.: As a Linux kernel regression tracker I'm getting a lot of reports > on my table. I can only look briefly into most of them. Unfortunately > therefore I sometimes will get things wrong or miss something important. > I hope that's not the case here; if you think it is, don't hesitate to > tell me about it in a public reply, that's in everyone's interest. > > BTW, I have no personal interest in this issue, which is tracked using > regzbot, my Linux kernel regression tracking bot > (https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting > this mail to get things rolling again and hence don't need to be CC on > all further activities wrt to this regression. >
On Mon, Jan 03, 2022 at 08:26:24PM +0200, Leon Romanovsky wrote: > The proposals are: > 1. Return back to v1, which had dummy umem, so so DM memory regions will > behave as regular user created verbs object. > 2. Add extra flag to is_user/is_kernel for mlx5 mr struct and update all > paths to rely on that flag. > 3. Create separate dereg MR function that will treat DM differently. It is not DM that is the problem, ti is that user and kernel has been mixed together in this mess despite being completely different. I've been slowly disentangling them and the series you just sent 'MR cache enhancment' removes the last blocker from completely giving kernel MRs their own struct. So, the solution here is to move in the direction of making the kernel MRs different. There is only one place that destroys a kernel MR, just have it call a special 'destroy kernel MR' function that doesn't touch any umem stuff at all. Remove the kernel-only parts entirely from the current function. After Aharon's series we can give them different types. Notice the union is already completely disjoint except for the little bit tracking the cache which evaporates once the cache only stores the mkey # and not the struct memory. Jason
On 03.01.22 19:26, Leon Romanovsky wrote: > On Mon, Jan 03, 2022 at 02:15:59PM +0100, Thorsten Leemhuis wrote: >> Hi, this is your Linux kernel regression tracker speaking. >> >> On 03.01.22 10:51, Leon Romanovsky wrote: >>> On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote: >>>> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote: >>>>> From: Maor Gottlieb <maorg@nvidia.com> >>>>> >>>>> The cited commit moved umem into the union, hence >>>>> umem could be accessed only for user MRs. Add udata check >>>>> before access umem in the dereg flow. >>>>> >>>>> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") >>>>> Tested-by: Chuck Lever <chuck.lever@oracle.com> >>>>> Signed-off-by: Maor Gottlieb <maorg@nvidia.com> >>>>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com> >>>>> --- >>>>> drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- >>>>> drivers/infiniband/hw/mlx5/mr.c | 4 ++-- >>>>> drivers/infiniband/hw/mlx5/odp.c | 4 ++-- >>>>> 3 files changed, 5 insertions(+), 5 deletions(-) >>>>> >>>>> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h >>>> >>>> This patch was tested and works for me in our environment for SMC. It >>>> wouldn't panic when release link and call ib_dereg_mr. >>>> >>>> Tested-by: Tony Lu <tonylu@linux.alibaba.com> >>> >>> Thanks, unfortunately, this patch is incomplete. >> >> Could you be a bit more verbose and give a status update? It's hard to >> follow from the outside. But according to the "Fixes: f0ae4afe3d35" >> above this was supposed to fix a regression introduced in v5.16-rc5 that >> was also reported here: >> https://lore.kernel.org/linux-rdma/9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com/ > > The problematic commit f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") > should be reverted https://lore.kernel.org/all/20211222101312.1358616-1-maorg@nvidia.com > and rewritten from the beginning. Thx for the clarification. Is anyone tasked for sending the revert upstream, to make sure the revert makes it into 5.16, which is due on Sunday night? And someone likely should ensure the change backported to 5.15.y as e3bc4d4b50cae7db08e50dbe43f771c906e97701 is reverted as well. CCing a few lists and Greg to make sure everyone is in the loop. Ciao, Thorsten
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 4a7a56ed740b..29d439cebd22 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1296,7 +1296,7 @@ int mlx5_ib_update_mr_pas(struct mlx5_ib_mr *mr, unsigned int flags); struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd, int access_flags); void mlx5_ib_free_implicit_mr(struct mlx5_ib_mr *mr); -void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr); +void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr, struct ib_udata *udata); struct ib_mr *mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start, u64 length, u64 virt_addr, int access_flags, struct ib_pd *pd, struct ib_udata *udata); diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 63e2129f1142..dc833071949f 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1977,7 +1977,7 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) return rc; } - if (mr->umem) { + if (udata && mr->umem) { bool is_odp = is_odp_mr(mr); if (!is_odp) @@ -1985,7 +1985,7 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) &dev->mdev->priv.reg_pages); ib_umem_release(mr->umem); if (is_odp) - mlx5_ib_free_odp_mr(mr); + mlx5_ib_free_odp_mr(mr, udata); } if (mr->cache_ent) { diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 91eb615b89ee..3928576b6696 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -530,7 +530,7 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd, return ERR_PTR(err); } -void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr) +void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr, struct ib_udata *udata) { struct mlx5_ib_mr *mtt; unsigned long idx; @@ -541,7 +541,7 @@ void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr) */ xa_for_each(&mr->implicit_children, idx, mtt) { xa_erase(&mr->implicit_children, idx); - mlx5_ib_dereg_mr(&mtt->ibmr, NULL); + mlx5_ib_dereg_mr(&mtt->ibmr, udata); } }