[v2,0/2] enable fsdax rdma migration

Message

Li Zhijian Aug. 23, 2021, 3:33 a.m. UTC

Previous qemu are facing 2 problems when migrating a fsdax memory backend with
RDMA protocol.
(1) ibv_reg_mr failed with Operation not supported
(2) requester(source) side could receive RNR NAK.

For the (1), we can try to register memory region with ODP feature which
has already been implemented in some modern HCA hardware/drivers.
For the (2), IB provides advise API to prefetch pages in specific memory
region. It can help driver reduce the page fault on responder(destination)
side during RDMA_WRITE.

CC: [e-mail]

Li Zhijian (2):
  migration/rdma: Try to register On-Demand Paging memory region
  migration/rdma: advise prefetch write for ODP region

 migration/rdma.c       | 117 +++++++++++++++++++++++++++++++++--------
 migration/trace-events |   2 +
 2 files changed, 98 insertions(+), 21 deletions(-)

Comments

Zhijian Li (Fujitsu) Aug. 23, 2021, 8:41 a.m. UTC | #1

CCing  Marcel


On 23/08/2021 11:33, Li Zhijian wrote:
> Previous qemu are facing 2 problems when migrating a fsdax memory backend with
> RDMA protocol.
> (1) ibv_reg_mr failed with Operation not supported
> (2) requester(source) side could receive RNR NAK.
>
> For the (1), we can try to register memory region with ODP feature which
> has already been implemented in some modern HCA hardware/drivers.
> For the (2), IB provides advise API to prefetch pages in specific memory
> region. It can help driver reduce the page fault on responder(destination)
> side during RDMA_WRITE.
>
> CC: [e-mail]
>
> Li Zhijian (2):
>    migration/rdma: Try to register On-Demand Paging memory region
>    migration/rdma: advise prefetch write for ODP region
>
>   migration/rdma.c       | 117 +++++++++++++++++++++++++++++++++--------
>   migration/trace-events |   2 +
>   2 files changed, 98 insertions(+), 21 deletions(-)
>

Marcel Apfelbaum Aug. 23, 2021, 8:53 a.m. UTC | #2

Hi Zhijian,

On Mon, Aug 23, 2021 at 11:41 AM [e-mail]
<[e-mail]> wrote:
>
> CCing  Marcel
>
>
> On 23/08/2021 11:33, Li Zhijian wrote:
> > Previous qemu are facing 2 problems when migrating a fsdax memory backend with
> > RDMA protocol.
> > (1) ibv_reg_mr failed with Operation not supported
> > (2) requester(source) side could receive RNR NAK.
> >
> > For the (1), we can try to register memory region with ODP feature which
> > has already been implemented in some modern HCA hardware/drivers.
> > For the (2), IB provides advise API to prefetch pages in specific memory
> > region. It can help driver reduce the page fault on responder(destination)
> > side during RDMA_WRITE.
> >
> > CC: [e-mail]
> >
> > Li Zhijian (2):
> >    migration/rdma: Try to register On-Demand Paging memory region
> >    migration/rdma: advise prefetch write for ODP region
> >
> >   migration/rdma.c       | 117 +++++++++++++++++++++++++++++++++--------
> >   migration/trace-events |   2 +
> >   2 files changed, 98 insertions(+), 21 deletions(-)
> >

Series
Reviewed-by: Marcel Apfelbaum <[e-mail]>

Thanks,
Marcel