[0/2] rdma: Utilize ibv_reg_mr_iova for memory registration

Message

Yuval Shaia Aug. 18, 2019, 1:21 p.m. UTC

The virtual address that is provided by the guest in post_send and
post_recv operations is related to the guest address space. This
address
space is unknown to the HCA resides on host so extra step in these
operations is needed to adjust the address to host virtual address.

This step, which is done in data-path affects performances.

An enhanced verion of MR registration introduced here
https://patchwork.kernel.org/patch/11044467/ can be used so that the
guest virtual address space for this MR is known to the HCA in host.

This will save the data-path adjustment.

patch #1 deals with what is needed to detect if the library installed in
the host supports this function
patch #2 enhance the data-path ops by utilizing the new function

Yuval Shaia (2):
  configure: Check if we can use ibv_reg_mr_iova
  hw/rdma: Utilize ibv_reg_mr_iova for memory registration

 configure                 | 28 ++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.c    | 13 +++++++++++++
 hw/rdma/rdma_backend.h    |  5 +++++
 hw/rdma/rdma_rm.c         |  5 +++++
 hw/rdma/vmw/pvrdma_main.c |  6 ++++++
 5 files changed, 57 insertions(+)

Comments

Marcel Apfelbaum Aug. 31, 2019, 7:26 p.m. UTC | #1

Hi Yuval,

On 8/18/19 4:21 PM, Yuval Shaia wrote:
> The virtual address that is provided by the guest in post_send and
> post_recv operations is related to the guest address space. This
> address
> space is unknown to the HCA resides on host so extra step in these
> operations is needed to adjust the address to host virtual address.
>
> This step, which is done in data-path affects performances.
>
> An enhanced verion of MR registration introduced here
> https://patchwork.kernel.org/patch/11044467/ can be used so that the
> guest virtual address space for this MR is known to the HCA in host.

Nice work on kernel side !
Thanks,
Marcel

>
> This will save the data-path adjustment.
>
> patch #1 deals with what is needed to detect if the library installed in
> the host supports this function
> patch #2 enhance the data-path ops by utilizing the new function
>
> Yuval Shaia (2):
>    configure: Check if we can use ibv_reg_mr_iova
>    hw/rdma: Utilize ibv_reg_mr_iova for memory registration
>
>   configure                 | 28 ++++++++++++++++++++++++++++
>   hw/rdma/rdma_backend.c    | 13 +++++++++++++
>   hw/rdma/rdma_backend.h    |  5 +++++
>   hw/rdma/rdma_rm.c         |  5 +++++
>   hw/rdma/vmw/pvrdma_main.c |  6 ++++++
>   5 files changed, 57 insertions(+)
>

Marcel Apfelbaum Nov. 5, 2019, 11:23 a.m. UTC | #2

Hi Yuval,

On 8/18/19 4:21 PM, Yuval Shaia wrote:
> The virtual address that is provided by the guest in post_send and
> post_recv operations is related to the guest address space. This
> address
> space is unknown to the HCA resides on host so extra step in these
> operations is needed to adjust the address to host virtual address.
>
> This step, which is done in data-path affects performances.
>
> An enhanced verion of MR registration introduced here
> https://patchwork.kernel.org/patch/11044467/ can be used so that the
> guest virtual address space for this MR is known to the HCA in host.
>
> This will save the data-path adjustment.
>
> patch #1 deals with what is needed to detect if the library installed in
> the host supports this function
> patch #2 enhance the data-path ops by utilizing the new function
>
> Yuval Shaia (2):
>    configure: Check if we can use ibv_reg_mr_iova
>    hw/rdma: Utilize ibv_reg_mr_iova for memory registration
>
>   configure                 | 28 ++++++++++++++++++++++++++++
>   hw/rdma/rdma_backend.c    | 13 +++++++++++++
>   hw/rdma/rdma_backend.h    |  5 +++++
>   hw/rdma/rdma_rm.c         |  5 +++++
>   hw/rdma/vmw/pvrdma_main.c |  6 ++++++
>   5 files changed, 57 insertions(+)
>


Can you please rebase on master and add to the patchset your other patches?

Thanks,
Marcel