| Message ID | 20181129214805.2749-24-yuval.shaia@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Add support for RDMA MAD | expand |
On 11/29/18 11:48 PM, Yuval Shaia wrote: > Interface with the device is changed with the addition of support for > MAD packets. > Adjust documentation accordingly. > > While there fix a minor mistake which may lead to think that there is a > relation between using RXE on host and the compatibility with bare-metal > peers. > > Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com> > --- > docs/pvrdma.txt | 126 ++++++++++++++++++++++++++++++++++++++++-------- > 1 file changed, 107 insertions(+), 19 deletions(-) > > diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt > index 5599318159..5175251b47 100644 > --- a/docs/pvrdma.txt > +++ b/docs/pvrdma.txt > @@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for any special guest > modifications. > > While it complies with the VMware device, it can also communicate with bare > -metal RDMA-enabled machines and does not require an RDMA HCA in the host, it > -can work with Soft-RoCE (rxe). > +metal RDMA-enabled machines as peers. > + > +It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe). > > It does not require the whole guest RAM to be pinned allowing memory > over-commit and, even if not implemented yet, migration support will be > @@ -78,29 +79,116 @@ the required RDMA libraries. > > 3. Usage > ======== > + > + > +3.1 VM Memory settings > +====================== > Currently the device is working only with memory backed RAM > and it must be mark as "shared": > -m 1G \ > -object memory-backend-ram,id=mb1,size=1G,share \ > -numa node,memdev=mb1 \ > > -The pvrdma device is composed of two functions: > - - Function 0 is a vmxnet Ethernet Device which is redundant in Guest > - but is required to pass the ibdevice GID using its MAC. > - Examples: > - For an rxe backend using eth0 interface it will use its mac: > - -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC> > - For an SRIOV VF, we take the Ethernet Interface exposed by it: > - -device vmxnet3,multifunction=on,mac=<RoCE eth MAC> > - - Function 1 is the actual device: > - -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port> > - where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) > - Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC. > - The rules of conversion are part of the RoCE spec, but since manual conversion > - is not required, spotting problems is not hard: > - Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a > - MAC: 7c:fe:90:cb:74:3a > - Note the difference between the first byte of the MAC and the GID. > + > +3.2 MAD Multiplexer > +=================== > +MAD Multiplexer is a service that exposes MAD-like interface for VMs in > +order to overcome the limitation where only single entity can register with > +MAD layer to send and receive RDMA-CM MAD packets. > + > +To build rdmacm-mux run > +# make rdmacm-mux > + > +The application accepts 3 command line arguments and exposes a UNIX socket > +to pass control and data to it. > +-d rdma-device-name Name of RDMA device to register with > +-s unix-socket-path Path to unix socket to listen (default /var/run/rdmacm-mux) > +-p rdma-device-port Port number of RDMA device to register with (default 1) > +The final UNIX socket file name is a concatenation of the 3 arguments so > +for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2 > +will be created. > + > +pvrdma requires this service. > + > +Please refer to contrib/rdmacm-mux for more details. > + > + > +3.3 Service exposed by libvirt daemon > +===================================== > +The control over the RDMA device's GID table is done by updating the > +device's Ethernet function addresses. > +Usually the first GID entry is determined by the MAC address, the second by > +the first IPv6 address and the third by the IPv4 address. Other entries can > +be added by adding more IP addresses. The opposite is the same, i.e. > +whenever an address is removed, the corresponding GID entry is removed. > +The process is done by the network and RDMA stacks. Whenever an address is > +added the ib_core driver is notified and calls the device driver add_gid > +function which in turn update the device. > +To support this in pvrdma device the device hooks into the create_bind and > +destroy_bind HW commands triggered by pvrdma driver in guest. > + > +Whenever changed is made to the pvrdma port's GID table a special QMP > +messages is sent to be processed by libvirt to update the address of the > +backend Ethernet device. > + > +pvrdma requires that libvirt service will be up. > + > + > +3.4 PCI devices settings > +======================== > +RoCE device exposes two functions - an Ethernet and RDMA. > +To support it, pvrdma device is composed of two PCI functions, an Ethernet > +device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The > +Ethernet function can be used for other Ethernet purposes such as IP. > + > + > +3.5 Device parameters > +===================== > +- netdev: Specifies the Ethernet device function name on the host for > + example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet > + device used to create it. > +- ibdev: The IB device name on host for example rxe0, mlx5_0 etc. > +- mad-chardev: The name of the MAD multiplexer char device. > +- ibport: In case of multi-port device (such as Mellanox's HCA) this > + specify the port to use. If not set 1 will be used. > +- dev-caps-max-mr-size: The maximum size of MR. > +- dev-caps-max-qp: Maximum number of QPs. > +- dev-caps-max-sge: Maximum number of SGE elements in WR. > +- dev-caps-max-cq: Maximum number of CQs. > +- dev-caps-max-mr: Maximum number of MRs. > +- dev-caps-max-pd: Maximum number of PDs. > +- dev-caps-max-ah: Maximum number of AHs. > + > +Notes: > +- The first 3 parameters are mandatory settings, the rest have their > + defaults. > +- The last 8 parameters (the ones that prefixed by dev-caps) defines the top > + limits but the final values is adjusted by the backend device limitations. > +- netdev can be extracted from ibdev's sysfs > + (/sys/class/infiniband/<ibdev>/device/net/) > + > + > +3.6 Example > +=========== > +Define bridge device with vmxnet3 network backend: > +<interface type='bridge'> > + <mac address='56:b4:44:e9:62:dc'/> > + <source bridge='bridge1'/> > + <model type='vmxnet3'/> > + <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0' multifunction='on'/> > +</interface> > + > +Define pvrdma device: > +<qemu:commandline> > + <qemu:arg value='-object'/> > + <qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/> > + <qemu:arg value='-numa'/> > + <qemu:arg value='node,memdev=mb1'/> > + <qemu:arg value='-chardev'/> > + <qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/> > + <qemu:arg value='-device'/> > + <qemu:arg value='pvrdma,addr=10.1,ibdev=rxe0,netdev=bridge0,mad-chardev=mads'/> > +</qemu:commandline> > > > Reviewed-by: Marcel Apfelbaum<marcel.apfelbaum@gmail.com> Thanks, Marcel
diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt index 5599318159..5175251b47 100644 --- a/docs/pvrdma.txt +++ b/docs/pvrdma.txt @@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for any special guest modifications. While it complies with the VMware device, it can also communicate with bare -metal RDMA-enabled machines and does not require an RDMA HCA in the host, it -can work with Soft-RoCE (rxe). +metal RDMA-enabled machines as peers. + +It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe). It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be @@ -78,29 +79,116 @@ the required RDMA libraries. 3. Usage ======== + + +3.1 VM Memory settings +====================== Currently the device is working only with memory backed RAM and it must be mark as "shared": -m 1G \ -object memory-backend-ram,id=mb1,size=1G,share \ -numa node,memdev=mb1 \ -The pvrdma device is composed of two functions: - - Function 0 is a vmxnet Ethernet Device which is redundant in Guest - but is required to pass the ibdevice GID using its MAC. - Examples: - For an rxe backend using eth0 interface it will use its mac: - -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC> - For an SRIOV VF, we take the Ethernet Interface exposed by it: - -device vmxnet3,multifunction=on,mac=<RoCE eth MAC> - - Function 1 is the actual device: - -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port> - where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) - Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC. - The rules of conversion are part of the RoCE spec, but since manual conversion - is not required, spotting problems is not hard: - Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a - MAC: 7c:fe:90:cb:74:3a - Note the difference between the first byte of the MAC and the GID. + +3.2 MAD Multiplexer +=================== +MAD Multiplexer is a service that exposes MAD-like interface for VMs in +order to overcome the limitation where only single entity can register with +MAD layer to send and receive RDMA-CM MAD packets. + +To build rdmacm-mux run +# make rdmacm-mux + +The application accepts 3 command line arguments and exposes a UNIX socket +to pass control and data to it. +-d rdma-device-name Name of RDMA device to register with +-s unix-socket-path Path to unix socket to listen (default /var/run/rdmacm-mux) +-p rdma-device-port Port number of RDMA device to register with (default 1) +The final UNIX socket file name is a concatenation of the 3 arguments so +for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2 +will be created. + +pvrdma requires this service. + +Please refer to contrib/rdmacm-mux for more details. + + +3.3 Service exposed by libvirt daemon +===================================== +The control over the RDMA device's GID table is done by updating the +device's Ethernet function addresses. +Usually the first GID entry is determined by the MAC address, the second by +the first IPv6 address and the third by the IPv4 address. Other entries can +be added by adding more IP addresses. The opposite is the same, i.e. +whenever an address is removed, the corresponding GID entry is removed. +The process is done by the network and RDMA stacks. Whenever an address is +added the ib_core driver is notified and calls the device driver add_gid +function which in turn update the device. +To support this in pvrdma device the device hooks into the create_bind and +destroy_bind HW commands triggered by pvrdma driver in guest. + +Whenever changed is made to the pvrdma port's GID table a special QMP +messages is sent to be processed by libvirt to update the address of the +backend Ethernet device. + +pvrdma requires that libvirt service will be up. + + +3.4 PCI devices settings +======================== +RoCE device exposes two functions - an Ethernet and RDMA. +To support it, pvrdma device is composed of two PCI functions, an Ethernet +device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The +Ethernet function can be used for other Ethernet purposes such as IP. + + +3.5 Device parameters +===================== +- netdev: Specifies the Ethernet device function name on the host for + example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet + device used to create it. +- ibdev: The IB device name on host for example rxe0, mlx5_0 etc. +- mad-chardev: The name of the MAD multiplexer char device. +- ibport: In case of multi-port device (such as Mellanox's HCA) this + specify the port to use. If not set 1 will be used. +- dev-caps-max-mr-size: The maximum size of MR. +- dev-caps-max-qp: Maximum number of QPs. +- dev-caps-max-sge: Maximum number of SGE elements in WR. +- dev-caps-max-cq: Maximum number of CQs. +- dev-caps-max-mr: Maximum number of MRs. +- dev-caps-max-pd: Maximum number of PDs. +- dev-caps-max-ah: Maximum number of AHs. + +Notes: +- The first 3 parameters are mandatory settings, the rest have their + defaults. +- The last 8 parameters (the ones that prefixed by dev-caps) defines the top + limits but the final values is adjusted by the backend device limitations. +- netdev can be extracted from ibdev's sysfs + (/sys/class/infiniband/<ibdev>/device/net/) + + +3.6 Example +=========== +Define bridge device with vmxnet3 network backend: +<interface type='bridge'> + <mac address='56:b4:44:e9:62:dc'/> + <source bridge='bridge1'/> + <model type='vmxnet3'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0' multifunction='on'/> +</interface> + +Define pvrdma device: +<qemu:commandline> + <qemu:arg value='-object'/> + <qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/> + <qemu:arg value='-numa'/> + <qemu:arg value='node,memdev=mb1'/> + <qemu:arg value='-chardev'/> + <qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/> + <qemu:arg value='-device'/> + <qemu:arg value='pvrdma,addr=10.1,ibdev=rxe0,netdev=bridge0,mad-chardev=mads'/> +</qemu:commandline>
Interface with the device is changed with the addition of support for MAD packets. Adjust documentation accordingly. While there fix a minor mistake which may lead to think that there is a relation between using RXE on host and the compatibility with bare-metal peers. Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com> --- docs/pvrdma.txt | 126 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 107 insertions(+), 19 deletions(-)