| Message ID | 20221205170436.2977336-1-eperezma@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vDPA-net inflight descriptors migration with SVQ | expand |
On Tue, Dec 6, 2022 at 1:04 AM Eugenio Pérez <eperezma@redhat.com> wrote: > > The state of the descriptors (avail or used) may not be recoverable just > looking at the guest memory. Out of order used descriptor may override > previous avail ones in the descriptor table or avail vring. > > Currently we're not migrating this status in net devices because virtio-net, > vhost-kernel etc use the descriptors in order, Note that this might not be the truth (when zerocopy is enabled). > so the information always > recoverable from guest's memory. However, vDPA devices may use them out of > order, and other kind of devices like block need this support. > > Shadow virtqueue is able to track these and resend them at the destination. As discussed, there's a bootstrap issue here: When SVQ needs to be enabled on demand, do we still need another way to get inflight ones without the help of SVQ? Thanks > Add them to the virtio-net migration description so they are not lose in the > process. > > This is a very early RFC just to validate the first draft so expect leftovers. > To fetch and request the descriptors from a device without SVQ need to be > implemented on top. Some other notable pending items are: > * Do not send the descriptors actually recoverable from the guest memory. > * Properly version the migrate data. > * Properly abstract the descriptors access from virtio-net to SVQ. > * Do not use VirtQueueElementOld but migrate directly VirtQueueElement. > * Replace lots of assertions with runtime conditionals. > * Other TODOs in the patch message or code changes. > > Thanks. > > Eugenio Pérez (13): > vhost: add available descriptor list in SVQ > vhost: iterate only available descriptors at SVQ stop > vhost: merge avail list and next avail descriptors detach > vhost: add vhost_svq_save_inflight > virtio: Specify uint32_t as VirtQueueElementOld members type > virtio: refactor qemu_get_virtqueue_element > virtio: refactor qemu_put_virtqueue_element > virtio: expose VirtQueueElementOld > virtio: add vmstate_virtqueue_element_old > virtio-net: Migrate vhost inflight descriptors > virtio-net: save inflight descriptors at vhost shutdown > vhost: expose vhost_svq_add_element > vdpa: Recover inflight descriptors > > hw/virtio/vhost-shadow-virtqueue.h | 9 ++ > include/hw/virtio/virtio-net.h | 2 + > include/hw/virtio/virtio.h | 32 ++++++ > include/migration/vmstate.h | 22 ++++ > hw/net/vhost_net.c | 56 ++++++++++ > hw/net/virtio-net.c | 129 +++++++++++++++++++++++ > hw/virtio/vhost-shadow-virtqueue.c | 52 +++++++-- > hw/virtio/vhost-vdpa.c | 11 -- > hw/virtio/virtio.c | 162 ++++++++++++++++++----------- > 9 files changed, 392 insertions(+), 83 deletions(-) > > -- > 2.31.1 > >
On Tue, Dec 6, 2022 at 8:08 AM Jason Wang <jasowang@redhat.com> wrote: > > On Tue, Dec 6, 2022 at 1:04 AM Eugenio Pérez <eperezma@redhat.com> wrote: > > > > The state of the descriptors (avail or used) may not be recoverable just > > looking at the guest memory. Out of order used descriptor may override > > previous avail ones in the descriptor table or avail vring. > > > > Currently we're not migrating this status in net devices because virtio-net, > > vhost-kernel etc use the descriptors in order, > > Note that this might not be the truth (when zerocopy is enabled). > Good point. So will virtio-net wait for those to complete then? How does qemu handle if there are still inflight descriptors? > > so the information always > > recoverable from guest's memory. However, vDPA devices may use them out of > > order, and other kind of devices like block need this support. > > > > Shadow virtqueue is able to track these and resend them at the destination. > > As discussed, there's a bootstrap issue here: > > When SVQ needs to be enabled on demand, do we still need another way > to get inflight ones without the help of SVQ? > To send and retrieve the descriptor without SVQ needs to be developed on top of this. I should have made that more clear here in the cover letter. Thanks! > Thanks > > > Add them to the virtio-net migration description so they are not lose in the > > process. > > > > This is a very early RFC just to validate the first draft so expect leftovers. > > To fetch and request the descriptors from a device without SVQ need to be > > implemented on top. Some other notable pending items are: > > * Do not send the descriptors actually recoverable from the guest memory. > > * Properly version the migrate data. > > * Properly abstract the descriptors access from virtio-net to SVQ. > > * Do not use VirtQueueElementOld but migrate directly VirtQueueElement. > > * Replace lots of assertions with runtime conditionals. > > * Other TODOs in the patch message or code changes. > > > > Thanks. > > > > Eugenio Pérez (13): > > vhost: add available descriptor list in SVQ > > vhost: iterate only available descriptors at SVQ stop > > vhost: merge avail list and next avail descriptors detach > > vhost: add vhost_svq_save_inflight > > virtio: Specify uint32_t as VirtQueueElementOld members type > > virtio: refactor qemu_get_virtqueue_element > > virtio: refactor qemu_put_virtqueue_element > > virtio: expose VirtQueueElementOld > > virtio: add vmstate_virtqueue_element_old > > virtio-net: Migrate vhost inflight descriptors > > virtio-net: save inflight descriptors at vhost shutdown > > vhost: expose vhost_svq_add_element > > vdpa: Recover inflight descriptors > > > > hw/virtio/vhost-shadow-virtqueue.h | 9 ++ > > include/hw/virtio/virtio-net.h | 2 + > > include/hw/virtio/virtio.h | 32 ++++++ > > include/migration/vmstate.h | 22 ++++ > > hw/net/vhost_net.c | 56 ++++++++++ > > hw/net/virtio-net.c | 129 +++++++++++++++++++++++ > > hw/virtio/vhost-shadow-virtqueue.c | 52 +++++++-- > > hw/virtio/vhost-vdpa.c | 11 -- > > hw/virtio/virtio.c | 162 ++++++++++++++++++----------- > > 9 files changed, 392 insertions(+), 83 deletions(-) > > > > -- > > 2.31.1 > > > > >
On Wed, Dec 7, 2022 at 5:00 PM Eugenio Perez Martin <eperezma@redhat.com> wrote: > > On Tue, Dec 6, 2022 at 8:08 AM Jason Wang <jasowang@redhat.com> wrote: > > > > On Tue, Dec 6, 2022 at 1:04 AM Eugenio Pérez <eperezma@redhat.com> wrote: > > > > > > The state of the descriptors (avail or used) may not be recoverable just > > > looking at the guest memory. Out of order used descriptor may override > > > previous avail ones in the descriptor table or avail vring. > > > > > > Currently we're not migrating this status in net devices because virtio-net, > > > vhost-kernel etc use the descriptors in order, > > > > Note that this might not be the truth (when zerocopy is enabled). > > > > Good point. So will virtio-net wait for those to complete then? Vhost-net will wait for all the inflight descriptors to be completed. > How > does qemu handle if there are still inflight descriptors? It doesn't care right now. For networking devices, devices can choose to flush inflight before suspending. But this won't work for other type of device like block. Thanks > > > > so the information always > > > recoverable from guest's memory. However, vDPA devices may use them out of > > > order, and other kind of devices like block need this support. > > > > > > Shadow virtqueue is able to track these and resend them at the destination. > > > > As discussed, there's a bootstrap issue here: > > > > When SVQ needs to be enabled on demand, do we still need another way > > to get inflight ones without the help of SVQ? > > > > To send and retrieve the descriptor without SVQ needs to be developed > on top of this. I should have made that more clear here in the cover > letter. > > Thanks! > > > Thanks > > > > > Add them to the virtio-net migration description so they are not lose in the > > > process. > > > > > > This is a very early RFC just to validate the first draft so expect leftovers. > > > To fetch and request the descriptors from a device without SVQ need to be > > > implemented on top. Some other notable pending items are: > > > * Do not send the descriptors actually recoverable from the guest memory. > > > * Properly version the migrate data. > > > * Properly abstract the descriptors access from virtio-net to SVQ. > > > * Do not use VirtQueueElementOld but migrate directly VirtQueueElement. > > > * Replace lots of assertions with runtime conditionals. > > > * Other TODOs in the patch message or code changes. > > > > > > Thanks. > > > > > > Eugenio Pérez (13): > > > vhost: add available descriptor list in SVQ > > > vhost: iterate only available descriptors at SVQ stop > > > vhost: merge avail list and next avail descriptors detach > > > vhost: add vhost_svq_save_inflight > > > virtio: Specify uint32_t as VirtQueueElementOld members type > > > virtio: refactor qemu_get_virtqueue_element > > > virtio: refactor qemu_put_virtqueue_element > > > virtio: expose VirtQueueElementOld > > > virtio: add vmstate_virtqueue_element_old > > > virtio-net: Migrate vhost inflight descriptors > > > virtio-net: save inflight descriptors at vhost shutdown > > > vhost: expose vhost_svq_add_element > > > vdpa: Recover inflight descriptors > > > > > > hw/virtio/vhost-shadow-virtqueue.h | 9 ++ > > > include/hw/virtio/virtio-net.h | 2 + > > > include/hw/virtio/virtio.h | 32 ++++++ > > > include/migration/vmstate.h | 22 ++++ > > > hw/net/vhost_net.c | 56 ++++++++++ > > > hw/net/virtio-net.c | 129 +++++++++++++++++++++++ > > > hw/virtio/vhost-shadow-virtqueue.c | 52 +++++++-- > > > hw/virtio/vhost-vdpa.c | 11 -- > > > hw/virtio/virtio.c | 162 ++++++++++++++++++----------- > > > 9 files changed, 392 insertions(+), 83 deletions(-) > > > > > > -- > > > 2.31.1 > > > > > > > > >