[00/12,RFC,PATCHv2] Add vhost-user Video decode

Message

Albert Esteve March 22, 2023, 2:21 p.m. UTC

v1: https://patchew.org/QEMU/[e-mail]/
v1 -> v2:
    - Address the feedback found in the original patch
    - Rebase patch so that it applies to latest master
    - Add PoC implementation for dmabuf sharing (using udmabuf for creation)
    - Test the patch on a real HW codec

This series adds support for virtio-video decoder devices in Qemu
and also provides a vhost-user-video vmm implementation.

The vhost-user-video vmm currently parses virtio-video v3 protocol
(as that still was what the Linux frontend driver prototype implemented
at the timei, we kept working over this).
It then converts that to a v4l2 mem2mem stateful decoder device.
Currently, this has been tested using v4l2 vicodec test driver in Linux [1],
and also with a real HW codec, i.e., QCom Venus driver, which implements
a v4l2 stateful decoder/encoder. In order to support vicodec driver, the
virtio backend supports the FWHT format.

The primary goal so far has been to allow testing with Qemu. Using
vicodec on the host allows a purely virtual dev env, and for
CI integration in the future by kernelci, etc.

This series adds the virtio_video.h header for testing purposes,
but the driver code has not been upstreamed yet, as the specs
are still under discussion.
Therefore, the commit does not need to be reviewed at this stage.

This version includes a proof of concept implementation of dma-buf
sharing mechanism. Since the infrastructure is not yet available in Qemu,
we employed udmabuf device to create new dma-buf structures when the
driver asks for a shared object memory stream, and then we copy the
contents of the shared memory buffer into the newly created buffer.
Also, note that this PoC was necessary for making the patch work with
QC Venus driver as it does not support USERPTR memory in its latest
release. In any case, a thorough review of these two commits is not
necessary in its current stage, as the implementation is subjected
to big changes.

We tested this VMM using v4l2-ctl from v4l2 utils in the guest
to do a video decode to a file. This can then be validated using ffplay
v4l2-compliance tool in the guest has also been run which stresses the
interface and issues lots of syscall level tests.

See the vhost-user-video.rst for example commands on how to configure
guest kernel and do a video decode using Qemu, vicodec using this VMM.

Next steps:
Recently there was a big update for virtio-video specs being discussed
in the virtio-dev mailing list [2], with a proposal for moving the specs
closer to the v4l2 framework.
Following these news, the next step for this series is to update the
implementation to the new specs approach and join efforts in verifying
the driver implementation [3], and the specs themselves.

Linux virtio-video frontend driver code:
https://github.com/aesteve-rh/linux
(adds FWHT support on top of https://github.com/Gnurou/linux)

Qemu vmm code:
https://github.com/aesteve-rh/qemu/tree/virtio_video_v3

Applies cleanly to:
git://git.qemu.org/qemu.git master(c283ff89d11ff123efc9af49128ef58511f73012)

[1] https://lwn.net/Articles/760650/
[2] https://www.mail-archive.com/[e-mail]/msg09126.html
[3] https://github.com/Gnurou/linux/blob/virtio-v4l2/drivers/media/virtio-v4l2/virtio_v4l2_driver.c

Albert Esteve (5):
  vhost-user.json: add video type
  tests/qtest: add virtio-video test
  vhost-user-video: add dev_type to CLI
  vhost-user-video-udmabuf: add udmabuf helpers
  Add support for v4l2_memory_dmabuf

Peter Griffin (7):
  docs: Add a vhost-virtio-video rst file
  MAINTAINERS: Add virtio-video section
  vhost-user-video: boiler plate code for vhost-user-video device
  vhost-user-video: add meson subdir build logic
  standard-headers: Add virtio_video.h
  hw/display: add vhost-user-video-pci
  tools/vhost-user-video: Add initial vhost-user-video vmm

 MAINTAINERS                                   |    8 +
 docs/interop/vhost-user.json                  |    2 +
 docs/system/devices/vhost-user-video.rst      |  124 ++
 hw/display/Kconfig                            |    5 +
 hw/display/meson.build                        |    3 +
 hw/display/vhost-user-video-pci.c             |   82 +
 hw/display/vhost-user-video.c                 |  419 ++++
 include/hw/virtio/vhost-user-video.h          |   43 +
 include/standard-headers/linux/virtio_video.h |  513 +++++
 tests/qtest/libqos/meson.build                |    1 +
 tests/qtest/libqos/virtio-video.c             |  179 ++
 tests/qtest/libqos/virtio-video.h             |   39 +
 tests/qtest/vhost-user-test.c                 |   32 +
 tools/meson.build                             |    7 +
 tools/vhost-user-video/50-qemu-video.json.in  |    5 +
 tools/vhost-user-video/meson.build            |   10 +
 tools/vhost-user-video/v4l2_backend.c         | 1838 +++++++++++++++++
 tools/vhost-user-video/v4l2_backend.h         |  111 +
 tools/vhost-user-video/vhost-user-video.c     | 1787 ++++++++++++++++
 tools/vhost-user-video/virtio_video_helpers.c |  466 +++++
 tools/vhost-user-video/virtio_video_helpers.h |  198 ++
 tools/vhost-user-video/virtio_video_udmabuf.c |  180 ++
 tools/vhost-user-video/vuvideo.h              |   51 +
 23 files changed, 6103 insertions(+)
 create mode 100644 docs/system/devices/vhost-user-video.rst
 create mode 100644 hw/display/vhost-user-video-pci.c
 create mode 100644 hw/display/vhost-user-video.c
 create mode 100644 include/hw/virtio/vhost-user-video.h
 create mode 100644 include/standard-headers/linux/virtio_video.h
 create mode 100644 tests/qtest/libqos/virtio-video.c
 create mode 100644 tests/qtest/libqos/virtio-video.h
 create mode 100644 tools/vhost-user-video/50-qemu-video.json.in
 create mode 100644 tools/vhost-user-video/meson.build
 create mode 100644 tools/vhost-user-video/v4l2_backend.c
 create mode 100644 tools/vhost-user-video/v4l2_backend.h
 create mode 100644 tools/vhost-user-video/vhost-user-video.c
 create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
 create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
 create mode 100644 tools/vhost-user-video/virtio_video_udmabuf.c
 create mode 100644 tools/vhost-user-video/vuvideo.h