| Message ID | 20240607080045.1337046-1-aesteve@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | virtio-media: Add device specification | expand |
On Fri, Jun 07, 2024 at 10:00:44AM +0200, Albert Esteve wrote: > Hi, > > This a formal attempt of including virtio-media > device specification. This was sent to a wrong mailing list. Please repost to the correct list. Albert, when you subscribed to virtio-dev you should have received a message that read, in part: When to use this list: - questions and change proposals for Virtio drivers and devices implementing the specification. When not to use this list: - questions and change proposals for the Virtio specification, including the specification of basic functionality, transports and devices (please use the virtio-comment mailing list [mailto:virtio-comment@lists.linux.dev] for this). To do: - send email preferably in text format (best for archiving) Not to do: - copy both virtio-dev and virtio-comment mailing lists (instead, pick one) - send full copies of the virtio spec (in any format) So what made you send a specification change proposal to virtio-dev, and also copy the defunct, non-functional list at virtio-dev@lists.oasis-open.org ? Was the message unclear? If yes I'd like to improve it.
On Fri, Jun 07, 2024 at 11:48:36AM +0200, Albert Esteve wrote: > > > On Fri, Jun 7, 2024 at 11:41 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Jun 07, 2024 at 10:00:44AM +0200, Albert Esteve wrote: > > Hi, > > > > This a formal attempt of including virtio-media > > device specification. > > This was sent to a wrong mailing list. > Please repost to the correct list. > > Albert, when you subscribed to virtio-dev you should > have received a message that read, in part: > > When to use this list: > - questions and change proposals for Virtio drivers and devices > implementing the specification. > > When not to use this list: > - questions and change proposals for the Virtio specification, > including the specification of basic functionality, transports and > devices (please use the virtio-comment mailing list > [mailto:virtio-comment@lists.linux.dev] for this). > > To do: > - send email preferably in text format (best for archiving) > > Not to do: > - copy both virtio-dev and virtio-comment mailing lists (instead, pick > one) > - send full copies of the virtio spec (in any format) > > > > So what made you send a specification change proposal to > virtio-dev, and also copy the defunct, non-functional > list at virtio-dev@lists.oasis-open.org ? > > > Was the message unclear? If yes I'd like to improve it. > > > My bad. > The instructions were clear, and I set up the mail to be sent to the > correct virtio-dev mailing list address. Hmm no, it's actually virtio-comment@lists.linux.dev for change proposals for the Virtio specification which is what you have here. > But I had the configuration > of the project set to automatically add both of the old addresses to the > patch. I only noticed after sending, so the damage was already > made. > > I will fix it for the next drop. > > > > > -- > MST > >
Hi Albert, On 07.06.24 10:00, Albert Esteve wrote: > Hi, > > This a formal attempt of including virtio-media > device specification. > > Virtio-media came from a discussion on virtio-dev > mailing list, which lead to presenting virtio-v4l2[1] > specification as an alternative to virtio-video. > > Later, virtio-v4l2 was renamed to virtio-media[2] > and published through: > > https://github.com/chromeos/virtio-media > > The repository above includes a virtio-media driver able > to pass v4l2-compliance when proxying the vivid/vicodec > virtual devices or an actual UVC camera using the > V4L2 vhost device (available in the repository). > Steps to reproduce are also detailed[3]. > > There is some overlap with virtio-video in regards > to which devices it can handle. However, > as virtio-media will likely be the virtualization > solution for ChromeOS (already landed into the chromeos > organization) and possibly other Google projects for > media devices, it would be desirable to include the > specification in the next virtio release despite > the aforementioned overlap. > > The device ID in this document differs from > the ID in the virtio-media project repository. > And it will probably need some discussion on which > would be the correct definitive ID. > > Full PDF: https://drive.google.com/file/d/1pNCFP06VoV8Zuyx0aDVQ7HAT4xp-pZ0a/view?usp=sharing > PDF with the media section only: https://drive.google.com/file/d/1sn3NUUeCm46zVyJKHkpytTIgGw1fUt5T/view?usp=sharing > > [1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0 > [2] https://www.mail-archive.com/virtio-dev@lists.oasis-open.org/msg12665.html > [3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md > > Albert Esteve (1): > virtio-media: Add virtio media device specification > > conformance.tex | 13 +- > content.tex | 1 + > device-types/media/description.tex | 578 ++++++++++++++++++++++ > device-types/media/device-conformance.tex | 11 + > device-types/media/driver-conformance.tex | 9 + > 5 files changed, 608 insertions(+), 4 deletions(-) > create mode 100644 device-types/media/description.tex > create mode 100644 device-types/media/device-conformance.tex > create mode 100644 device-types/media/driver-conformance.tex I'd like to add some general considerations: 1. virtio-media device capability discovery is very chatty V4L2 requires potentially hundreds of system calls to get the full view of the device capabilities. This is inherited by virtio-media. AFAIU V4L2 developers agree there is room for enhancement here, see [1], [2]. Also I've been told VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS were added for similar reasons. From the point of view of virtualization developers like my colleagues and me hundreds of hypervisor "exits" are excessive and costly. It can noticeably increase boot times, which is something we fight against in automotive. AFAIU other virtio developers agree with this, see [3]. In contrast virtio-video has been doing this in one command from day one. Yes, the data was incomplete. That's why in draft v9 I added TLV as a way to store all possible capabilities. I hope the linux-media folks could take a look on it. Maybe this is something, that can be adopted in V4L2? 2. virtio-media has a hard dependency on V4L2 There are certainly some "patches" on top of V4L2 in virtio-media, like the representation of the extended controls (which actually looks similar to the representation of the controls in virtio-video) or VIRTIO_MEDIA_EVT_DQBUF. But how far can this go? Is it OK to add your own ioctl ids in virtio-media? AFAIK the linux-media maintainers have been overloaded for years, see [4]. Would they be happy to deal with the additional requirements? Would virtio community like to have a dependency here? 2.1. an example: format modifiers There is a patchset aiming at unifying V4L2 pixel formats and extending them with DRM format modifiers. It is now at version 7 submitted in 2023, see [5]. The first version was submitted in 2019, see [6]. Not merged yet AFAIK. In virtio-video I just added them in draft v9. I'm absolutely not trying to criticize here. I just try to highlight that there is a lot of legacy and the process is painful. Right now we have an opportunity to design a new API according to the current state of the art of the stateful codecs. 3. uncertainty with cameras AFAIK there is still no agreement about how cameras should be virtualized, see [3], [7], [8], [9]. virtio-media provides support for cameras in a specific way, which might not be desirable. 4. (minor) is it possible/hard to implement the device in hardware/on a micro controller? This is something I thought about recently, there might be a use-case for it in the future. One of the concerns is that dynamic memory allocations are IMHO inevitable in virtio-media or virtio-video up to draft v8. I think multiple virtqueues in virtio-video draft v9 would help here. Not sure yet... There are also other minor concerns, that are probably tolerable. [1] https://lore.kernel.org/linux-media/20230922100303.GF19112@pendragon.ideasonboard.com/ [2] Page 6 in https://hverkuil.home.xs4all.nl/mediasummit2023-pdfs/Hsia-Jun%20Li%20-%20V4L2%20M2M%20EXT%20API%20enhancement.pdf [3] https://old.linaro.org/blog/the-challenges-of-abstracting-virtio/ [4] https://lore.kernel.org/linux-media/4b6b1355-9baa-ff1e-e1c0-89dfdc83ad04@xs4all.nl/ [5] https://lore.kernel.org/linux-media/20230206043308.28365-2-ayaka@soulik.info/ [6] https://lore.kernel.org/linux-media/20190319145243.25047-1-boris.brezillon@collabora.com/ [7] https://lore.kernel.org/virtio-dev/87354dtp30.fsf@linaro.org/ [8] https://lore.kernel.org/linux-media/00f53c06-e66d-aa46-ca4f-c3baab6cf455@xs4all.nl/ [9] https://lore.kernel.org/virtio-dev/CAAFQd5BrhDZtFX3vdhBVSLXthe8CykYsZzVQ9ddZxVNvNS3ArA@mail.gmail.com/
On Fri, Jun 7, 2024 at 10:29 PM Alexander Gordeev <alexander.gordeev@opensynergy.com> wrote: > > Hi Albert, > > On 07.06.24 10:00, Albert Esteve wrote: > > Hi, > > > > This a formal attempt of including virtio-media > > device specification. > > > > Virtio-media came from a discussion on virtio-dev > > mailing list, which lead to presenting virtio-v4l2[1] > > specification as an alternative to virtio-video. > > > > Later, virtio-v4l2 was renamed to virtio-media[2] > > and published through: > > > > https://github.com/chromeos/virtio-media > > > > The repository above includes a virtio-media driver able > > to pass v4l2-compliance when proxying the vivid/vicodec > > virtual devices or an actual UVC camera using the > > V4L2 vhost device (available in the repository). > > Steps to reproduce are also detailed[3]. > > > > There is some overlap with virtio-video in regards > > to which devices it can handle. However, > > as virtio-media will likely be the virtualization > > solution for ChromeOS (already landed into the chromeos > > organization) and possibly other Google projects for > > media devices, it would be desirable to include the > > specification in the next virtio release despite > > the aforementioned overlap. > > > > The device ID in this document differs from > > the ID in the virtio-media project repository. > > And it will probably need some discussion on which > > would be the correct definitive ID. > > > > Full PDF: https://drive.google.com/file/d/1pNCFP06VoV8Zuyx0aDVQ7HAT4xp-pZ0a/view?usp=sharing > > PDF with the media section only: https://drive.google.com/file/d/1sn3NUUeCm46zVyJKHkpytTIgGw1fUt5T/view?usp=sharing > > > > [1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0 > > [2] https://www.mail-archive.com/virtio-dev@lists.oasis-open.org/msg12665.html > > [3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md > > > > Albert Esteve (1): > > virtio-media: Add virtio media device specification > > > > conformance.tex | 13 +- > > content.tex | 1 + > > device-types/media/description.tex | 578 ++++++++++++++++++++++ > > device-types/media/device-conformance.tex | 11 + > > device-types/media/driver-conformance.tex | 9 + > > 5 files changed, 608 insertions(+), 4 deletions(-) > > create mode 100644 device-types/media/description.tex > > create mode 100644 device-types/media/device-conformance.tex > > create mode 100644 device-types/media/driver-conformance.tex > > I'd like to add some general considerations: > > 1. virtio-media device capability discovery is very chatty > > V4L2 requires potentially hundreds of system calls to get the full view > of the device capabilities. This is inherited by virtio-media. > AFAIU V4L2 developers agree there is room for enhancement here, see [1], > [2]. Also I've been told VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS were > added for similar reasons. > > From the point of view of virtualization developers like my colleagues > and me hundreds of hypervisor "exits" are excessive and costly. It can > noticeably increase boot times, which is something we fight against in > automotive. AFAIU other virtio developers agree with this, see [3]. There is a simple way to mitigate this: allow an arbitrary number of commands to be submitted by the driver with each command buffer. The host processes the commands sequentially and stops at the first error, if any. That way, when enumerating formats, the guest can place a bunch of VIDIOC_ENUM_FMT ioctls covering an arbitrary range 0..n in its command buffer, and get all these formats in one vmexit instead of n. In the case of a video decoder supporting 5 input formats, all the formats are enumerated in 5 exits. Having this option could also enable other optimizations, like the guest being able to queue all its CAPTURE buffers in a single vmexit, although I doubt this would result in a noticeable performance boost. But at the very least this should address concerns about noisy enumeration. I haven't noticed any slowness due to format enumeration with the current scheme, but the option is trivial to enable. > > In contrast virtio-video has been doing this in one command from day > one. Yes, the data was incomplete. That's why in draft v9 I added TLV as > a way to store all possible capabilities. I hope the linux-media folks > could take a look on it. Maybe this is something, that can be adopted in > V4L2? > > 2. virtio-media has a hard dependency on V4L2 > > There are certainly some "patches" on top of V4L2 in virtio-media, like > the representation of the extended controls (which actually looks > similar to the representation of the controls in virtio-video) or > VIRTIO_MEDIA_EVT_DQBUF. But how far can this go? Is it OK to add your > own ioctl ids in virtio-media? virtio-media doesn't add any ioctl ID and the extended controls use the same structure as their V4L2 counterparts. > > AFAIK the linux-media maintainers have been overloaded for years, see > [4]. Would they be happy to deal with the additional requirements? Would > virtio community like to have a dependency here? The burden of having a couple new events in virtio-media seems lighter to me on the V4L2 maintainers than the one of having a whole new video protocol to maintain. > > 2.1. an example: format modifiers > > There is a patchset aiming at unifying V4L2 pixel formats and extending > them with DRM format modifiers. It is now at version 7 submitted in > 2023, see [5]. The first version was submitted in 2019, see [6]. Not > merged yet AFAIK. > > In virtio-video I just added them in draft v9. > > I'm absolutely not trying to criticize here. I just try to highlight > that there is a lot of legacy and the process is painful. Right now we > have an opportunity to design a new API according to the current state > of the art of the stateful codecs. > > 3. uncertainty with cameras > > AFAIK there is still no agreement about how cameras should be > virtualized, see [3], [7], [8], [9]. virtio-media provides support for > cameras in a specific way, which might not be desirable. There is definitely an agreement on how cameras should work on V4L2 ; and thanks to that you can operate a camera using virtio-media, *today*. I'm sure the virtio-camera specification, driver, and host devices will be very impressive once they are released ; it's just that I would enjoy a solution, however imperfect, within my life span. > > 4. (minor) is it possible/hard to implement the device in hardware/on a > micro controller? > > This is something I thought about recently, there might be a use-case > for it in the future. One of the concerns is that dynamic memory > allocations are IMHO inevitable in virtio-media or virtio-video up to > draft v8. I think multiple virtqueues in virtio-video draft v9 would > help here. Not sure yet... V4L2's MMAP buffer type has that covered. > > There are also other minor concerns, that are probably tolerable. > > [1] > https://lore.kernel.org/linux-media/20230922100303.GF19112@pendragon.ideasonboard.com/ > [2] Page 6 in > https://hverkuil.home.xs4all.nl/mediasummit2023-pdfs/Hsia-Jun%20Li%20-%20V4L2%20M2M%20EXT%20API%20enhancement.pdf > [3] https://old.linaro.org/blog/the-challenges-of-abstracting-virtio/ > [4] > https://lore.kernel.org/linux-media/4b6b1355-9baa-ff1e-e1c0-89dfdc83ad04@xs4all.nl/ > [5] > https://lore.kernel.org/linux-media/20230206043308.28365-2-ayaka@soulik.info/ > [6] > https://lore.kernel.org/linux-media/20190319145243.25047-1-boris.brezillon@collabora.com/ > [7] https://lore.kernel.org/virtio-dev/87354dtp30.fsf@linaro.org/ > [8] > https://lore.kernel.org/linux-media/00f53c06-e66d-aa46-ca4f-c3baab6cf455@xs4all.nl/ > [9] > https://lore.kernel.org/virtio-dev/CAAFQd5BrhDZtFX3vdhBVSLXthe8CykYsZzVQ9ddZxVNvNS3ArA@mail.gmail.com/ > > -- > Alexander Gordeev > Senior Software Engineer > > OpenSynergy GmbH > Rotherstr. 20, 10245 Berlin > www.opensynergy.com