| Message ID | 20220628021909.14620-1-ming.qian@nxp.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag | expand |
Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, > user-space should be able to hint decoder > the vb2 only contains codec config header, > but does not contain any frame data. > It's only used for parsing header, and can't be decoded. This is copied from OMX specification. I think we we import this, we should at least refer to the original. > > Current, it's usually used by android. > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > --- > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ > include/uapi/linux/videodev2.h | 2 ++ > 2 files changed, 11 insertions(+) > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst > index 4638ec64db00..acdc4556f4f4 100644 > --- a/Documentation/userspace-api/media/v4l/buffer.rst > +++ b/Documentation/userspace-api/media/v4l/buffer.rst > @@ -607,6 +607,15 @@ Buffer Flags > the format. Any subsequent call to the > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, > but return an ``EPIPE`` error code. > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: > + > + - ``V4L2_BUF_FLAG_CODECCONFIG`` > + - 0x00200000 > + - This flag may be set when the buffer only contains codec config > + header, but does not contain any frame data. Usually the codec config > + header is merged to the next idr frame, with the flag > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will > + split the header and queue it separately. I think the documentation is clear. Now, if a driver uses this, will existing userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ? So we need existing driver to do this when flagged to, and just copy/append when the userland didn't opt-in that feature ? > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: > > - ``V4L2_BUF_FLAG_REQUEST_FD`` > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h > index 5311ac4fde35..8708ef257710 100644 > --- a/include/uapi/linux/videodev2.h > +++ b/include/uapi/linux/videodev2.h > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv) > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 > /* mem2mem encoder/decoder */ > #define V4L2_BUF_FLAG_LAST 0x00100000 > +/* Buffer only contains codec header */ > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 > /* request_fd is valid */ > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 >
> From: Nicolas Dufresne <nicolas@ndufresne.ca> > Sent: 2022年7月4日 23:53 > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org; > hverkuil-cisco@xs4all.nl > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de; > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx > <linux-imx@nxp.com>; linux-media@vger.kernel.org; > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org > Subject: [EXT] Re: [PATCH] media: videobuf2: add > V4L2_BUF_FLAG_CODECCONFIG flag > > Caution: EXT Email > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be > > able to hint decoder the vb2 only contains codec config header, but > > does not contain any frame data. > > It's only used for parsing header, and can't be decoded. > > This is copied from OMX specification. I think we we import this, we should at > least refer to the original. > Hi Nicolas, Do you mean OMX_BUFFERFLAG_CODECCONFIG? I'm sorry that I didn't notice it before. Currently we only encounter this requirement on Android, I'm not sure if it has a reference to omx. And thank you very much for pointing out it. Ming > > > > Current, it's usually used by android. > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > > --- > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ > > include/uapi/linux/videodev2.h | 2 ++ > > 2 files changed, 11 insertions(+) > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst > > b/Documentation/userspace-api/media/v4l/buffer.rst > > index 4638ec64db00..acdc4556f4f4 100644 > > --- a/Documentation/userspace-api/media/v4l/buffer.rst > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst > > @@ -607,6 +607,15 @@ Buffer Flags > > the format. Any subsequent call to the > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, > > but return an ``EPIPE`` error code. > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: > > + > > + - ``V4L2_BUF_FLAG_CODECCONFIG`` > > + - 0x00200000 > > + - This flag may be set when the buffer only contains codec config > > + header, but does not contain any frame data. Usually the codec config > > + header is merged to the next idr frame, with the flag > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will > > + split the header and queue it separately. > > I think the documentation is clear. Now, if a driver uses this, will existing > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will > break ? > So we need existing driver to do this when flagged to, and just copy/append > when the userland didn't opt-in that feature ? > > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` > > diff --git a/include/uapi/linux/videodev2.h > > b/include/uapi/linux/videodev2.h index 5311ac4fde35..8708ef257710 > > 100644 > > --- a/include/uapi/linux/videodev2.h > > +++ b/include/uapi/linux/videodev2.h > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const > struct timeval *tv) > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 > > /* mem2mem encoder/decoder */ > > #define V4L2_BUF_FLAG_LAST 0x00100000 > > +/* Buffer only contains codec header */ > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 > > /* request_fd is valid */ > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 > >
>>From: Ming Qian >>Sent: 2022年7月5日 9:52 >>To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org; >>hverkuil-cisco@xs4all.nl >>Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de; >>kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx >><linux-imx@nxp.com>; linux-media@vger.kernel.org; >>linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org >>Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add >>V4L2_BUF_FLAG_CODECCONFIG flag >> >>> From: Nicolas Dufresne <nicolas@ndufresne.ca> >>> Sent: 2022年7月4日 23:53 >>> To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org; >>> hverkuil-cisco@xs4all.nl >>> Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de; >>> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx >>> <linux-imx@nxp.com>; linux-media@vger.kernel.org; >>> linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org >>> Subject: [EXT] Re: [PATCH] media: videobuf2: add >>> V4L2_BUF_FLAG_CODECCONFIG flag >>> >>> Caution: EXT Email >>> >>> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : >>> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be >>> > able to hint decoder the vb2 only contains codec config header, but >>> > does not contain any frame data. >>> > It's only used for parsing header, and can't be decoded. >>> >>> This is copied from OMX specification. I think we we import this, we >>> should at least refer to the original. >>> >> >>Hi Nicolas, >> Do you mean OMX_BUFFERFLAG_CODECCONFIG? >> I'm sorry that I didn't notice it before. >> Currently we only encounter this requirement on Android, I'm not sure if >>it has a reference to omx. >> And thank you very much for pointing out it. >> > > Android media stack has been based on OMX for the last decade. They are slowly moving to CODEC2 which more or less is a similar abstraction with similar ideas. Let's research prior art, so we don't screw compatibility. > I got it, I'll try to study the android codec2, and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like OMX_BUFFERFLAG_CODECCONFIG? Or is there any other solution that can handle this case? >>Ming >> >>> > >>> > Current, it's usually used by android. >>> > >>> > Signed-off-by: Ming Qian <ming.qian@nxp.com> >>> > --- >>> > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ >>> > include/uapi/linux/videodev2.h | 2 ++ >>> > 2 files changed, 11 insertions(+) >>> > >>> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst >>> > b/Documentation/userspace-api/media/v4l/buffer.rst >>> > index 4638ec64db00..acdc4556f4f4 100644 >>> > --- a/Documentation/userspace-api/media/v4l/buffer.rst >>> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst >>> > @@ -607,6 +607,15 @@ Buffer Flags >>> > the format. Any subsequent call to the >>> > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, >>> > but return an ``EPIPE`` error code. >>> > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: >>> > + >>> > + - ``V4L2_BUF_FLAG_CODECCONFIG`` >>> > + - 0x00200000 >>> > + - This flag may be set when the buffer only contains codec config >>> > + header, but does not contain any frame data. Usually the codec >>config >>> > + header is merged to the next idr frame, with the flag >>> > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that >>will >>> > + split the header and queue it separately. >>> >>> I think the documentation is clear. Now, if a driver uses this, will >>> existing userland (perhaps good to check GStreamer, FFMPEG and >>> Chromium ?) will break ? >>> So we need existing driver to do this when flagged to, and just >>> copy/append when the userland didn't opt-in that feature ? >>> >>> > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: >>> > >>> > - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git >>> > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h >>> > index 5311ac4fde35..8708ef257710 >>> > 100644 >>> > --- a/include/uapi/linux/videodev2.h >>> > +++ b/include/uapi/linux/videodev2.h >>> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const >>> struct timeval *tv) >>> > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 >>> > /* mem2mem encoder/decoder */ >>> > #define V4L2_BUF_FLAG_LAST 0x00100000 >>> > +/* Buffer only contains codec header */ >>> > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 >>> > /* request_fd is valid */ >>> > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 >>> >
Hi Ming and Nicolas On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote: > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, > > user-space should be able to hint decoder > > the vb2 only contains codec config header, > > but does not contain any frame data. > > It's only used for parsing header, and can't be decoded. > > This is copied from OMX specification. I think we we import this, we should at > least refer to the original. > > > > > Current, it's usually used by android. > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > > --- > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ > > include/uapi/linux/videodev2.h | 2 ++ > > 2 files changed, 11 insertions(+) > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst > > index 4638ec64db00..acdc4556f4f4 100644 > > --- a/Documentation/userspace-api/media/v4l/buffer.rst > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst > > @@ -607,6 +607,15 @@ Buffer Flags > > the format. Any subsequent call to the > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, > > but return an ``EPIPE`` error code. > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: > > + > > + - ``V4L2_BUF_FLAG_CODECCONFIG`` > > + - 0x00200000 > > + - This flag may be set when the buffer only contains codec config > > + header, but does not contain any frame data. Usually the codec config > > + header is merged to the next idr frame, with the flag > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will > > + split the header and queue it separately. > > I think the documentation is clear. Now, if a driver uses this, will existing > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ? > So we need existing driver to do this when flagged to, and just copy/append when > the userland didn't opt-in that feature ? The commit text says it is for userspace feeding data into a video decoder, so it's a userspace choice instead of driver. For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1] which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects _SEPARATE by default [2], whilst the default is normally _JOINED_WITH_1ST_FRAME. It does raise the question as to whether all decoders will support header byte only buffers, and does there need to be a capabilities flag to denote that it is supported. And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set it on the headers only buffers? A number of undefined elements of how this should be implemented/used :-( Dave [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196 > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h > > index 5311ac4fde35..8708ef257710 100644 > > --- a/include/uapi/linux/videodev2.h > > +++ b/include/uapi/linux/videodev2.h > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv) > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 > > /* mem2mem encoder/decoder */ > > #define V4L2_BUF_FLAG_LAST 0x00100000 > > +/* Buffer only contains codec header */ > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 > > /* request_fd is valid */ > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 > > >
Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit : > > > From: Ming Qian > > > Sent: 2022年7月5日 9:52 > > > To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org; > > > hverkuil-cisco@xs4all.nl > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de; > > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx > > > <linux-imx@nxp.com>; linux-media@vger.kernel.org; > > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org > > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add > > > V4L2_BUF_FLAG_CODECCONFIG flag > > > > > > > From: Nicolas Dufresne <nicolas@ndufresne.ca> > > > > Sent: 2022年7月4日 23:53 > > > > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org; > > > > hverkuil-cisco@xs4all.nl > > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de; > > > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx > > > > <linux-imx@nxp.com>; linux-media@vger.kernel.org; > > > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org > > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add > > > > V4L2_BUF_FLAG_CODECCONFIG flag > > > > > > > > Caution: EXT Email > > > > > > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : > > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be > > > > > able to hint decoder the vb2 only contains codec config header, but > > > > > does not contain any frame data. > > > > > It's only used for parsing header, and can't be decoded. > > > > > > > > This is copied from OMX specification. I think we we import this, we > > > > should at least refer to the original. > > > > > > > > > > Hi Nicolas, > > > Do you mean OMX_BUFFERFLAG_CODECCONFIG? > > > I'm sorry that I didn't notice it before. > > > Currently we only encounter this requirement on Android, I'm not sure > > > if > > > it has a reference to omx. > > > And thank you very much for pointing out it. > > > > > > > Android media stack has been based on OMX for the last decade. They are > > slowly moving to CODEC2 which more or less is a similar abstraction with > > similar ideas. Let's research prior art, so we don't screw compatibility. > > > > I got it, I'll try to study the android codec2, > and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like > OMX_BUFFERFLAG_CODECCONFIG? > Or is there any other solution that can handle this case? We can probably discuss the name. CODECCONFIG is a bit strange, could be CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines. I'm just wondering what is the best rule, since more specification is needed here. Current userland expect full frames into each encoded buffer. If we start splitting these up, we'll break non-android userland (and Android userland does not seems to be very upstream thing, every vendor forks it). I also think that what the CODECCONFIG contains and its format need to be strictly documented for every CODEC that would allow it. In H.264 notably, the headers could be packed in Annex B. or AVCc NAL headers. If we look at FFMPEG, which uses codec_data name, they only requires this when the header are not "inline", which means only for AVCc. Also, many codec_data is for other codecs get wrapped into ISOMP4 or Matroska (webm) envelope. On the other hand, I don't remember if the 1 frame per buffer is an actual rule or simply what existing userland expect. Also, I'll be fair, there is no reason this must come from the driver. Android OMX or CODEC2 COMPONENT is a userland component, it could do bitstream scanning (using traditional boyer-more) to find and split appart the config to satisfy its internal API. This can be done with low overhead and zero-copy. > > > > Ming > > > > > > > > > > > > > Current, it's usually used by android. > > > > > > > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > > > > > --- > > > > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ > > > > > include/uapi/linux/videodev2.h | 2 ++ > > > > > 2 files changed, 11 insertions(+) > > > > > > > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst > > > > > b/Documentation/userspace-api/media/v4l/buffer.rst > > > > > index 4638ec64db00..acdc4556f4f4 100644 > > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst > > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst > > > > > @@ -607,6 +607,15 @@ Buffer Flags > > > > > the format. Any subsequent call to the > > > > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, > > > > > but return an ``EPIPE`` error code. > > > > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: > > > > > + > > > > > + - ``V4L2_BUF_FLAG_CODECCONFIG`` > > > > > + - 0x00200000 > > > > > + - This flag may be set when the buffer only contains codec > > > > > config > > > > > + header, but does not contain any frame data. Usually the codec > > > config > > > > > + header is merged to the next idr frame, with the flag > > > > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that > > > will > > > > > + split the header and queue it separately. > > > > > > > > I think the documentation is clear. Now, if a driver uses this, will > > > > existing userland (perhaps good to check GStreamer, FFMPEG and > > > > Chromium ?) will break ? > > > > So we need existing driver to do this when flagged to, and just > > > > copy/append when the userland didn't opt-in that feature ? > > > > > > > > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: > > > > > > > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git > > > > > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h > > > > > index 5311ac4fde35..8708ef257710 > > > > > 100644 > > > > > --- a/include/uapi/linux/videodev2.h > > > > > +++ b/include/uapi/linux/videodev2.h > > > > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const > > > > struct timeval *tv) > > > > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 > > > > > /* mem2mem encoder/decoder */ > > > > > #define V4L2_BUF_FLAG_LAST 0x00100000 > > > > > +/* Buffer only contains codec header */ > > > > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 > > > > > /* request_fd is valid */ > > > > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 > > > > > >
Le mardi 05 juillet 2022 à 13:34 +0100, Dave Stevenson a écrit : > Hi Ming and Nicolas > > On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <nicolas@ndufresne.ca> wrote: > > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, > > > user-space should be able to hint decoder > > > the vb2 only contains codec config header, > > > but does not contain any frame data. > > > It's only used for parsing header, and can't be decoded. > > > > This is copied from OMX specification. I think we we import this, we should at > > least refer to the original. > > > > > > > > Current, it's usually used by android. > > > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > > > --- > > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ > > > include/uapi/linux/videodev2.h | 2 ++ > > > 2 files changed, 11 insertions(+) > > > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst > > > index 4638ec64db00..acdc4556f4f4 100644 > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst > > > @@ -607,6 +607,15 @@ Buffer Flags > > > the format. Any subsequent call to the > > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, > > > but return an ``EPIPE`` error code. > > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: > > > + > > > + - ``V4L2_BUF_FLAG_CODECCONFIG`` > > > + - 0x00200000 > > > + - This flag may be set when the buffer only contains codec config > > > + header, but does not contain any frame data. Usually the codec config > > > + header is merged to the next idr frame, with the flag > > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will > > > + split the header and queue it separately. > > > > I think the documentation is clear. Now, if a driver uses this, will existing > > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ? > > So we need existing driver to do this when flagged to, and just copy/append when > > the userland didn't opt-in that feature ? > > The commit text says it is for userspace feeding data into a video > decoder, so it's a userspace choice instead of driver. I see, the spec needs to be more clear then. > > For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1] > which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or > V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects > _SEPARATE by default [2], whilst the default is normally > _JOINED_WITH_1ST_FRAME. I did miss the addition of this API, thanks for the reminder. The problem right now is that things are being added with the needed cross-reference. > > It does raise the question as to whether all decoders will support > header byte only buffers, and does there need to be a capabilities > flag to denote that it is supported. > And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set > it on the headers only buffers? What about: - Document better that in absence of V4L2_CID_MPEG_VIDEO_HEADER_MODE in a driver, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME is to be assumed - Document that in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, driver should signal the header buffer with the newly added buffer flag (I would suggest V4L2_BUF_FLAG_HEADERS_ONLY) - Document for each support CODECs if V4L2_CID_MPEG_VIDEO_HEADER_MODE can be supported, and the expected headers and the packing of these - Document that decoders that didn't implement V4L2_CID_MPEG_VIDEO_HEADER_MODE should be assumed to only support V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME decoding (even though it allowed to support any alignment, even random). - Cross-reference V4L2_CID_MPEG_VIDEO_HEADER_MODE into the new flag documentation (whatever this new flag will be called). > > A number of undefined elements of how this should be implemented/used :-( > > Dave > > [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html > [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196 > > > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: > > > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` > > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h > > > index 5311ac4fde35..8708ef257710 100644 > > > --- a/include/uapi/linux/videodev2.h > > > +++ b/include/uapi/linux/videodev2.h > > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv) > > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 > > > /* mem2mem encoder/decoder */ > > > #define V4L2_BUF_FLAG_LAST 0x00100000 > > > +/* Buffer only contains codec header */ > > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 > > > /* request_fd is valid */ > > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 > > > > >
Hi Ming, I love your patch! Perhaps something to improve: [auto build test WARNING on media-tree/master] [also build test WARNING on linus/master v5.19-rc5 next-20220705] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/intel-lab-lkp/linux/commits/Ming-Qian/media-videobuf2-add-V4L2_BUF_FLAG_CODECCONFIG-flag/20220628-103906 base: git://linuxtv.org/media_tree.git master reproduce: make htmldocs If you fix the issue, kindly add following tag where applicable Reported-by: kernel test robot <lkp@intel.com> All warnings (new ones prefixed by >>): >> Documentation/userspace-api/media/v4l/buffer.rst:614: WARNING: Bullet list ends without a blank line; unexpected unindent. >> Documentation/userspace-api/media/v4l/buffer.rst:623: WARNING: Block quote ends without a blank line; unexpected unindent. >> Documentation/userspace-api/media/v4l/buffer.rst:460: WARNING: Error parsing content block for the "flat-table" directive: exactly one bullet list expected. vim +614 Documentation/userspace-api/media/v4l/buffer.rst > 460 461 .. flat-table:: 462 :header-rows: 0 463 :stub-columns: 0 464 :widths: 65 18 70 465 466 * .. _`V4L2-BUF-FLAG-MAPPED`: 467 468 - ``V4L2_BUF_FLAG_MAPPED`` 469 - 0x00000001 470 - The buffer resides in device memory and has been mapped into the 471 application's address space, see :ref:`mmap` for details. 472 Drivers set or clear this flag when the 473 :ref:`VIDIOC_QUERYBUF`, 474 :ref:`VIDIOC_QBUF` or 475 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Set by the 476 driver. 477 * .. _`V4L2-BUF-FLAG-QUEUED`: 478 479 - ``V4L2_BUF_FLAG_QUEUED`` 480 - 0x00000002 481 - Internally drivers maintain two buffer queues, an incoming and 482 outgoing queue. When this flag is set, the buffer is currently on 483 the incoming queue. It automatically moves to the outgoing queue 484 after the buffer has been filled (capture devices) or displayed 485 (output devices). Drivers set or clear this flag when the 486 ``VIDIOC_QUERYBUF`` ioctl is called. After (successful) calling 487 the ``VIDIOC_QBUF``\ ioctl it is always set and after 488 ``VIDIOC_DQBUF`` always cleared. 489 * .. _`V4L2-BUF-FLAG-DONE`: 490 491 - ``V4L2_BUF_FLAG_DONE`` 492 - 0x00000004 493 - When this flag is set, the buffer is currently on the outgoing 494 queue, ready to be dequeued from the driver. Drivers set or clear 495 this flag when the ``VIDIOC_QUERYBUF`` ioctl is called. After 496 calling the ``VIDIOC_QBUF`` or ``VIDIOC_DQBUF`` it is always 497 cleared. Of course a buffer cannot be on both queues at the same 498 time, the ``V4L2_BUF_FLAG_QUEUED`` and ``V4L2_BUF_FLAG_DONE`` flag 499 are mutually exclusive. They can be both cleared however, then the 500 buffer is in "dequeued" state, in the application domain so to 501 say. 502 * .. _`V4L2-BUF-FLAG-ERROR`: 503 504 - ``V4L2_BUF_FLAG_ERROR`` 505 - 0x00000040 506 - When this flag is set, the buffer has been dequeued successfully, 507 although the data might have been corrupted. This is recoverable, 508 streaming may continue as normal and the buffer may be reused 509 normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is 510 called. 511 * .. _`V4L2-BUF-FLAG-IN-REQUEST`: 512 513 - ``V4L2_BUF_FLAG_IN_REQUEST`` 514 - 0x00000080 515 - This buffer is part of a request that hasn't been queued yet. 516 * .. _`V4L2-BUF-FLAG-KEYFRAME`: 517 518 - ``V4L2_BUF_FLAG_KEYFRAME`` 519 - 0x00000008 520 - Drivers set or clear this flag when calling the ``VIDIOC_DQBUF`` 521 ioctl. It may be set by video capture devices when the buffer 522 contains a compressed image which is a key frame (or field), i. e. 523 can be decompressed on its own. Also known as an I-frame. 524 Applications can set this bit when ``type`` refers to an output 525 stream. 526 * .. _`V4L2-BUF-FLAG-PFRAME`: 527 528 - ``V4L2_BUF_FLAG_PFRAME`` 529 - 0x00000010 530 - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags predicted frames 531 or fields which contain only differences to a previous key frame. 532 Applications can set this bit when ``type`` refers to an output 533 stream. 534 * .. _`V4L2-BUF-FLAG-BFRAME`: 535 536 - ``V4L2_BUF_FLAG_BFRAME`` 537 - 0x00000020 538 - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags a bi-directional 539 predicted frame or field which contains only the differences 540 between the current frame and both the preceding and following key 541 frames to specify its content. Applications can set this bit when 542 ``type`` refers to an output stream. 543 * .. _`V4L2-BUF-FLAG-TIMECODE`: 544 545 - ``V4L2_BUF_FLAG_TIMECODE`` 546 - 0x00000100 547 - The ``timecode`` field is valid. Drivers set or clear this flag 548 when the ``VIDIOC_DQBUF`` ioctl is called. Applications can set 549 this bit and the corresponding ``timecode`` structure when 550 ``type`` refers to an output stream. 551 * .. _`V4L2-BUF-FLAG-PREPARED`: 552 553 - ``V4L2_BUF_FLAG_PREPARED`` 554 - 0x00000400 555 - The buffer has been prepared for I/O and can be queued by the 556 application. Drivers set or clear this flag when the 557 :ref:`VIDIOC_QUERYBUF`, 558 :ref:`VIDIOC_PREPARE_BUF <VIDIOC_QBUF>`, 559 :ref:`VIDIOC_QBUF` or 560 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. 561 * .. _`V4L2-BUF-FLAG-NO-CACHE-INVALIDATE`: 562 563 - ``V4L2_BUF_FLAG_NO_CACHE_INVALIDATE`` 564 - 0x00000800 565 - Caches do not have to be invalidated for this buffer. Typically 566 applications shall use this flag if the data captured in the 567 buffer is not going to be touched by the CPU, instead the buffer 568 will, probably, be passed on to a DMA-capable hardware unit for 569 further processing or output. This flag is ignored unless the 570 queue is used for :ref:`memory mapping <mmap>` streaming I/O and 571 reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS 572 <V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability. 573 * .. _`V4L2-BUF-FLAG-NO-CACHE-CLEAN`: 574 575 - ``V4L2_BUF_FLAG_NO_CACHE_CLEAN`` 576 - 0x00001000 577 - Caches do not have to be cleaned for this buffer. Typically 578 applications shall use this flag for output buffers if the data in 579 this buffer has not been created by the CPU but by some 580 DMA-capable unit, in which case caches have not been used. This flag 581 is ignored unless the queue is used for :ref:`memory mapping <mmap>` 582 streaming I/O and reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS 583 <V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability. 584 * .. _`V4L2-BUF-FLAG-M2M-HOLD-CAPTURE-BUF`: 585 586 - ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` 587 - 0x00000200 588 - Only valid if struct :c:type:`v4l2_requestbuffers` flag ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` is 589 set. It is typically used with stateless decoders where multiple 590 output buffers each decode to a slice of the decoded frame. 591 Applications can set this flag when queueing the output buffer 592 to prevent the driver from dequeueing the capture buffer after 593 the output buffer has been decoded (i.e. the capture buffer is 594 'held'). If the timestamp of this output buffer differs from that 595 of the previous output buffer, then that indicates the start of a 596 new frame and the previously held capture buffer is dequeued. 597 * .. _`V4L2-BUF-FLAG-LAST`: 598 599 - ``V4L2_BUF_FLAG_LAST`` 600 - 0x00100000 601 - Last buffer produced by the hardware. mem2mem codec drivers set 602 this flag on the capture queue for the last buffer when the 603 :ref:`VIDIOC_QUERYBUF` or 604 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Due to 605 hardware limitations, the last buffer may be empty. In this case 606 the driver will set the ``bytesused`` field to 0, regardless of 607 the format. Any subsequent call to the 608 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, 609 but return an ``EPIPE`` error code. 610 * .. _`V4L2-BUF-FLAG-CODECCONFIG`: 611 612 - ``V4L2_BUF_FLAG_CODECCONFIG`` 613 - 0x00200000 > 614 - This flag may be set when the buffer only contains codec config 615 header, but does not contain any frame data. Usually the codec config 616 header is merged to the next idr frame, with the flag 617 ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will 618 split the header and queue it separately. 619 * .. _`V4L2-BUF-FLAG-REQUEST-FD`: 620 621 - ``V4L2_BUF_FLAG_REQUEST_FD`` 622 - 0x00800000 > 623 - The ``request_fd`` field contains a valid file descriptor. 624 * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`: 625 626 - ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` 627 - 0x0000e000 628 - Mask for timestamp types below. To test the timestamp type, mask 629 out bits not belonging to timestamp type by performing a logical 630 and operation with buffer flags and timestamp mask. 631 * .. _`V4L2-BUF-FLAG-TIMESTAMP-UNKNOWN`: 632 633 - ``V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN`` 634 - 0x00000000 635 - Unknown timestamp type. This type is used by drivers before Linux 636 3.9 and may be either monotonic (see below) or realtime (wall 637 clock). Monotonic clock has been favoured in embedded systems 638 whereas most of the drivers use the realtime clock. Either kinds 639 of timestamps are available in user space via 640 :c:func:`clock_gettime` using clock IDs ``CLOCK_MONOTONIC`` 641 and ``CLOCK_REALTIME``, respectively. 642 * .. _`V4L2-BUF-FLAG-TIMESTAMP-MONOTONIC`: 643 644 - ``V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC`` 645 - 0x00002000 646 - The buffer timestamp has been taken from the ``CLOCK_MONOTONIC`` 647 clock. To access the same clock outside V4L2, use 648 :c:func:`clock_gettime`. 649 * .. _`V4L2-BUF-FLAG-TIMESTAMP-COPY`: 650 651 - ``V4L2_BUF_FLAG_TIMESTAMP_COPY`` 652 - 0x00004000 653 - The CAPTURE buffer timestamp has been taken from the corresponding 654 OUTPUT buffer. This flag applies only to mem2mem devices. 655 * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-MASK`: 656 657 - ``V4L2_BUF_FLAG_TSTAMP_SRC_MASK`` 658 - 0x00070000 659 - Mask for timestamp sources below. The timestamp source defines the 660 point of time the timestamp is taken in relation to the frame. 661 Logical 'and' operation between the ``flags`` field and 662 ``V4L2_BUF_FLAG_TSTAMP_SRC_MASK`` produces the value of the 663 timestamp source. Applications must set the timestamp source when 664 ``type`` refers to an output stream and 665 ``V4L2_BUF_FLAG_TIMESTAMP_COPY`` is set. 666 * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-EOF`: 667 668 - ``V4L2_BUF_FLAG_TSTAMP_SRC_EOF`` 669 - 0x00000000 670 - End Of Frame. The buffer timestamp has been taken when the last 671 pixel of the frame has been received or the last pixel of the 672 frame has been transmitted. In practice, software generated 673 timestamps will typically be read from the clock a small amount of 674 time after the last pixel has been received or transmitten, 675 depending on the system and other activity in it. 676 * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-SOE`: 677 678 - ``V4L2_BUF_FLAG_TSTAMP_SRC_SOE`` 679 - 0x00010000 680 - Start Of Exposure. The buffer timestamp has been taken when the 681 exposure of the frame has begun. This is only valid for the 682 ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type. 683
>From: Nicolas Dufresne <nicolas@ndufresne.ca> >Sent: 2022年7月5日 21:13 >To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org; >hverkuil-cisco@xs4all.nl >Cc: shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de; >kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx ><linux-imx@nxp.com>; linux-media@vger.kernel.org; >linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org >Subject: Re: [EXT] Re: [PATCH] media: videobuf2: add >V4L2_BUF_FLAG_CODECCONFIG flag > >Caution: EXT Email > >Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit : >> > > From: Ming Qian >> > > Sent: 2022年7月5日 9:52 >> > > To: Nicolas Dufresne <nicolas@ndufresne.ca>; mchehab@kernel.org; >> > > hverkuil-cisco@xs4all.nl >> > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; >> > > s.hauer@pengutronix.de; kernel@pengutronix.de; festevam@gmail.com; >> > > dl-linux-imx <linux-imx@nxp.com>; linux-media@vger.kernel.org; >> > > linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org >> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add >> > > V4L2_BUF_FLAG_CODECCONFIG flag >> > > >> > > > From: Nicolas Dufresne <nicolas@ndufresne.ca> >> > > > Sent: 2022年7月4日 23:53 >> > > > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org; >> > > > hverkuil-cisco@xs4all.nl >> > > > Cc: shawnguo@kernel.org; robh+dt@kernel.org; >> > > > s.hauer@pengutronix.de; kernel@pengutronix.de; >> > > > festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>; >> > > > linux-media@vger.kernel.org; linux-kernel@vger.kernel.org; >> > > > linux-arm-kernel@lists.infradead.org >> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add >> > > > V4L2_BUF_FLAG_CODECCONFIG flag >> > > > >> > > > Caution: EXT Email >> > > > >> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit : >> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space >> > > > > should be able to hint decoder the vb2 only contains codec >> > > > > config header, but does not contain any frame data. >> > > > > It's only used for parsing header, and can't be decoded. >> > > > >> > > > This is copied from OMX specification. I think we we import >> > > > this, we should at least refer to the original. >> > > > >> > > >> > > Hi Nicolas, >> > > Do you mean OMX_BUFFERFLAG_CODECCONFIG? >> > > I'm sorry that I didn't notice it before. >> > > Currently we only encounter this requirement on Android, I'm >> > > not sure if it has a reference to omx. >> > > And thank you very much for pointing out it. >> > > >> > >> > Android media stack has been based on OMX for the last decade. They >> > are slowly moving to CODEC2 which more or less is a similar >> > abstraction with similar ideas. Let's research prior art, so we don't screw >compatibility. >> > >> >> I got it, I'll try to study the android codec2, and do you agree that >> we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like >> OMX_BUFFERFLAG_CODECCONFIG? >> Or is there any other solution that can handle this case? > >We can probably discuss the name. CODECCONFIG is a bit strange, could be >CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines. >I'm just wondering what is the best rule, since more specification is needed >here. >Current userland expect full frames into each encoded buffer. If we start >splitting these up, we'll break non-android userland (and Android userland does >not seems to be very upstream thing, every vendor forks it). > >I also think that what the CODECCONFIG contains and its format need to be >strictly documented for every CODEC that would allow it. In H.264 notably, the >headers could be packed in Annex B. or AVCc NAL headers. If we look at >FFMPEG, which uses codec_data name, they only requires this when the header >are not "inline", which means only for AVCc. Also, many codec_data is for other >codecs get wrapped into ISOMP4 or Matroska (webm) envelope. > >On the other hand, I don't remember if the 1 frame per buffer is an actual rule >or simply what existing userland expect. Also, I'll be fair, there is no reason this >must come from the driver. Android OMX or CODEC2 COMPONENT is a userland >component, it could do bitstream scanning (using traditional boyer-more) to >find and split appart the config to satisfy its internal API. This can be done with >low overhead and zero-copy. > OK, I'll try to change the name, maybe CODEC_HEADER is better. And I think we add the flag just to support more scenarios, it won't break the existing userland flow. It's not easy to change the android codec2, and we indeed want to support it. >> >> > > Ming >> > > >> > > > > >> > > > > Current, it's usually used by android. >> > > > > >> > > > > Signed-off-by: Ming Qian <ming.qian@nxp.com> >> > > > > --- >> > > > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ >> > > > > include/uapi/linux/videodev2.h | 2 ++ >> > > > > 2 files changed, 11 insertions(+) >> > > > > >> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst >> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst >> > > > > index 4638ec64db00..acdc4556f4f4 100644 >> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst >> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst >> > > > > @@ -607,6 +607,15 @@ Buffer Flags >> > > > > the format. Any subsequent call to the >> > > > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block >anymore, >> > > > > but return an ``EPIPE`` error code. >> > > > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: >> > > > > + >> > > > > + - ``V4L2_BUF_FLAG_CODECCONFIG`` >> > > > > + - 0x00200000 >> > > > > + - This flag may be set when the buffer only contains >> > > > > + codec >> > > > > config >> > > > > + header, but does not contain any frame data. Usually the >> > > > > + codec >> > > config >> > > > > + header is merged to the next idr frame, with the flag >> > > > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some >> > > > > + scenes that >> > > will >> > > > > + split the header and queue it separately. >> > > > >> > > > I think the documentation is clear. Now, if a driver uses this, >> > > > will existing userland (perhaps good to check GStreamer, FFMPEG >> > > > and Chromium ?) will break ? >> > > > So we need existing driver to do this when flagged to, and just >> > > > copy/append when the userland didn't opt-in that feature ? >> > > > >> > > > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`: >> > > > > >> > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git >> > > > > a/include/uapi/linux/videodev2.h >> > > > > b/include/uapi/linux/videodev2.h index >> > > > > 5311ac4fde35..8708ef257710 >> > > > > 100644 >> > > > > --- a/include/uapi/linux/videodev2.h >> > > > > +++ b/include/uapi/linux/videodev2.h >> > > > > @@ -1131,6 +1131,8 @@ static inline __u64 >> > > > > v4l2_timeval_to_ns(const >> > > > struct timeval *tv) >> > > > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 >> > > > > /* mem2mem encoder/decoder */ >> > > > > #define V4L2_BUF_FLAG_LAST 0x00100000 >> > > > > +/* Buffer only contains codec header */ >> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 >> > > > > /* request_fd is valid */ >> > > > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000 >> > > > > >>
diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 4638ec64db00..acdc4556f4f4 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -607,6 +607,15 @@ Buffer Flags the format. Any subsequent call to the :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore, but return an ``EPIPE`` error code. + * .. _`V4L2-BUF-FLAG-CODECCONFIG`: + + - ``V4L2_BUF_FLAG_CODECCONFIG`` + - 0x00200000 + - This flag may be set when the buffer only contains codec config + header, but does not contain any frame data. Usually the codec config + header is merged to the next idr frame, with the flag + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will + split the header and queue it separately. * .. _`V4L2-BUF-FLAG-REQUEST-FD`: - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 5311ac4fde35..8708ef257710 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv) #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000 /* mem2mem encoder/decoder */ #define V4L2_BUF_FLAG_LAST 0x00100000 +/* Buffer only contains codec header */ +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000 /* request_fd is valid */ #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be able to hint decoder the vb2 only contains codec config header, but does not contain any frame data. It's only used for parsing header, and can't be decoded. Current, it's usually used by android. Signed-off-by: Ming Qian <ming.qian@nxp.com> --- Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++ include/uapi/linux/videodev2.h | 2 ++ 2 files changed, 11 insertions(+)