[v4,0/2] Enable JPEG encoding on rk3588

Message

Link Mauve April 18, 2024, 2:15 p.m. UTC

Only the JPEG encoder is available for now, although there are patches
for the undocumented VP8 encoder floating around[0].

This has been tested on a rock-5b, resulting in four /dev/video*
encoders.  The userspace program I’ve been using to test them is
Onix[1], using the jpeg-encoder example, it will pick one of these four
at random (but displays the one it picked):
% ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
% jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg

[0] https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885
[1] https://crates.io/crates/onix

Changes since v3:
- Keep all the previous tags, sorry I forgot to do that in the previous
  version.
- Remove an extra media: in the first patch.

Changes since v2:
- Only expose a single VEPU121, since the driver doesn’t yet support
  exposing them all as a single video node to userspace.

Changes since v1:
- Dropped patches 1 and 4.
- Use the proper compatible form, since this device should be fully
  compatible with the VEPU of rk356x.
- Describe where the VEPU121 name comes from, and list other encoders
  and decoders present in this SoC.
- Properly test the device tree changes, I previously couldn’t since I
  was using a too recent version of python-jsonschema…

Emmanuel Gil Peyrot (2):
  media: dt-binding: Document rk3588’s VEPU121
  arm64: dts: rockchip: Add one VEPU121 to rk3588

 .../bindings/media/rockchip,rk3568-vepu.yaml  |  8 +++--
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 29 +++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

Comments

Jianfeng Liu April 27, 2024, 7:16 a.m. UTC | #1

Hi Emmanuel,

On Thu, 18 Apr 2024 16:15:04 +0200, Emmanuel Gil Peyrot wrote:
>- Only expose a single VEPU121, since the driver doesn’t yet support
>  exposing them all as a single video node to userspace.
I'm also doing work with the hantro node on rk3588 which has both vdpu
and vepu. Discussions can be found here[1]. If I enable the jpeg encoder
at feb50000, would it cause any side effects if other jpeg encoders in
your patch are enabled? And what's the disadvantage of enabling multi
jpeg endoers in devicetree?

[1] https://patchwork.kernel.org/project/linux-rockchip/patch/[e-mail]/

Best regards,
Jianfeng

Heiko Stuebner April 28, 2024, 5:01 p.m. UTC | #2

Hi Jianfeng,

Am Samstag, 27. April 2024, 09:16:38 CEST schrieb Jianfeng Liu:
> Hi Emmanuel,
> 
> On Thu, 18 Apr 2024 16:15:04 +0200, Emmanuel Gil Peyrot wrote:
> >- Only expose a single VEPU121, since the driver doesn’t yet support
> >  exposing them all as a single video node to userspace.
> I'm also doing work with the hantro node on rk3588 which has both vdpu
> and vepu. Discussions can be found here[1]. If I enable the jpeg encoder
> at feb50000, would it cause any side effects if other jpeg encoders in
> your patch are enabled? And what's the disadvantage of enabling multi
> jpeg endoers in devicetree?

the basic problem is that exposing multiple jpeg encoders would require
userspace to do the scheduling. Which is bad.
I.e. all userspace programms would need to know the existence of
all jpeg encoders and then somehow negotiate how to use all of them
most efficiently.

Think multiple different programs that would need to negotiate to
spread across all of them in the best way.

Doing this in the kernel, we just expose one encoder (and queue) all
programs would just pile onto that one encoder and the kernel then
would be on the hook to do the scheduling - which the kernel can do
better (with relevant code implemented)

Jianfeng Liu April 28, 2024, 5:29 p.m. UTC | #3

Hi Heiko,

On Sun, 28 Apr 2024 19:01:38 +0200, Heiko Stübner wrote:
>the basic problem is that exposing multiple jpeg encoders would require
>userspace to do the scheduling. Which is bad.
>I.e. all userspace programms would need to know the existence of
>all jpeg encoders and then somehow negotiate how to use all of them
>most efficiently.
>
>Think multiple different programs that would need to negotiate to
>spread across all of them in the best way.
>
>Doing this in the kernel, we just expose one encoder (and queue) all
>programs would just pile onto that one encoder and the kernel then
>would be on the hook to do the scheduling - which the kernel can do
>better (with relevant code implemented)

Yeah let kernel do the scheduling is indeed better. And I'm happy to
hear this method.

So I will keep the vpu at feb50000 with jpeg endoder disabled until
multi encoder scheduling is implemented.

Best regards,
Jianfeng