| Message ID | 20210426173852.484368-1-harry.wentland@amd.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | A drm_plane API to support HDR planes | expand |
On Mon, Apr 26, 2021 at 01:38:49PM -0400, Harry Wentland wrote: > > ## Introduction > > We are looking to enable HDR support for a couple of single-plane and > multi-plane scenarios. To do this effectively we recommend new > interfaces to drm_plane. Below I'll give a bit of background on HDR and > why we propose these interfaces. I think this is on of the topics that would tremendously benefit from the uapi rfc process, with lots of compositor people involved. https://dri.freedesktop.org/docs/drm/gpu/rfc/ Also for this I think we really do need a pretty solid understanding of the involve compositor protocols, otherwise the kernel uapi is going to be for naught. -Daniel > > > ## Defining a pixel's luminance > > Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms. > > Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows. > > A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits. > > Using a content that's defined in PQ space we can approximate the real world in a much better way. > > Here are some examples of real-life objects and their approximate luminance values: > > | Object | Luminance in nits | > | ----------------- | ----------------- | > | Sun | 1.6 million | > | Fluorescent light | 10,000 | > | Highlights | 1,000 - sunlight | > | White Objects | 250 - 1,000 | > | Typical objects | 1 - 250 | > | Shadows | 0.01 - 1 | > | Ultra Blacks | 0 - 0.0005 | > > > ## Describing the luminance space > > **We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are: > > | EOTF | Description | > | --------- |:------------------------------------------------------------------------- | > | Gamma 2.2 | a simple 2.2 gamma | > | sRGB | 2.4 gamma with small initial linear section | > | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | > | Linear | Linear relationship between pixel value and luminance value | > > > ## Mastering Luminances > > Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays. > > | Display | Luminance range in nits | > | ------------------------ | ----------------------- | > | Typical PC display | 0.3 - 200 | > | Excellent LCD HDTV | 0.3 - 400 | > | HDR LCD w/ local dimming | 0.05 - 1,500 | > > Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content. > > The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property. > > It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently. > > > ## SDR Luminance > > Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable. > > **We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space. > > > ## Let There Be Color > > So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value. > > A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue. > > Examples of color spaces are: > > | Color Space | Description | > | ----------- | ------------------------------------------ | > | BT 601 | similar to BT 709 | > | BT 709 | used by sRGB content; ~53% of BT 2020 | > | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | > | BT 2020 | standard for most HDR content | > > The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane. > > **We propose to add definitions for the RGB variants of the BT color spaces.** > > > ## Color Primaries and White Point > > Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut. > > > ## Displays and Tonemapping > > External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata. > > Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping. > > > ## Pixel Formats > > The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF. > > > ## Proposed use-cases > > Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases: > > **One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set. > > **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set. > > **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane. > > In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed. > > > ## Summary of proposed interface changes > > per drm_plane: > - new RGB color space definitions, mirroring the existing YUV color space definitions > - new transfer function property > - new SDR maximum white level property > > > ## References > > [1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer > > > ## Further Reading > > https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst > http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf > https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf > > > Bhawanpreet Lakha (3): > drm/color: Add RGB Color encodings > drm/color: Add Color transfer functions for HDR/SDR > drm/color: Add sdr boost property > > .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- > .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- > drivers/gpu/drm/arm/malidp_planes.c | 4 +- > drivers/gpu/drm/armada/armada_overlay.c | 4 +- > drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ > drivers/gpu/drm/drm_color_mgmt.c | 84 +++++++++++++++++-- > drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- > .../drm/i915/display/skl_universal_plane.c | 4 +- > drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- > drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- > drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- > drivers/gpu/drm/tidss/tidss_plane.c | 6 +- > include/drm/drm_color_mgmt.h | 25 +++++- > include/drm/drm_plane.h | 30 +++++++ > 14 files changed, 173 insertions(+), 16 deletions(-) > > -- > 2.31.0 > > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/dri-devel
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland <harry.wentland@amd.com> wrote: > ## Introduction > > We are looking to enable HDR support for a couple of single-plane and > multi-plane scenarios. To do this effectively we recommend new > interfaces to drm_plane. Below I'll give a bit of background on HDR > and why we propose these interfaces. > > > ## Defining a pixel's luminance > > Currently the luminance space of pixels in a framebuffer/plane > presented to the display is not well defined. It's usually assumed to > be in a 2.2 or 2.4 gamma space and has no mapping to an absolute > luminance value but is interpreted in relative terms. > > Luminance can be measured and described in absolute terms as candela > per meter squared, or cd/m2, or nits. Even though a pixel value can > be mapped to luminance in a linear fashion to do so without losing a > lot of detail requires 16-bpc color depth. The reason for this is > that human perception can distinguish roughly between a 0.5-1% > luminance delta. A linear representation is suboptimal, wasting > precision in the highlights and losing precision in the shadows. > > A gamma curve is a decent approximation to a human's perception of > luminance, but the PQ (perceptual quantizer) function [1] improves on > it. It also defines the luminance values in absolute terms, with the > highest value being 10,000 nits and the lowest 0.0005 nits. > > Using a content that's defined in PQ space we can approximate the > real world in a much better way. > > Here are some examples of real-life objects and their approximate > luminance values: > > | Object | Luminance in nits | > | ----------------- | ----------------- | > | Sun | 1.6 million | > | Fluorescent light | 10,000 | > | Highlights | 1,000 - sunlight | > | White Objects | 250 - 1,000 | > | Typical objects | 1 - 250 | > | Shadows | 0.01 - 1 | > | Ultra Blacks | 0 - 0.0005 | > > > ## Describing the luminance space > > **We propose a new drm_plane property to describe the Eletro-Optical > Transfer Function (EOTF) with which its framebuffer was composed.** > Examples of EOTF are: > > | EOTF | Description | > | --------- |:------------------------------------------------------------------------- | > | Gamma 2.2 | a simple 2.2 gamma | > | sRGB | 2.4 gamma with small initial linear section | > | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | > | Linear | Linear relationship between pixel value and luminance value | > The definitions agree with what I have learnt so far. However, with these EOTF definitions, only PQ defines absolute luminance values while the others do not. So this is not enough information to blend planes together if they do not all use the same EOTF with the same dynamic range. More below. > > ## Mastering Luminances > > Now we are able to use the PQ 2084 EOTF to define the luminance of > pixels in absolute terms. Unfortunately we're again presented with > physical limitations of the display technologies on the market today. > Here are a few examples of luminance ranges of displays. > > | Display | Luminance range in nits | > | ------------------------ | ----------------------- | > | Typical PC display | 0.3 - 200 | > | Excellent LCD HDTV | 0.3 - 400 | > | HDR LCD w/ local dimming | 0.05 - 1,500 | > > Since no display can currently show the full 0.0005 to 10,000 nits > luminance range the display will need to tonemap the HDR content, i.e > to fit the content within a display's capabilities. To assist with > tonemapping HDR content is usually accompanied with a metadata that > describes (among other things) the minimum and maximum mastering > luminance, i.e. the maximum and minimum luminance of the display that > was used to master the HDR content. > > The HDR metadata is currently defined on the drm_connector via the > hdr_output_metadata blob property. > > It might be useful to define per-plane hdr metadata, as different > planes might have been mastered differently. I don't think this would directly help with the dynamic range blending problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ? > ## SDR Luminance > > Since SDR covers a smaller luminance range than HDR, an SDR plane > might look dark when blended with HDR content. Since the max HDR > luminance can be quite variable (200-1,500 nits on actual displays) > it is best to make the SDR maximum luminance value configurable. > > **We propose a drm_plane property to specfy the desired maximum > luminance of the SDR plane in nits.** This allows us to map the SDR > content predictably into HDR's absolute luminance space. What would be the mapping? Simple linear scaling? A more complicated tone mapping? Rather than "SDR luminance", do you perhaps intend this to configure the dynamic range of the non-absolute-luminance EOTFs? In that case maybe you'd need a black luminance level too? > ## Let There Be Color > > So far we've only talked about luminance, ignoring colors altogether. > Just like in the luminance space, traditionally the color space of > display outputs has not been well defined. Similar to how an EOTF > defines a mapping of pixel data to an absolute luminance value, the > color space maps color information for each pixel onto the CIE 1931 > chromaticity space. This can be thought of as a mapping to an > absolute, real-life, color value. > > A color space is defined by its primaries and white point. The > primaries and white point are expressed as coordinates in the CIE > 1931 color space. Think of the red primary as the reddest red that > can be displayed within the color space. Same for green and blue. > > Examples of color spaces are: > > | Color Space | Description | > | ----------- | ------------------------------------------ | > | BT 601 | similar to BT 709 | > | BT 709 | used by sRGB content; ~53% of BT 2020 | > | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | > | BT 2020 | standard for most HDR content | > > The color space is defined in DRM for YCbCr planes via the > color_encoding property of the drm_plane. I don't think that is quite right. As far I understand, COLOR_ENCODING property controls which matrix is used to convert from YCbCr to RGB, but that is all it does. It is not used for the actual color space. So while these BT standards do specify the chromaticities, they also specify the YCbCr encoding which is the part used in this property. YCbCr and RGB are color models. They are not color spaces. RGB is an additive color model while YCbCr is not, AFAIU. Blending requires an additive color model and linear luminance encoding. You need two color space definitions to create one color space transformation: source color space and destination color space. You also need an idea *how* the two color spaces should be mapped, which is called "rendering intent". You can't do anything with just one color space definition, except to pass it on along with the pixels. To be able to blend planes together, all planes need to be converted to the same color space first: the blending color space, whatever you choose it to be. I do not see where KMS would do this color space conversion, or where it would get the definition of the blending color space. > **We propose to add definitions for the RGB variants of the BT color > spaces.** Therefore I'm not sure this makes sense. > ## Color Primaries and White Point > > Just like displays can currently not represent the entire 0.0005 - > 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not > capable of representing the entire BT.2020 color Gamut. For this > reason video content will often specify the color primaries and white > point used to master the video, in order to allow displays to be able > to map the image as best as possible onto the display's gamut. > > > ## Displays and Tonemapping > > External displays are able to do their own tone and color mapping, > based on the mastering luminance, color primaries, and white space > defined in the HDR metadata. > > Internal panels (which are currently few and far between) usually > don't include the complex HW to do tone and color mapping on their > own and will require the display driver to perform appropriate > mapping. FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full control of all color conversions and tone mappings. OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display gamut and dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them. IMO, a display doing its tone mapping magically is only useful when you want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent. So there are two opposing goals: - Traditional color management wants absolute control of the display, leaving nothing unpredictable and preferably also nothing undefined. Undefined behaviour can always be measured (profiled) which makes it predictable and known. The viewing environment is controlled and constant. - Entertainment wants the most visually impressive image quality by dynamically adapting to both displayed content and to the viewing environment conditions. > ## Pixel Formats > > The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are > unrelated to color space and EOTF definitions. HDR pixels can be > formatted in different ways but in order to not lose precision HDR > content requires at least 10 bpc precision. For this reason > ARGB2101010, P010, and FP16 are the obvious candidates for HDR. > ARGB2101010 and P010 have the advantage of requiring only half the > bandwidth as FP16, while FP16 has the advantage of enough precision > to operate in a linear space, i.e. without EOTF. Right. > ## Proposed use-cases > > Although the userspace side of this work is still in the early stages > it is clear that we will want to support the following two use-cases: > > **One XRGB2101010 HDR Plane:** A single, composited plane of HDR > content. The use-case is a video player on a desktop with the > compositor owning the composition of SDR and HDR content. The content > shall be PQ BT.2020 formatted. The drm_connector's > hdr_output_metadata shall be set. This use case is already possible, right? > **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc > desktop plane, with a P010 HDR video plane underlayed. The HDR plane > shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR > boost value. The drm_connector's hdr_output_metadata shall be set. This use case requires blending in KMS, so is the primary goal I suppose. > **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth > transition we recommend an OS that supports HDR output to provide the > hdr_output_metadata on the drm_connector to configure the output for > HDR, even when the content is only SDR. This will allow for a smooth > transition between SDR-only and HDR content. In this use-case the SDR > max luminance value should be provided on the drm_plane. I think this might be already possible by crafting a CRTC GAMMA LUT? Not sure about precision. > In DCN we will de-PQ or de-Gamma all input in order to blend in > linear space. For SDR content we will also apply any desired boost > before blending. After blending we will then re-apply the PQ EOTF and > do RGB to YCbCr conversion if needed. This assumes the same color space over everything. > > ## Summary of proposed interface changes > > per drm_plane: > - new RGB color space definitions, mirroring the existing YUV color > space definitions > - new transfer function property > - new SDR maximum white level property How will these new KMS properties interact with per-plane DEGAMMA, CTM and/or GAMMA properties? Why go with your proposal instead of per-plane CTM and LUT? I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be: plane: FB -> M1 -> LUT1 -> M2 -> blending input CRTC: blending output -> LUT2 -> M3 -> connector FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector We also need to know where and how clipping happens. I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render intent to be defined by the userspace compositor, rather than building a fixed policy in the kernel. Userspace would be setting transformation operators, not color spaces, to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the same as the output color space, except in optical (linear) encoding. The output color space can be non-standard, e.g. measured with a display profiler equipment. I would expect gamut mapping, dynamic range mapping and tone mapping to be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast. Thanks, pq > ## References > > [1] > https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer > > > ## Further Reading > > https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst > http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf > https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf > > > Bhawanpreet Lakha (3): > drm/color: Add RGB Color encodings > drm/color: Add Color transfer functions for HDR/SDR > drm/color: Add sdr boost property > > .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- > .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- > drivers/gpu/drm/arm/malidp_planes.c | 4 +- > drivers/gpu/drm/armada/armada_overlay.c | 4 +- > drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ > drivers/gpu/drm/drm_color_mgmt.c | 84 > +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | > 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- > drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- > drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- > drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- > drivers/gpu/drm/tidss/tidss_plane.c | 6 +- > include/drm/drm_color_mgmt.h | 25 +++++- > include/drm/drm_plane.h | 30 +++++++ > 14 files changed, 173 insertions(+), 16 deletions(-) >
Hello Harry, Many of us in the mail chain have discussed this before, on what is the right way to blend and tone map a SDR and a HDR buffer from same/different color spaces, and what kind of DRM plane properties will be needed. As you can see from the previous comments, that the majority of the decision making will happen in the Compositor, as it's the only SW unit, which has the overall picture clear. Reference: (https://lists.freedesktop.org/archives/wayland-devel/2019-January/039808.html ) If we see a systematic approach of how do we make such blending policy, it will look like: - Compositor needs to understand the following values of each of the buffer: - Color space or Gamut: BT2020/SRGB/DCI-P3/BT709/BT601 etc - Color format (RGB/YCBCR) and subsampling (444/422/420) - Tone (SDR/HDR_A/HDR_B) - Then the Compositor needs to understand the capabilities of the output display, as this will be a clamping value - Output Gamut support (BT2020/SRGB/DCIP3) - Output max Luminance of the monitor in Nits (even in case of HDR content to HDR display) Based of all this information above, the compositor needs to set a blending target, which contains the following: - Output Colorspace of the blended output: say BT2020 - Output Luminance of the blended output: Match content, if monitor can support it - Output Color format of the blended output: Say YCBCR4:2:0 Let's assume compositor prepares a blending policy with output as: - Output Luminance: HDR 500 Nits - Output color space: BT2020 - Output color format: RGB888 - Output curve: ST2084 Assuming these details, A compositor will look for DRM color properties like these: 1. Degamma plane property : To make buffers linear for Gamut mapping 2. Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace 3. Color space conversion plane property: To convert from YCBCR->RGB 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve We will also need connector/CRTC properties to set AVI info-frames accordingly. A high level block diagram for blending on a generic HW should look like this: /* * SDR 200Nits┌────────────────┐ SDR 200 Nits ┌────────────────┐ SDR 200 ┌──────────────────┐HDR 500┌────────────────┐ HDR 500 * BT709 │ │ BT709 │ │ BT2020 │ │BT2020 │ │ BT2020 * ────────► │ Degamma ├─────────────► │ Gamut Mapping ├────────►│ Tone mapping ├──────►│ Gamma │ * RGB888 │ 2.2 │ RGB888 │ 709->2020 │ RGB888 │ S2H │RGB888 │ ST2084 │ RGB888 * Non Linear │ │ Linear │ │ Linear │ 200->500 │Linear │ │ ST2084 * └────────────────┘ └────────────────┘ └──────────────────┘ └────────────────┘ * * * * * * * * * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 */ Hope this helps to refine the series. Regards Shashank On 27/04/21 20:20, Pekka Paalanen wrote: > On Mon, 26 Apr 2021 13:38:49 -0400 > Harry Wentland <harry.wentland@amd.com> wrote: > >> ## Introduction >> >> We are looking to enable HDR support for a couple of single-plane and >> multi-plane scenarios. To do this effectively we recommend new >> interfaces to drm_plane. Below I'll give a bit of background on HDR >> and why we propose these interfaces. >> >> >> ## Defining a pixel's luminance >> >> Currently the luminance space of pixels in a framebuffer/plane >> presented to the display is not well defined. It's usually assumed to >> be in a 2.2 or 2.4 gamma space and has no mapping to an absolute >> luminance value but is interpreted in relative terms. >> >> Luminance can be measured and described in absolute terms as candela >> per meter squared, or cd/m2, or nits. Even though a pixel value can >> be mapped to luminance in a linear fashion to do so without losing a >> lot of detail requires 16-bpc color depth. The reason for this is >> that human perception can distinguish roughly between a 0.5-1% >> luminance delta. A linear representation is suboptimal, wasting >> precision in the highlights and losing precision in the shadows. >> >> A gamma curve is a decent approximation to a human's perception of >> luminance, but the PQ (perceptual quantizer) function [1] improves on >> it. It also defines the luminance values in absolute terms, with the >> highest value being 10,000 nits and the lowest 0.0005 nits. >> >> Using a content that's defined in PQ space we can approximate the >> real world in a much better way. >> >> Here are some examples of real-life objects and their approximate >> luminance values: >> >> | Object | Luminance in nits | >> | ----------------- | ----------------- | >> | Sun | 1.6 million | >> | Fluorescent light | 10,000 | >> | Highlights | 1,000 - sunlight | >> | White Objects | 250 - 1,000 | >> | Typical objects | 1 - 250 | >> | Shadows | 0.01 - 1 | >> | Ultra Blacks | 0 - 0.0005 | >> >> >> ## Describing the luminance space >> >> **We propose a new drm_plane property to describe the Eletro-Optical >> Transfer Function (EOTF) with which its framebuffer was composed.** >> Examples of EOTF are: >> >> | EOTF | Description | >> | --------- |:------------------------------------------------------------------------- | >> | Gamma 2.2 | a simple 2.2 gamma | >> | sRGB | 2.4 gamma with small initial linear section | >> | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | >> | Linear | Linear relationship between pixel value and luminance value | >> > The definitions agree with what I have learnt so far. However, with > these EOTF definitions, only PQ defines absolute luminance values > while the others do not. So this is not enough information to blend > planes together if they do not all use the same EOTF with the same > dynamic range. More below. > > >> ## Mastering Luminances >> >> Now we are able to use the PQ 2084 EOTF to define the luminance of >> pixels in absolute terms. Unfortunately we're again presented with >> physical limitations of the display technologies on the market today. >> Here are a few examples of luminance ranges of displays. >> >> | Display | Luminance range in nits | >> | ------------------------ | ----------------------- | >> | Typical PC display | 0.3 - 200 | >> | Excellent LCD HDTV | 0.3 - 400 | >> | HDR LCD w/ local dimming | 0.05 - 1,500 | >> >> Since no display can currently show the full 0.0005 to 10,000 nits >> luminance range the display will need to tonemap the HDR content, i.e >> to fit the content within a display's capabilities. To assist with >> tonemapping HDR content is usually accompanied with a metadata that >> describes (among other things) the minimum and maximum mastering >> luminance, i.e. the maximum and minimum luminance of the display that >> was used to master the HDR content. >> >> The HDR metadata is currently defined on the drm_connector via the >> hdr_output_metadata blob property. >> >> It might be useful to define per-plane hdr metadata, as different >> planes might have been mastered differently. > I don't think this would directly help with the dynamic range blending > problem. You still need to establish the mapping between the optical > values from two different EOTFs and dynamic ranges. Or can you know > which optical values match the mastering display maximum and minimum > luminances for not-PQ? > > >> ## SDR Luminance >> >> Since SDR covers a smaller luminance range than HDR, an SDR plane >> might look dark when blended with HDR content. Since the max HDR >> luminance can be quite variable (200-1,500 nits on actual displays) >> it is best to make the SDR maximum luminance value configurable. >> >> **We propose a drm_plane property to specfy the desired maximum >> luminance of the SDR plane in nits.** This allows us to map the SDR >> content predictably into HDR's absolute luminance space. > What would be the mapping? Simple linear scaling? A more complicated > tone mapping? > > Rather than "SDR luminance", do you perhaps intend this to configure > the dynamic range of the non-absolute-luminance EOTFs? > In that case maybe you'd need a black luminance level too? > > >> ## Let There Be Color >> >> So far we've only talked about luminance, ignoring colors altogether. >> Just like in the luminance space, traditionally the color space of >> display outputs has not been well defined. Similar to how an EOTF >> defines a mapping of pixel data to an absolute luminance value, the >> color space maps color information for each pixel onto the CIE 1931 >> chromaticity space. This can be thought of as a mapping to an >> absolute, real-life, color value. >> >> A color space is defined by its primaries and white point. The >> primaries and white point are expressed as coordinates in the CIE >> 1931 color space. Think of the red primary as the reddest red that >> can be displayed within the color space. Same for green and blue. >> >> Examples of color spaces are: >> >> | Color Space | Description | >> | ----------- | ------------------------------------------ | >> | BT 601 | similar to BT 709 | >> | BT 709 | used by sRGB content; ~53% of BT 2020 | >> | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | >> | BT 2020 | standard for most HDR content | >> >> The color space is defined in DRM for YCbCr planes via the >> color_encoding property of the drm_plane. > I don't think that is quite right. > > As far I understand, COLOR_ENCODING property controls which matrix is > used to convert from YCbCr to RGB, but that is all it does. It is not > used for the actual color space. So while these BT standards do > specify the chromaticities, they also specify the YCbCr encoding which > is the part used in this property. > > YCbCr and RGB are color models. They are not color spaces. RGB is an > additive color model while YCbCr is not, AFAIU. Blending requires an > additive color model and linear luminance encoding. > > You need two color space definitions to create one color space > transformation: source color space and destination color space. You > also need an idea *how* the two color spaces should be mapped, which is > called "rendering intent". You can't do anything with just one color > space definition, except to pass it on along with the pixels. > > To be able to blend planes together, all planes need to be converted to > the same color space first: the blending color space, whatever you > choose it to be. I do not see where KMS would do this color space > conversion, or where it would get the definition of the blending color > space. > >> **We propose to add definitions for the RGB variants of the BT color >> spaces.** > Therefore I'm not sure this makes sense. > > >> ## Color Primaries and White Point >> >> Just like displays can currently not represent the entire 0.0005 - >> 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not >> capable of representing the entire BT.2020 color Gamut. For this >> reason video content will often specify the color primaries and white >> point used to master the video, in order to allow displays to be able >> to map the image as best as possible onto the display's gamut. >> >> >> ## Displays and Tonemapping >> >> External displays are able to do their own tone and color mapping, >> based on the mastering luminance, color primaries, and white space >> defined in the HDR metadata. >> >> Internal panels (which are currently few and far between) usually >> don't include the complex HW to do tone and color mapping on their >> own and will require the display driver to perform appropriate >> mapping. > FWIW, when designing Weston's color management, we are aiming for > the latter "simple" panels foremost, because that gives us full control > of all color conversions and tone mappings. > > OTOH, if Weston has to present to a display which only accepts e.g. > BT.2020/PQ signal, the display might always mangle the image in > unexpected ways. Therefore I expect that by default Weston will do > everything it can to try to make the display not apply anything magic > image enhancement: trust that EDID description of the display gamut and > dynamic range are correct, and use HDR metadata to tell the display > that those values are exactly what we are using. And we use them. > > IMO, a display doing its tone mapping magically is only useful when you > want to be able to use "simple" playback devices that cannot adapt to > the display they are driving. Magic tone mapping is also a way for > hardware vendors to differentiate, which from the color management > perspective is harmful as it makes it more difficult or impossible to > predict the display behaviour or to keep it consistent. > > So there are two opposing goals: > > - Traditional color management wants absolute control of the display, > leaving nothing unpredictable and preferably also nothing undefined. > Undefined behaviour can always be measured (profiled) which makes it > predictable and known. The viewing environment is controlled and > constant. > > - Entertainment wants the most visually impressive image quality by > dynamically adapting to both displayed content and to the viewing > environment conditions. > >> ## Pixel Formats >> >> The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are >> unrelated to color space and EOTF definitions. HDR pixels can be >> formatted in different ways but in order to not lose precision HDR >> content requires at least 10 bpc precision. For this reason >> ARGB2101010, P010, and FP16 are the obvious candidates for HDR. >> ARGB2101010 and P010 have the advantage of requiring only half the >> bandwidth as FP16, while FP16 has the advantage of enough precision >> to operate in a linear space, i.e. without EOTF. > Right. > >> ## Proposed use-cases >> >> Although the userspace side of this work is still in the early stages >> it is clear that we will want to support the following two use-cases: >> >> **One XRGB2101010 HDR Plane:** A single, composited plane of HDR >> content. The use-case is a video player on a desktop with the >> compositor owning the composition of SDR and HDR content. The content >> shall be PQ BT.2020 formatted. The drm_connector's >> hdr_output_metadata shall be set. > This use case is already possible, right? > >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane >> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR >> boost value. The drm_connector's hdr_output_metadata shall be set. > This use case requires blending in KMS, so is the primary goal I > suppose. > >> **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth >> transition we recommend an OS that supports HDR output to provide the >> hdr_output_metadata on the drm_connector to configure the output for >> HDR, even when the content is only SDR. This will allow for a smooth >> transition between SDR-only and HDR content. In this use-case the SDR >> max luminance value should be provided on the drm_plane. > I think this might be already possible by crafting a CRTC GAMMA LUT? Not > sure about precision. > >> In DCN we will de-PQ or de-Gamma all input in order to blend in >> linear space. For SDR content we will also apply any desired boost >> before blending. After blending we will then re-apply the PQ EOTF and >> do RGB to YCbCr conversion if needed. > This assumes the same color space over everything. > >> ## Summary of proposed interface changes >> >> per drm_plane: >> - new RGB color space definitions, mirroring the existing YUV color >> space definitions >> - new transfer function property >> - new SDR maximum white level property > How will these new KMS properties interact with per-plane DEGAMMA, CTM > and/or GAMMA properties? > > Why go with your proposal instead of per-plane CTM and LUT? > > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs > both per-plane and on CRTC, would be: > > plane: > FB -> M1 -> LUT1 -> M2 -> blending input > > CRTC: > blending output -> LUT2 -> M3 -> connector > > FB: framebuffer > M1: matrix transform, capable of converting e.g. YCbCr to RGB > LUT1: 1D LUT for content EOTF, to produce light-linear RGB > M2: matrix transform for color space transformation > > LUT2: 1D LUT for applying monitor EOTF^-1 > M3: matrix transform, e.g. if you need to push YCbCr on the connector > > We also need to know where and how clipping happens. > > I think this scheme would allow implementing everything you want, and > it would not be tied to rigid enumerations, and it won't have any > magical conversions done under the hood as you would need to do to > convert from one enum space to another. It leaves the render intent to > be defined by the userspace compositor, rather than building a fixed > policy in the kernel. > > Userspace would be setting transformation operators, not color spaces, > to the kernel, allowing the blending space to be chosen by userspace. > In Weston we aim to choose then blending color space to be the same as > the output color space, except in optical (linear) encoding. The output > color space can be non-standard, e.g. measured with a display profiler > equipment. > > I would expect gamut mapping, dynamic range mapping and tone mapping to > be places where most experimentation and innovation happens, so > implementing them in the kernel with just few or no controllable > parameters at this time seems like it could become useless fast. > > > Thanks, > pq > >> ## References >> >> [1] >> https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer >> >> >> ## Further Reading >> >> https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst >> http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf >> https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf >> >> >> Bhawanpreet Lakha (3): >> drm/color: Add RGB Color encodings >> drm/color: Add Color transfer functions for HDR/SDR >> drm/color: Add sdr boost property >> >> .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- >> .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- >> drivers/gpu/drm/arm/malidp_planes.c | 4 +- >> drivers/gpu/drm/armada/armada_overlay.c | 4 +- >> drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ >> drivers/gpu/drm/drm_color_mgmt.c | 84 >> +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | >> 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- >> drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- >> drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- >> drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- >> drivers/gpu/drm/tidss/tidss_plane.c | 6 +- >> include/drm/drm_color_mgmt.h | 25 +++++- >> include/drm/drm_plane.h | 30 +++++++ >> 14 files changed, 173 insertions(+), 16 deletions(-) >>
On Wed, 28 Apr 2021 13:24:27 +0530 Shashank Sharma <shashank.sharma@amd.com> wrote: > Assuming these details, A compositor will look for DRM color properties like these: > > 1. Degamma plane property : To make buffers linear for Gamut mapping > > 2. Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace > > 3. Color space conversion plane property: To convert from YCBCR->RGB > > 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H > > 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve > > ... > * > * > * > * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ > * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 > * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ > * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 > * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 > * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 > */ Hi Shashank, I think you might have degamma and color model conversion reversed, or is that a new thing in the HDR specs? Usually the YCbCr/RGB conversion matrix applies to non-linear values AFAIU. There is also confusion with OETF vs. EOTF. I got that initially wrong too. OETF is not just a name for inverse-EOTF but it is used in a different context. Though here it seems to be just a typo. OETF is inherent to a camera when it converts light into electrical signals. EOTF is inherent to a monitor when it converts electrical signals to light. Depending on what the electrical signals have been defined to be in each step of a broadcasting chain, you might need OETF or EOTF or their inverse or a different OETF or EOTF or their inverse. As we are talking about displays and likely assuming display-referred content (not scene-referred content), we probably have no use for OETF, but we could have several different EOTFs. Thanks, pq
Hello Pekka, On 30/04/21 15:13, Pekka Paalanen wrote: > On Wed, 28 Apr 2021 13:24:27 +0530 > Shashank Sharma <shashank.sharma@amd.com> wrote: > >> Assuming these details, A compositor will look for DRM color properties like these: >> >> 1. Degamma plane property : To make buffers linear for Gamut mapping >> >> 2. Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace >> >> 3. Color space conversion plane property: To convert from YCBCR->RGB >> >> 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H >> >> 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve >> >> > ... > >> * >> * >> * >> * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ >> * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 >> * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ >> * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 >> * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 >> * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 >> */ > Hi Shashank, > > I think you might have degamma and color model conversion reversed, or > is that a new thing in the HDR specs? > > Usually the YCbCr/RGB conversion matrix applies to non-linear values > AFAIU. Ah, that was due to the Gamut mapping block. You are right, color format conversion can happen on non-linear data (doesn't mean it can't happen on linear), but in the sequential block above, there was gamut mapping (color space conversion), which needs to be done on Linear space, and I was a bit too lazy to create separate blocks, so I just re[placed the block titles :D. > There is also confusion with OETF vs. EOTF. I got that initially wrong > too. OETF is not just a name for inverse-EOTF but it is used in a > different context. Though here it seems to be just a typo. > OETF is inherent to a camera when it converts light into > electrical signals. EOTF is inherent to a monitor when it converts > electrical signals to light. Depending on what the electrical signals > have been defined to be in each step of a broadcasting chain, you might > need OETF or EOTF or their inverse or a different OETF or EOTF or their > inverse. Yes, that was a typo. The intention was to call it inverse curve for HDR encoded buffers. It's almost 4 years (and 2 companies) since I last did HDR, so I am a bit rusty on the topic ;) . - Shashank > > As we are talking about displays and likely assuming display-referred > content (not scene-referred content), we probably have no use for OETF, > but we could have several different EOTFs. > > > Thanks, > pq
On 2021-04-30 6:39 a.m., Shashank Sharma wrote: > Hello Pekka, > > On 30/04/21 15:13, Pekka Paalanen wrote: >> On Wed, 28 Apr 2021 13:24:27 +0530 >> Shashank Sharma <shashank.sharma@amd.com> wrote: >> >>> Assuming these details, A compositor will look for DRM color properties like these: >>> >>> 1. Degamma plane property : To make buffers linear for Gamut mapping >>> >>> 2. Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace >>> >>> 3. Color space conversion plane property: To convert from YCBCR->RGB >>> >>> 4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H >>> >>> 5. Gamma plane/CRTC property: to re-apply the output ST2084 curve >>> >>> >> ... >> >>> * >>> * >>> * >>> * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ >>> * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 >>> * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ >>> * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 >>> * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 >>> * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 >>> */ >> Hi Shashank, >> >> I think you might have degamma and color model conversion reversed, or >> is that a new thing in the HDR specs? >> >> Usually the YCbCr/RGB conversion matrix applies to non-linear values >> AFAIU. > Ah, that was due to the Gamut mapping block. You are right, color format conversion can happen on non-linear data (doesn't mean it can't happen on linear), but in the sequential block above, there was gamut mapping (color space conversion), which needs to be done on Linear space, and I was a bit too lazy to create separate blocks, so I just re[placed the block titles :D. >> There is also confusion with OETF vs. EOTF. I got that initially wrong >> too. OETF is not just a name for inverse-EOTF but it is used in a >> different context. Though here it seems to be just a typo. >> OETF is inherent to a camera when it converts light into >> electrical signals. EOTF is inherent to a monitor when it converts >> electrical signals to light. Depending on what the electrical signals >> have been defined to be in each step of a broadcasting chain, you might >> need OETF or EOTF or their inverse or a different OETF or EOTF or their >> inverse. > > Yes, that was a typo. The intention was to call it inverse curve for HDR encoded buffers. It's almost 4 years (and 2 companies) since I last did HDR, so I am a bit rusty on the topic ;) . > > - Shashank > Thanks, Ville and Shashank. This is indeed helpful. I apologize for the late response but I needed to take some time to do more reading and internalize some of the HDR and CM concepts. I will send out a v2 of my patchset but realize that it is only a small step toward the right KMS interface for HDR and CM. Harry >> >> As we are talking about displays and likely assuming display-referred >> content (not scene-referred content), we probably have no use for OETF, >> but we could have several different EOTFs. >> >> >> Thanks, >> pq
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: > On Mon, 26 Apr 2021 13:38:49 -0400 > Harry Wentland <harry.wentland@amd.com> wrote: > >> ## Introduction >> >> We are looking to enable HDR support for a couple of single-plane and >> multi-plane scenarios. To do this effectively we recommend new >> interfaces to drm_plane. Below I'll give a bit of background on HDR >> and why we propose these interfaces. >> >> >> ## Defining a pixel's luminance >> >> Currently the luminance space of pixels in a framebuffer/plane >> presented to the display is not well defined. It's usually assumed to >> be in a 2.2 or 2.4 gamma space and has no mapping to an absolute >> luminance value but is interpreted in relative terms. >> >> Luminance can be measured and described in absolute terms as candela >> per meter squared, or cd/m2, or nits. Even though a pixel value can >> be mapped to luminance in a linear fashion to do so without losing a >> lot of detail requires 16-bpc color depth. The reason for this is >> that human perception can distinguish roughly between a 0.5-1% >> luminance delta. A linear representation is suboptimal, wasting >> precision in the highlights and losing precision in the shadows. >> >> A gamma curve is a decent approximation to a human's perception of >> luminance, but the PQ (perceptual quantizer) function [1] improves on >> it. It also defines the luminance values in absolute terms, with the >> highest value being 10,000 nits and the lowest 0.0005 nits. >> >> Using a content that's defined in PQ space we can approximate the >> real world in a much better way. >> >> Here are some examples of real-life objects and their approximate >> luminance values: >> >> | Object | Luminance in nits | >> | ----------------- | ----------------- | >> | Sun | 1.6 million | >> | Fluorescent light | 10,000 | >> | Highlights | 1,000 - sunlight | >> | White Objects | 250 - 1,000 | >> | Typical objects | 1 - 250 | >> | Shadows | 0.01 - 1 | >> | Ultra Blacks | 0 - 0.0005 | >> >> >> ## Describing the luminance space >> >> **We propose a new drm_plane property to describe the Eletro-Optical >> Transfer Function (EOTF) with which its framebuffer was composed.** >> Examples of EOTF are: >> >> | EOTF | Description | >> | --------- |:------------------------------------------------------------------------- | >> | Gamma 2.2 | a simple 2.2 gamma | >> | sRGB | 2.4 gamma with small initial linear section | >> | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | >> | Linear | Linear relationship between pixel value and luminance value | >> > > The definitions agree with what I have learnt so far. However, with > these EOTF definitions, only PQ defines absolute luminance values > while the others do not. So this is not enough information to blend > planes together if they do not all use the same EOTF with the same > dynamic range. More below. > Good point. > >> >> ## Mastering Luminances >> >> Now we are able to use the PQ 2084 EOTF to define the luminance of >> pixels in absolute terms. Unfortunately we're again presented with >> physical limitations of the display technologies on the market today. >> Here are a few examples of luminance ranges of displays. >> >> | Display | Luminance range in nits | >> | ------------------------ | ----------------------- | >> | Typical PC display | 0.3 - 200 | >> | Excellent LCD HDTV | 0.3 - 400 | >> | HDR LCD w/ local dimming | 0.05 - 1,500 | >> >> Since no display can currently show the full 0.0005 to 10,000 nits >> luminance range the display will need to tonemap the HDR content, i.e >> to fit the content within a display's capabilities. To assist with >> tonemapping HDR content is usually accompanied with a metadata that >> describes (among other things) the minimum and maximum mastering >> luminance, i.e. the maximum and minimum luminance of the display that >> was used to master the HDR content. >> >> The HDR metadata is currently defined on the drm_connector via the >> hdr_output_metadata blob property. >> >> It might be useful to define per-plane hdr metadata, as different >> planes might have been mastered differently. > > I don't think this would directly help with the dynamic range blending > problem. You still need to establish the mapping between the optical > values from two different EOTFs and dynamic ranges. Or can you know > which optical values match the mastering display maximum and minimum > luminances for not-PQ? > My understanding of this is probably best illustrated by this example: Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel. If you do not know the mastering luminance is 500 nits you would have to compress 10,000 nits down to 400 (assuming PQ), losing quite a bit of the full 400 nits available as you'll need room to map the 500 to 10,000 nits range which in reality is completely unused. You might end up with mapping 500 nits to 350 nits, instead of mapping it to 400. > >> ## SDR Luminance >> >> Since SDR covers a smaller luminance range than HDR, an SDR plane >> might look dark when blended with HDR content. Since the max HDR >> luminance can be quite variable (200-1,500 nits on actual displays) >> it is best to make the SDR maximum luminance value configurable. >> >> **We propose a drm_plane property to specfy the desired maximum >> luminance of the SDR plane in nits.** This allows us to map the SDR >> content predictably into HDR's absolute luminance space. > > What would be the mapping? Simple linear scaling? A more complicated > tone mapping? > > Rather than "SDR luminance", do you perhaps intend this to configure > the dynamic range of the non-absolute-luminance EOTFs? > In that case maybe you'd need a black luminance level too? > SDR luminance is essentially how AMD HW lets us deal with this, but conceptually you're right, this could extend beyond SDR to any non-absolute-luminance (NAL?) EOTF. I don't think AMD HW currently has a way to program a black luminance level unless we turn the white/black luminance into a LUT. > >> ## Let There Be Color >> >> So far we've only talked about luminance, ignoring colors altogether. >> Just like in the luminance space, traditionally the color space of >> display outputs has not been well defined. Similar to how an EOTF >> defines a mapping of pixel data to an absolute luminance value, the >> color space maps color information for each pixel onto the CIE 1931 >> chromaticity space. This can be thought of as a mapping to an >> absolute, real-life, color value. >> >> A color space is defined by its primaries and white point. The >> primaries and white point are expressed as coordinates in the CIE >> 1931 color space. Think of the red primary as the reddest red that >> can be displayed within the color space. Same for green and blue. >> >> Examples of color spaces are: >> >> | Color Space | Description | >> | ----------- | ------------------------------------------ | >> | BT 601 | similar to BT 709 | >> | BT 709 | used by sRGB content; ~53% of BT 2020 | >> | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | >> | BT 2020 | standard for most HDR content | >> >> The color space is defined in DRM for YCbCr planes via the >> color_encoding property of the drm_plane. > > I don't think that is quite right. > > As far I understand, COLOR_ENCODING property controls which matrix is > used to convert from YCbCr to RGB, but that is all it does. It is not > used for the actual color space. So while these BT standards do > specify the chromaticities, they also specify the YCbCr encoding which > is the part used in this property. > > YCbCr and RGB are color models. They are not color spaces. RGB is an > additive color model while YCbCr is not, AFAIU. Blending requires an > additive color model and linear luminance encoding. > > You need two color space definitions to create one color space > transformation: source color space and destination color space. You > also need an idea *how* the two color spaces should be mapped, which is > called "rendering intent". You can't do anything with just one color > space definition, except to pass it on along with the pixels. > > To be able to blend planes together, all planes need to be converted to > the same color space first: the blending color space, whatever you > choose it to be. I do not see where KMS would do this color space > conversion, or where it would get the definition of the blending color > space. > >> **We propose to add definitions for the RGB variants of the BT color >> spaces.** > > Therefore I'm not sure this makes sense. > You and Ville are absolutely correct and a reading of https://en.wikipedia.org/wiki/YCbCr set me straight. :) > >> ## Color Primaries and White Point >> >> Just like displays can currently not represent the entire 0.0005 - >> 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not >> capable of representing the entire BT.2020 color Gamut. For this >> reason video content will often specify the color primaries and white >> point used to master the video, in order to allow displays to be able >> to map the image as best as possible onto the display's gamut. >> >> >> ## Displays and Tonemapping >> >> External displays are able to do their own tone and color mapping, >> based on the mastering luminance, color primaries, and white space >> defined in the HDR metadata. >> >> Internal panels (which are currently few and far between) usually >> don't include the complex HW to do tone and color mapping on their >> own and will require the display driver to perform appropriate >> mapping. > > FWIW, when designing Weston's color management, we are aiming for > the latter "simple" panels foremost, because that gives us full control > of all color conversions and tone mappings. > > OTOH, if Weston has to present to a display which only accepts e.g. > BT.2020/PQ signal, the display might always mangle the image in > unexpected ways. Therefore I expect that by default Weston will do > everything it can to try to make the display not apply anything magic > image enhancement: trust that EDID description of the display gamut and > dynamic range are correct, and use HDR metadata to tell the display > that those values are exactly what we are using. And we use them. > > IMO, a display doing its tone mapping magically is only useful when you > want to be able to use "simple" playback devices that cannot adapt to > the display they are driving. Magic tone mapping is also a way for > hardware vendors to differentiate, which from the color management > perspective is harmful as it makes it more difficult or impossible to > predict the display behaviour or to keep it consistent. > > So there are two opposing goals: > > - Traditional color management wants absolute control of the display, > leaving nothing unpredictable and preferably also nothing undefined. > Undefined behaviour can always be measured (profiled) which makes it > predictable and known. The viewing environment is controlled and > constant. > > - Entertainment wants the most visually impressive image quality by > dynamically adapting to both displayed content and to the viewing > environment conditions. > I feel much of the mess around HDR and CM is all the magic that everyone tries to do to differentiate themselves. I'm not a fan of that. One thing to watch out for is that the definitions for some of these things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair bit between HW vendors and we'd want to find a way to hide these complexities to KMS clients. >> ## Pixel Formats >> >> The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are >> unrelated to color space and EOTF definitions. HDR pixels can be >> formatted in different ways but in order to not lose precision HDR >> content requires at least 10 bpc precision. For this reason >> ARGB2101010, P010, and FP16 are the obvious candidates for HDR. >> ARGB2101010 and P010 have the advantage of requiring only half the >> bandwidth as FP16, while FP16 has the advantage of enough precision >> to operate in a linear space, i.e. without EOTF. > > Right. > >> ## Proposed use-cases >> >> Although the userspace side of this work is still in the early stages >> it is clear that we will want to support the following two use-cases: >> >> **One XRGB2101010 HDR Plane:** A single, composited plane of HDR >> content. The use-case is a video player on a desktop with the >> compositor owning the composition of SDR and HDR content. The content >> shall be PQ BT.2020 formatted. The drm_connector's >> hdr_output_metadata shall be set. > > This use case is already possible, right? > Yes, not sure how well tested it is but the AMD driver at least is programming the LUTs into bypass if we don't have a degamma or regamma LUT. >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane >> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR >> boost value. The drm_connector's hdr_output_metadata shall be set. > > This use case requires blending in KMS, so is the primary goal I > suppose. > Correct, my understanding is that the key here is to ensure blending in KMS is done in same blending space (linear RGB) as blending using GPU shaders and SDR boost is applied in the same fashion. >> **One XRGB8888 SDR Plane - HDR output:** In order to support a smooth >> transition we recommend an OS that supports HDR output to provide the >> hdr_output_metadata on the drm_connector to configure the output for >> HDR, even when the content is only SDR. This will allow for a smooth >> transition between SDR-only and HDR content. In this use-case the SDR >> max luminance value should be provided on the drm_plane. > > I think this might be already possible by crafting a CRTC GAMMA LUT? Not > sure about precision. > >> In DCN we will de-PQ or de-Gamma all input in order to blend in >> linear space. For SDR content we will also apply any desired boost >> before blending. After blending we will then re-apply the PQ EOTF and >> do RGB to YCbCr conversion if needed. > > This assumes the same color space over everything. > >> >> ## Summary of proposed interface changes >> >> per drm_plane: >> - new RGB color space definitions, mirroring the existing YUV color >> space definitions >> - new transfer function property >> - new SDR maximum white level property > > How will these new KMS properties interact with per-plane DEGAMMA, CTM > and/or GAMMA properties? > > Why go with your proposal instead of per-plane CTM and LUT? > > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs > both per-plane and on CRTC, would be: > > plane: > FB -> M1 -> LUT1 -> M2 -> blending input > > CRTC: > blending output -> LUT2 -> M3 -> connector > > FB: framebuffer > M1: matrix transform, capable of converting e.g. YCbCr to RGB > LUT1: 1D LUT for content EOTF, to produce light-linear RGB > M2: matrix transform for color space transformation > > LUT2: 1D LUT for applying monitor EOTF^-1 > M3: matrix transform, e.g. if you need to push YCbCr on the connector > > We also need to know where and how clipping happens. > > I think this scheme would allow implementing everything you want, and > it would not be tied to rigid enumerations, and it won't have any > magical conversions done under the hood as you would need to do to > convert from one enum space to another. It leaves the render intent to > be defined by the userspace compositor, rather than building a fixed > policy in the kernel. > > Userspace would be setting transformation operators, not color spaces, > to the kernel, allowing the blending space to be chosen by userspace. > In Weston we aim to choose then blending color space to be the same as > the output color space, except in optical (linear) encoding. The output > color space can be non-standard, e.g. measured with a display profiler > equipment. > > I would expect gamut mapping, dynamic range mapping and tone mapping to > be places where most experimentation and innovation happens, so > implementing them in the kernel with just few or no controllable > parameters at this time seems like it could become useless fast. > How important from a Wayland compositor perspective would it be to have fully configurable LUTs per plane, as opposed to providing KMS with a named transfer function, such as PQ or BT709, for LUT1 and LUT2? M1 on AMD's driver is currently hidden in our driver and essentially always set to BT709 at the moment, though we should look at color_encoding to pick the right M1. Are the color_encoding and color_range properties enough for what you'd like to achieve for M1? M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be applied at M2 by AMD's driver. I need to dig into this a bit more. It would make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane). I did a bunch of reading and thinking since your comments and created a v2 of the patchset. I don't think it's where we need it to be but I hope it's a step in the right direction. I will send it shortly. Harry > > Thanks, > pq > >> ## References >> >> [1] >> https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer >> >> >> ## Further Reading >> >> https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable/color-management/color.rst >> http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf >> https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf >> >> >> Bhawanpreet Lakha (3): >> drm/color: Add RGB Color encodings >> drm/color: Add Color transfer functions for HDR/SDR >> drm/color: Add sdr boost property >> >> .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- >> .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- >> drivers/gpu/drm/arm/malidp_planes.c | 4 +- >> drivers/gpu/drm/armada/armada_overlay.c | 4 +- >> drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ >> drivers/gpu/drm/drm_color_mgmt.c | 84 >> +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | >> 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- >> drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- >> drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- >> drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- >> drivers/gpu/drm/tidss/tidss_plane.c | 6 +- >> include/drm/drm_color_mgmt.h | 25 +++++- >> include/drm/drm_plane.h | 30 +++++++ >> 14 files changed, 173 insertions(+), 16 deletions(-) >> >
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland <harry.wentland@amd.com> wrote: > On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: > > On Mon, 26 Apr 2021 13:38:49 -0400 > > Harry Wentland <harry.wentland@amd.com> wrote: ... > >> ## Mastering Luminances > >> > >> Now we are able to use the PQ 2084 EOTF to define the luminance of > >> pixels in absolute terms. Unfortunately we're again presented with > >> physical limitations of the display technologies on the market today. > >> Here are a few examples of luminance ranges of displays. > >> > >> | Display | Luminance range in nits | > >> | ------------------------ | ----------------------- | > >> | Typical PC display | 0.3 - 200 | > >> | Excellent LCD HDTV | 0.3 - 400 | > >> | HDR LCD w/ local dimming | 0.05 - 1,500 | > >> > >> Since no display can currently show the full 0.0005 to 10,000 nits > >> luminance range the display will need to tonemap the HDR content, i.e > >> to fit the content within a display's capabilities. To assist with > >> tonemapping HDR content is usually accompanied with a metadata that > >> describes (among other things) the minimum and maximum mastering > >> luminance, i.e. the maximum and minimum luminance of the display that > >> was used to master the HDR content. > >> > >> The HDR metadata is currently defined on the drm_connector via the > >> hdr_output_metadata blob property. > >> > >> It might be useful to define per-plane hdr metadata, as different > >> planes might have been mastered differently. > > > > I don't think this would directly help with the dynamic range blending > > problem. You still need to establish the mapping between the optical > > values from two different EOTFs and dynamic ranges. Or can you know > > which optical values match the mastering display maximum and minimum > > luminances for not-PQ? > > > > My understanding of this is probably best illustrated by this example: > > Assume HDR was mastered on a display with a maximum white level of 500 > nits and played back on a display that supports a max white level of 400 > nits. If you know the mastering white level of 500 you know that this is > the maximum value you need to compress down to 400 nits, allowing you to > use the full extent of the 400 nits panel. Right, but in the kernel, where do you get these nits values from? hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor. EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences. What about black levels? Do you want to do black level adjustment? How exactly should the compression work? Where do you map the mid-tones? What if the end user wants something different? > If you do not know the mastering luminance is 500 nits you would > have to compress 10,000 nits down to 400 (assuming PQ), losing quite > a bit of the full 400 nits available as you'll need room to map the 500 > to 10,000 nits range which in reality is completely unused. You might end > up with mapping 500 nits to 350 nits, instead of mapping it to 400. The quality of the result depends on the compression (tone-mapping) algorithm. I believe no-one will ever do a simple linear compression of ranges. Instead, you probably do something smooth in the black end, keep mid-tones roughly as they are, and the again do a smooth saturation to some "reasonable" level that goes well with the monitor in the current lighting environment without introducing coloring artifacts, and just clip the huge overshoot of the full PQ-range. There are many big and small decisions to be made in how to map out-of-gamut or out-of-brightness values into the displayable range, and no algorithm fits all use cases. I believe this is why e.g. ICC has several different "render intents", some of which are so vaguely defined that they are practically undefined - just like what "a good picture" means. You have essentially three dimensions: luminance, hue, and saturation. Which one will you sacrifice, shift or emphasize and to what degree to fit the square content peg into the round display hole? A naive example: Let's say content has 300 nits red. Your display can show max 400 nits white or max 180 nits red, and anything in between. What will you show? The problem is, that if UAPI does not define exactly what happens, then taking advantage of these hardware capabilities means you have no idea what happens to your content. This may be fine for closed systems, where the software has been carefully built for the specific hardware revision and the use cases of the complete system have been pre-defined, so that it can assume what should and will happen. But should that be exposed as a generic UAPI, when generic userspace has no chance of knowing what it will do? ... > >> ## Displays and Tonemapping > >> > >> External displays are able to do their own tone and color mapping, > >> based on the mastering luminance, color primaries, and white space > >> defined in the HDR metadata. > >> > >> Internal panels (which are currently few and far between) usually > >> don't include the complex HW to do tone and color mapping on their > >> own and will require the display driver to perform appropriate > >> mapping. > > > > FWIW, when designing Weston's color management, we are aiming for > > the latter "simple" panels foremost, because that gives us full control > > of all color conversions and tone mappings. > > > > OTOH, if Weston has to present to a display which only accepts e.g. > > BT.2020/PQ signal, the display might always mangle the image in > > unexpected ways. Therefore I expect that by default Weston will do > > everything it can to try to make the display not apply anything magic > > image enhancement: trust that EDID description of the display gamut and > > dynamic range are correct, and use HDR metadata to tell the display > > that those values are exactly what we are using. And we use them. > > > > IMO, a display doing its tone mapping magically is only useful when you > > want to be able to use "simple" playback devices that cannot adapt to > > the display they are driving. Magic tone mapping is also a way for > > hardware vendors to differentiate, which from the color management > > perspective is harmful as it makes it more difficult or impossible to > > predict the display behaviour or to keep it consistent. > > > > So there are two opposing goals: > > > > - Traditional color management wants absolute control of the display, > > leaving nothing unpredictable and preferably also nothing undefined. > > Undefined behaviour can always be measured (profiled) which makes it > > predictable and known. The viewing environment is controlled and > > constant. > > > > - Entertainment wants the most visually impressive image quality by > > dynamically adapting to both displayed content and to the viewing > > environment conditions. > > > > I feel much of the mess around HDR and CM is all the magic that everyone > tries to do to differentiate themselves. I'm not a fan of that. > > One thing to watch out for is that the definitions for some of these > things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair > bit between HW vendors and we'd want to find a way to hide these > complexities to KMS clients. How could we hide them? Actually, what do you mean with "hide"? ... > >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc > >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane > >> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR > >> boost value. The drm_connector's hdr_output_metadata shall be set. > > > > This use case requires blending in KMS, so is the primary goal I > > suppose. > > > > Correct, my understanding is that the key here is to ensure blending in KMS > is done in same blending space (linear RGB) as blending using GPU shaders and > SDR boost is applied in the same fashion. Yes, but this is also something I think cannot be achieved by telling the kernel about color spaces and letting the driver come up with a color transformation. I believe this can only be achieved by telling the kernel the color transformation directly, and also describing to userspace what kind of color transformations are possible (1D LUT, 3D LUT, other; precision...). ... > >> ## Summary of proposed interface changes > >> > >> per drm_plane: > >> - new RGB color space definitions, mirroring the existing YUV color > >> space definitions > >> - new transfer function property > >> - new SDR maximum white level property > > > > How will these new KMS properties interact with per-plane DEGAMMA, CTM > > and/or GAMMA properties? > > > > Why go with your proposal instead of per-plane CTM and LUT? > > > > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs > > both per-plane and on CRTC, would be: > > > > plane: > > FB -> M1 -> LUT1 -> M2 -> blending input > > > > CRTC: > > blending output -> LUT2 -> M3 -> connector > > > > FB: framebuffer > > M1: matrix transform, capable of converting e.g. YCbCr to RGB > > LUT1: 1D LUT for content EOTF, to produce light-linear RGB > > M2: matrix transform for color space transformation > > > > LUT2: 1D LUT for applying monitor EOTF^-1 > > M3: matrix transform, e.g. if you need to push YCbCr on the connector > > > > We also need to know where and how clipping happens. > > > > I think this scheme would allow implementing everything you want, and > > it would not be tied to rigid enumerations, and it won't have any > > magical conversions done under the hood as you would need to do to > > convert from one enum space to another. It leaves the render intent to > > be defined by the userspace compositor, rather than building a fixed > > policy in the kernel. > > > > Userspace would be setting transformation operators, not color spaces, > > to the kernel, allowing the blending space to be chosen by userspace. > > In Weston we aim to choose then blending color space to be the same as > > the output color space, except in optical (linear) encoding. The output > > color space can be non-standard, e.g. measured with a display profiler > > equipment. > > > > I would expect gamut mapping, dynamic range mapping and tone mapping to > > be places where most experimentation and innovation happens, so > > implementing them in the kernel with just few or no controllable > > parameters at this time seems like it could become useless fast. > > > > How important from a Wayland compositor perspective would it be to have fully > configurable LUTs per plane, as opposed to providing KMS with a named transfer > function, such as PQ or BT709, for LUT1 and LUT2? For Wayland desktop compositors, absolutely essential in my opinion. I would strongly advice against any desktop compositor from using the "named transfer function" API and instead do everything in GPU shaders if no other KMS way exists, unless the KMS UAPI definition gives the exact mathematical formula of what that step will do. For entertainment devices using Wayland as an internal detail, who cares. The system manufacturer makes things work any way they want, and you cannot run non-curated apps on those devices, so there are no compatibility issues. Then there would be systems between those two extremes, which would likely use a mixture of those approaches, e.g. Kodi on bare hardware I imagine (no Wayland there, but Wayland is kind of unrelated except for the ideology). Mind, I'm not talking only about EOTFs here, but thinking of the whole color management. EOTFs per se are simple. Tone-mapping, including gamut mapping and luminance mapping, are hard because there is no one correct formula. However, if we are talking about *only* EOTFs, then an enumeration of the curves could be usable. But you were also talking about configuring dynamic ranges and doing luminance range mapping in the driver, which I think won't work with enums, but could perhaps be accounted for in M2 by userspace. Then again, if you have enums and not a LUT, you cannot implement custom non-linear per-channel mapping which might be useful for e.g. adapting SDR content for a HDR blending space. Or HDR content to a different HDR blending space. See, a LUT can be more than just EOTF: it can apply a part of the tone-mapping too. An enum cannot. I have a feeling that most tone-mapping curves would be non-linear with some sort of S-shape and clipping of black and white ends. But that is also a naive view, because tone-mapping is not a 1D-curve, it's a mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do 600 nit red for instance. On a 600 nit monitor you can do exactly one color with 600 nits: white. > M1 on AMD's driver is currently hidden in our driver and essentially always > set to BT709 at the moment, though we should look at color_encoding to pick > the right M1. Are the color_encoding and color_range properties enough for > what you'd like to achieve for M1? Probably yes, for standard format content, until a new standard comes up again and then you have to add another enum value in the UAPI. M1 is probably the best place where an enum fits in the pipeline though, and for M3 as well because non-RGB inputs to monitors are quite standardised, and at those steps I would assume there are no parts of tone-mapping done. > M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be > applied at M2 by AMD's driver. I need to dig into this a bit more. It would > make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane). Good, it's really important to make sure that all drivers agree on the pipeline layout that is defined(???) in the KMS UAPI specification. > I did a bunch of reading and thinking since your comments and created a v2 of > the patchset. I don't think it's where we need it to be but I hope it's a > step in the right direction. I will send it shortly. I have to say I'm still very sceptical about this enum-based approach as you have noticed. :-) Thanks, pq
On 2021-05-17 10:57, Pekka Paalanen wrote: > On Fri, 14 May 2021 17:05:11 -0400 > Harry Wentland <harry.wentland@amd.com> wrote: > >> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: >> > On Mon, 26 Apr 2021 13:38:49 -0400 >> > Harry Wentland <harry.wentland@amd.com> wrote: > > ... > >> >> ## Mastering Luminances >> >> >> >> Now we are able to use the PQ 2084 EOTF to define the luminance of >> >> pixels in absolute terms. Unfortunately we're again presented with >> >> physical limitations of the display technologies on the market today. >> >> Here are a few examples of luminance ranges of displays. >> >> >> >> | Display | Luminance range in nits | >> >> | ------------------------ | ----------------------- | >> >> | Typical PC display | 0.3 - 200 | >> >> | Excellent LCD HDTV | 0.3 - 400 | >> >> | HDR LCD w/ local dimming | 0.05 - 1,500 | >> >> >> >> Since no display can currently show the full 0.0005 to 10,000 nits >> >> luminance range the display will need to tonemap the HDR content, i.e >> >> to fit the content within a display's capabilities. To assist with >> >> tonemapping HDR content is usually accompanied with a metadata that >> >> describes (among other things) the minimum and maximum mastering >> >> luminance, i.e. the maximum and minimum luminance of the display that >> >> was used to master the HDR content. >> >> >> >> The HDR metadata is currently defined on the drm_connector via the >> >> hdr_output_metadata blob property. >> >> >> >> It might be useful to define per-plane hdr metadata, as different >> >> planes might have been mastered differently. >> > >> > I don't think this would directly help with the dynamic range blending >> > problem. You still need to establish the mapping between the optical >> > values from two different EOTFs and dynamic ranges. Or can you know >> > which optical values match the mastering display maximum and minimum >> > luminances for not-PQ? >> > >> >> My understanding of this is probably best illustrated by this example: >> >> Assume HDR was mastered on a display with a maximum white level of 500 >> nits and played back on a display that supports a max white level of >> 400 >> nits. If you know the mastering white level of 500 you know that this >> is >> the maximum value you need to compress down to 400 nits, allowing you >> to >> use the full extent of the 400 nits panel. > > Right, but in the kernel, where do you get these nits values from? > > hdr_output_metadata blob is infoframe data to the monitor. I think this > should be independent of the metadata used for color transformations in > the display pipeline before the monitor. > > EDID may tell us the monitor HDR metadata, but again what is used in > the color transformations should be independent, because EDIDs lie, > lighting environments change, and users have different preferences. > > What about black levels? > > Do you want to do black level adjustment? > > How exactly should the compression work? > > Where do you map the mid-tones? > > What if the end user wants something different? I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only. From a generic wayland compositor point of view this is uninteresting. I completely agree with what you said below though. I would even argue that all generic KMS abstract pipeline elements must have a well defined place in the pipeline and follow an exact specified formula. > >> If you do not know the mastering luminance is 500 nits you would >> have to compress 10,000 nits down to 400 (assuming PQ), losing quite >> a bit of the full 400 nits available as you'll need room to map the >> 500 >> to 10,000 nits range which in reality is completely unused. You might >> end >> up with mapping 500 nits to 350 nits, instead of mapping it to 400. > > The quality of the result depends on the compression (tone-mapping) > algorithm. I believe no-one will ever do a simple linear compression of > ranges. > > Instead, you probably do something smooth in the black end, keep > mid-tones roughly as they are, and the again do a smooth saturation to > some "reasonable" level that goes well with the monitor in the current > lighting environment without introducing coloring artifacts, and just > clip the huge overshoot of the full PQ-range. > > There are many big and small decisions to be made in how to map > out-of-gamut or out-of-brightness values into the displayable range, > and no algorithm fits all use cases. I believe this is why e.g. ICC > has several different "render intents", some of which are so vaguely > defined that they are practically undefined - just like what "a good > picture" means. You have essentially three dimensions: luminance, hue, > and saturation. Which one will you sacrifice, shift or emphasize and to > what degree to fit the square content peg into the round display hole? > > A naive example: Let's say content has 300 nits red. Your display can > show max 400 nits white or max 180 nits red, and anything in between. > What will you show? > > The problem is, that if UAPI does not define exactly what happens, then > taking advantage of these hardware capabilities means you have no idea > what happens to your content. This may be fine for closed systems, > where > the software has been carefully built for the specific hardware > revision and the use cases of the complete system have been > pre-defined, so that it can assume what should and will happen. But > should that be exposed as a generic UAPI, when generic userspace has no > chance of knowing what it will do? > > > ... > >> >> ## Displays and Tonemapping >> >> >> >> External displays are able to do their own tone and color mapping, >> >> based on the mastering luminance, color primaries, and white space >> >> defined in the HDR metadata. >> >> >> >> Internal panels (which are currently few and far between) usually >> >> don't include the complex HW to do tone and color mapping on their >> >> own and will require the display driver to perform appropriate >> >> mapping. >> > >> > FWIW, when designing Weston's color management, we are aiming for >> > the latter "simple" panels foremost, because that gives us full control >> > of all color conversions and tone mappings. >> > >> > OTOH, if Weston has to present to a display which only accepts e.g. >> > BT.2020/PQ signal, the display might always mangle the image in >> > unexpected ways. Therefore I expect that by default Weston will do >> > everything it can to try to make the display not apply anything magic >> > image enhancement: trust that EDID description of the display gamut and >> > dynamic range are correct, and use HDR metadata to tell the display >> > that those values are exactly what we are using. And we use them. >> > >> > IMO, a display doing its tone mapping magically is only useful when you >> > want to be able to use "simple" playback devices that cannot adapt to >> > the display they are driving. Magic tone mapping is also a way for >> > hardware vendors to differentiate, which from the color management >> > perspective is harmful as it makes it more difficult or impossible to >> > predict the display behaviour or to keep it consistent. >> > >> > So there are two opposing goals: >> > >> > - Traditional color management wants absolute control of the display, >> > leaving nothing unpredictable and preferably also nothing undefined. >> > Undefined behaviour can always be measured (profiled) which makes it >> > predictable and known. The viewing environment is controlled and >> > constant. >> > >> > - Entertainment wants the most visually impressive image quality by >> > dynamically adapting to both displayed content and to the viewing >> > environment conditions. >> > >> >> I feel much of the mess around HDR and CM is all the magic that >> everyone >> tries to do to differentiate themselves. I'm not a fan of that. >> >> One thing to watch out for is that the definitions for some of these >> things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair >> bit between HW vendors and we'd want to find a way to hide these >> complexities to KMS clients. > > How could we hide them? > > Actually, what do you mean with "hide"? > > > ... > >> >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc >> >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane >> >> shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR >> >> boost value. The drm_connector's hdr_output_metadata shall be set. >> > >> > This use case requires blending in KMS, so is the primary goal I >> > suppose. >> > >> >> Correct, my understanding is that the key here is to ensure blending >> in KMS >> is done in same blending space (linear RGB) as blending using GPU >> shaders and >> SDR boost is applied in the same fashion. > > Yes, but this is also something I think cannot be achieved by telling > the kernel about color spaces and letting the driver come up with a > color transformation. I believe this can only be achieved by telling > the kernel the color transformation directly, and also describing to > userspace what kind of color transformations are possible (1D LUT, 3D > LUT, other; precision...). > > ... > >> >> ## Summary of proposed interface changes >> >> >> >> per drm_plane: >> >> - new RGB color space definitions, mirroring the existing YUV color >> >> space definitions >> >> - new transfer function property >> >> - new SDR maximum white level property >> > >> > How will these new KMS properties interact with per-plane DEGAMMA, CTM >> > and/or GAMMA properties? >> > >> > Why go with your proposal instead of per-plane CTM and LUT? >> > >> > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs >> > both per-plane and on CRTC, would be: >> > >> > plane: >> > FB -> M1 -> LUT1 -> M2 -> blending input >> > >> > CRTC: >> > blending output -> LUT2 -> M3 -> connector >> > >> > FB: framebuffer >> > M1: matrix transform, capable of converting e.g. YCbCr to RGB >> > LUT1: 1D LUT for content EOTF, to produce light-linear RGB >> > M2: matrix transform for color space transformation >> > >> > LUT2: 1D LUT for applying monitor EOTF^-1 >> > M3: matrix transform, e.g. if you need to push YCbCr on the connector >> > >> > We also need to know where and how clipping happens. >> > >> > I think this scheme would allow implementing everything you want, and >> > it would not be tied to rigid enumerations, and it won't have any >> > magical conversions done under the hood as you would need to do to >> > convert from one enum space to another. It leaves the render intent to >> > be defined by the userspace compositor, rather than building a fixed >> > policy in the kernel. >> > >> > Userspace would be setting transformation operators, not color spaces, >> > to the kernel, allowing the blending space to be chosen by userspace. >> > In Weston we aim to choose then blending color space to be the same as >> > the output color space, except in optical (linear) encoding. The output >> > color space can be non-standard, e.g. measured with a display profiler >> > equipment. >> > >> > I would expect gamut mapping, dynamic range mapping and tone mapping to >> > be places where most experimentation and innovation happens, so >> > implementing them in the kernel with just few or no controllable >> > parameters at this time seems like it could become useless fast. >> > >> >> How important from a Wayland compositor perspective would it be to >> have fully >> configurable LUTs per plane, as opposed to providing KMS with a named >> transfer >> function, such as PQ or BT709, for LUT1 and LUT2? > > For Wayland desktop compositors, absolutely essential in my opinion. > > I would strongly advice against any desktop compositor from using the > "named transfer function" API and instead do everything in GPU shaders > if no other KMS way exists, unless the KMS UAPI definition gives the > exact mathematical formula of what that step will do. > > For entertainment devices using Wayland as an internal detail, who > cares. The system manufacturer makes things work any way they want, and > you cannot run non-curated apps on those devices, so there are no > compatibility issues. > > Then there would be systems between those two extremes, which would > likely use a mixture of those approaches, e.g. Kodi on bare hardware I > imagine (no Wayland there, but Wayland is kind of unrelated except for > the ideology). > > Mind, I'm not talking only about EOTFs here, but thinking of the whole > color management. EOTFs per se are simple. Tone-mapping, including > gamut > mapping and luminance mapping, are hard because there is no one correct > formula. > > However, if we are talking about *only* EOTFs, then an enumeration of > the curves could be usable. But you were also talking about configuring > dynamic ranges and doing luminance range mapping in the driver, which I > think won't work with enums, but could perhaps be accounted for in M2 > by userspace. Then again, if you have enums and not a LUT, you cannot > implement custom non-linear per-channel mapping which might be useful > for e.g. adapting SDR content for a HDR blending space. Or HDR content > to a different HDR blending space. > > See, a LUT can be more than just EOTF: it can apply a part of the > tone-mapping too. An enum cannot. > > I have a feeling that most tone-mapping curves would be non-linear with > some sort of S-shape and clipping of black and white ends. But that is > also a naive view, because tone-mapping is not a 1D-curve, it's a > mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do > 600 nit red for instance. On a 600 nit monitor you can do exactly one > color with 600 nits: white. > >> M1 on AMD's driver is currently hidden in our driver and essentially >> always >> set to BT709 at the moment, though we should look at color_encoding to >> pick >> the right M1. Are the color_encoding and color_range properties enough >> for >> what you'd like to achieve for M1? > > Probably yes, for standard format content, until a new standard comes > up > again and then you have to add another enum value in the UAPI. M1 is > probably the best place where an enum fits in the pipeline though, and > for M3 as well because non-RGB inputs to monitors are quite > standardised, and at those steps I would assume there are no parts of > tone-mapping done. > >> M3, currently the only matrix exposed by KMS (on drm_crtc) currently >> seems to be >> applied at M2 by AMD's driver. I need to dig into this a bit more. It >> would >> make sense to expose M2 and M3 as part of KMS (on drm_crtc and >> drm_plane). > > Good, it's really important to make sure that all drivers agree on the > pipeline layout that is defined(???) in the KMS UAPI specification. > >> I did a bunch of reading and thinking since your comments and created >> a v2 of >> the patchset. I don't think it's where we need it to be but I hope >> it's a >> step in the right direction. I will send it shortly. > > I have to say I'm still very sceptical about this enum-based approach > as you have noticed. :-) > > > Thanks, > pq
On 2021-05-17 12:48 p.m., Sebastian Wick wrote: > On 2021-05-17 10:57, Pekka Paalanen wrote: >> On Fri, 14 May 2021 17:05:11 -0400 >> Harry Wentland <harry.wentland@amd.com> wrote: >> >>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: >>> > On Mon, 26 Apr 2021 13:38:49 -0400 >>> > Harry Wentland <harry.wentland@amd.com> wrote: >> >> ... >> >>> >> ## Mastering Luminances >>> >> >>> >> Now we are able to use the PQ 2084 EOTF to define the luminance of >>> >> pixels in absolute terms. Unfortunately we're again presented with >>> >> physical limitations of the display technologies on the market >>> today. >>> >> Here are a few examples of luminance ranges of displays. >>> >> >>> >> | Display | Luminance range in nits | >>> >> | ------------------------ | ----------------------- | >>> >> | Typical PC display | 0.3 - 200 | >>> >> | Excellent LCD HDTV | 0.3 - 400 | >>> >> | HDR LCD w/ local dimming | 0.05 - 1,500 | >>> >> >>> >> Since no display can currently show the full 0.0005 to 10,000 nits >>> >> luminance range the display will need to tonemap the HDR content, >>> i.e >>> >> to fit the content within a display's capabilities. To assist with >>> >> tonemapping HDR content is usually accompanied with a metadata that >>> >> describes (among other things) the minimum and maximum mastering >>> >> luminance, i.e. the maximum and minimum luminance of the display >>> that >>> >> was used to master the HDR content. >>> >> >>> >> The HDR metadata is currently defined on the drm_connector via the >>> >> hdr_output_metadata blob property. >>> >> >>> >> It might be useful to define per-plane hdr metadata, as different >>> >> planes might have been mastered differently. >>> > >>> > I don't think this would directly help with the dynamic range >>> blending >>> > problem. You still need to establish the mapping between the optical >>> > values from two different EOTFs and dynamic ranges. Or can you know >>> > which optical values match the mastering display maximum and minimum >>> > luminances for not-PQ? >>> > >>> >>> My understanding of this is probably best illustrated by this example: >>> >>> Assume HDR was mastered on a display with a maximum white level of 500 >>> nits and played back on a display that supports a max white level of >>> 400 >>> nits. If you know the mastering white level of 500 you know that >>> this is >>> the maximum value you need to compress down to 400 nits, allowing >>> you to >>> use the full extent of the 400 nits panel. >> >> Right, but in the kernel, where do you get these nits values from? >> >> hdr_output_metadata blob is infoframe data to the monitor. I think this >> should be independent of the metadata used for color transformations in >> the display pipeline before the monitor. >> >> EDID may tell us the monitor HDR metadata, but again what is used in >> the color transformations should be independent, because EDIDs lie, >> lighting environments change, and users have different preferences. >> >> What about black levels? >> >> Do you want to do black level adjustment? >> >> How exactly should the compression work? >> >> Where do you map the mid-tones? >> >> What if the end user wants something different? > > I suspect that this is not about tone mapping at all. The use cases > listed always have the display in PQ mode and just assume that no > content exceeds the PQ limitations. Then you can simply bring all > content to the color space with a matrix multiplication and then map the > linear light content somewhere into the PQ range. Tone mapping is > performed in the display only. > > From a generic wayland compositor point of view this is uninteresting. > It a compositor's decision to provide or not the metadata property to the kernel. The metadata can be available from one or multiple clients or most likely not available at all. Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM occurs in display ) or NATIVE mode and do not attach any metadata to the connector and do TM in compositor. It is all about user preference or compositor design, or a combination of both options. > I completely agree with what you said below though. I would even argue > that all generic KMS abstract pipeline elements must have a well defined > place in the pipeline and follow an exact specified formula. > >> >>> If you do not know the mastering luminance is 500 nits you would >>> have to compress 10,000 nits down to 400 (assuming PQ), losing quite >>> a bit of the full 400 nits available as you'll need room to map the 500 >>> to 10,000 nits range which in reality is completely unused. You >>> might end >>> up with mapping 500 nits to 350 nits, instead of mapping it to 400. >> >> The quality of the result depends on the compression (tone-mapping) >> algorithm. I believe no-one will ever do a simple linear compression of >> ranges. >> >> Instead, you probably do something smooth in the black end, keep >> mid-tones roughly as they are, and the again do a smooth saturation to >> some "reasonable" level that goes well with the monitor in the current >> lighting environment without introducing coloring artifacts, and just >> clip the huge overshoot of the full PQ-range. >> >> There are many big and small decisions to be made in how to map >> out-of-gamut or out-of-brightness values into the displayable range, >> and no algorithm fits all use cases. I believe this is why e.g. ICC >> has several different "render intents", some of which are so vaguely >> defined that they are practically undefined - just like what "a good >> picture" means. You have essentially three dimensions: luminance, hue, >> and saturation. Which one will you sacrifice, shift or emphasize and to >> what degree to fit the square content peg into the round display hole? >> >> A naive example: Let's say content has 300 nits red. Your display can >> show max 400 nits white or max 180 nits red, and anything in between. >> What will you show? >> >> The problem is, that if UAPI does not define exactly what happens, then >> taking advantage of these hardware capabilities means you have no idea >> what happens to your content. This may be fine for closed systems, where >> the software has been carefully built for the specific hardware >> revision and the use cases of the complete system have been >> pre-defined, so that it can assume what should and will happen. But >> should that be exposed as a generic UAPI, when generic userspace has no >> chance of knowing what it will do? >> >> >> ... >> >>> >> ## Displays and Tonemapping >>> >> >>> >> External displays are able to do their own tone and color mapping, >>> >> based on the mastering luminance, color primaries, and white space >>> >> defined in the HDR metadata. >>> >> >>> >> Internal panels (which are currently few and far between) usually >>> >> don't include the complex HW to do tone and color mapping on their >>> >> own and will require the display driver to perform appropriate >>> >> mapping. >>> > >>> > FWIW, when designing Weston's color management, we are aiming for >>> > the latter "simple" panels foremost, because that gives us full >>> control >>> > of all color conversions and tone mappings. >>> > >>> > OTOH, if Weston has to present to a display which only accepts e.g. >>> > BT.2020/PQ signal, the display might always mangle the image in >>> > unexpected ways. Therefore I expect that by default Weston will do >>> > everything it can to try to make the display not apply anything magic >>> > image enhancement: trust that EDID description of the display >>> gamut and >>> > dynamic range are correct, and use HDR metadata to tell the display >>> > that those values are exactly what we are using. And we use them. >>> > >>> > IMO, a display doing its tone mapping magically is only useful >>> when you >>> > want to be able to use "simple" playback devices that cannot adapt to >>> > the display they are driving. Magic tone mapping is also a way for >>> > hardware vendors to differentiate, which from the color management >>> > perspective is harmful as it makes it more difficult or impossible to >>> > predict the display behaviour or to keep it consistent. >>> > >>> > So there are two opposing goals: >>> > >>> > - Traditional color management wants absolute control of the display, >>> > leaving nothing unpredictable and preferably also nothing >>> undefined. >>> > Undefined behaviour can always be measured (profiled) which >>> makes it >>> > predictable and known. The viewing environment is controlled and >>> > constant. >>> > >>> > - Entertainment wants the most visually impressive image quality by >>> > dynamically adapting to both displayed content and to the viewing >>> > environment conditions. >>> > >>> >>> I feel much of the mess around HDR and CM is all the magic that >>> everyone >>> tries to do to differentiate themselves. I'm not a fan of that. occures >>> >>> One thing to watch out for is that the definitions for some of these >>> things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair >>> bit between HW vendors and we'd want to find a way to hide these >>> complexities to KMS clients. >> >> How could we hide them? >> >> Actually, what do you mean with "hide"? >> >> >> ... >> >>> >> **One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc >>> >> desktop plane, with a P010 HDR video plane underlayed. The HDR plane >>> >> shall be PQ BT.2020 formatted. The desktop plane shall specify an >>> SDR >>> >> boost value. The drm_connector's hdr_output_metadata shall be set. >>> > >>> > This use case requires blending in KMS, so is the primary goal I >>> > suppose. >>> > >>> >>> Correct, my understanding is that the key here is to ensure blending >>> in KMS >>> is done in same blending space (linear RGB) as blending using GPU >>> shaders and >>> SDR boost is applied in the same fashion. >> >> Yes, but this is also something I think cannot be achieved by telling >> the kernel about color spaces and letting the driver come up with a >> color transformation. I believe this can only be achieved by telling >> the kernel the color transformation directly, and also describing to >> userspace what kind of color transformations are possible (1D LUT, 3D >> LUT, other; precision...). >> >> ... >> >>> >> ## Summary of proposed interface changes >>> >> >>> >> per drm_plane: >>> >> - new RGB color space definitions, mirroring the existing YUV color >>> >> space definitions >>> >> - new transfer function property >>> >> - new SDR maximum white level property >>> > >>> > How will these new KMS properties interact with per-plane DEGAMMA, >>> CTM >>> > and/or GAMMA properties? >>> > >>> > Why go with your proposal instead of per-plane CTM and LUT? >>> > >>> > I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs >>> > both per-plane and on CRTC, would be: >>> > >>> > plane: >>> > FB -> M1 -> LUT1 -> M2 -> blending input >>> > >>> > CRTC: >>> > blending output -> LUT2 -> M3 -> connector >>> > >>> > FB: framebuffer >>> > M1: matrix transform, capable of converting e.g. YCbCr to RGB >>> > LUT1: 1D LUT for content EOTF, to produce light-linear RGB >>> > M2: matrix transform for color space transformation >>> > >>> > LUT2: 1D LUT for applying monitor EOTF^-1 >>> > M3: matrix transform, e.g. if you need to push YCbCr on the connector >>> > >>> > We also need to know where and how clipping happens. >>> > >>> > I think this scheme would allow implementing everything you want, and >>> > it would not be tied to rigid enumerations, and it won't have any >>> > magical conversions done under the hood as you would need to do to >>> > convert from one enum space to another. It leaves the render >>> intent to >>> > be defined by the userspace compositor, rather than building a fixed >>> > policy in the kernel. >>> > >>> > Userspace would be setting transformation operators, not color >>> spaces, >>> > to the kernel, allowing the blending space to be chosen by userspace. >>> > In Weston we aim to choose then blending color space to be the >>> same as >>> > the output color space, except in optical (linear) encoding. The >>> output >>> > color space can be non-standard, e.g. measured with a display >>> profiler >>> > equipment. >>> > >>> > I would expect gamut mapping, dynamic range mapping and tone >>> mapping to >>> > be places where most experimentation and innovation happens, so >>> > implementing them in the kernel with just few or no controllable >>> > parameters at this time seems like it could become useless fast. >>> > >>> >>> How important from a Wayland compositor perspective would it be to >>> have fully >>> configurable LUTs per plane, as opposed to providing KMS with a >>> named transfer >>> function, such as PQ or BT709, for LUT1 and LUT2? >> >> For Wayland desktop compositors, absolutely essential in my opinion. >> >> I would strongly advice against any desktop compositor from using the >> "named transfer function" API and instead do everything in GPU shaders >> if no other KMS way exists, unless the KMS UAPI definition gives the >> exact mathematical formula of what that step will do. >> >> For entertainment devices using Wayland as an internal detail, who >> cares. The system manufacturer makes things work any way they want, and >> you cannot run non-curated apps on those devices, so there are no >> compatibility issues. >> >> Then there would be systems between those two extremes, which would >> likely use a mixture of those approaches, e.g. Kodi on bare hardware I >> imagine (no Wayland there, but Wayland is kind of unrelated except for >> the ideology). >> >> Mind, I'm not talking only about EOTFs here, but thinking of the whole >> color management. EOTFs per se are simple. Tone-mapping, including gamut >> mapping and luminance mapping, are hard because there is no one correct >> formula. >> >> However, if we are talking about *only* EOTFs, then an enumeration of >> the curves could be usable. But you were also talking about configuring >> dynamic ranges and doing luminance range mapping in the driver, which I >> think won't work with enums, but could perhaps be accounted for in M2 >> by userspace. Then again, if you have enums and not a LUT, you cannot >> implement custom non-linear per-channel mapping which might be useful >> for e.g. adapting SDR content for a HDR blending space. Or HDR content >> to a different HDR blending space. >> >> See, a LUT can be more than just EOTF: it can apply a part of the >> tone-mapping too. An enum cannot. >> >> I have a feeling that most tone-mapping curves would be non-linear with >> some sort of S-shape and clipping of black and white ends. But that is >> also a naive view, because tone-mapping is not a 1D-curve, it's a >> mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do >> 600 nit red for instance. On a 600 nit monitor you can do exactly one >> color with 600 nits: white. >> >>> M1 on AMD's driver is currently hidden in our driver and essentially >>> always >>> set to BT709 at the moment, though we should look at color_encoding >>> to pick >>> the right M1. Are the color_encoding and color_range properties >>> enough for >>> what you'd like to achieve for M1? >> >> Probably yes, for standard format content, until a new standard comes up >> again and then you have to add another enum value in the UAPI. M1 is >> probably the best place where an enum fits in the pipeline though, and >> for M3 as well because non-RGB inputs to monitors are quite >> standardised, and at those steps I would assume there are no parts of >> tone-mapping done. >> >>> M3, currently the only matrix exposed by KMS (on drm_crtc) currently >>> seems to be >>> applied at M2 by AMD's driver. I need to dig into this a bit more. >>> It would >>> make sense to expose M2 and M3 as part of KMS (on drm_crtc and >>> drm_plane). >> >> Good, it's really important to make sure that all drivers agree on the >> pipeline layout that is defined(???) in the KMS UAPI specification. >> >>> I did a bunch of reading and thinking since your comments and >>> created a v2 of >>> the patchset. I don't think it's where we need it to be but I hope >>> it's a >>> step in the right direction. I will send it shortly. >> >> I have to say I'm still very sceptical about this enum-based approach >> as you have noticed. :-) >> >> >> Thanks, >> pq
On Mon, 17 May 2021 15:39:03 -0400 Vitaly Prosyak <vitaly.prosyak@amd.com> wrote: > On 2021-05-17 12:48 p.m., Sebastian Wick wrote: > > On 2021-05-17 10:57, Pekka Paalanen wrote: > >> On Fri, 14 May 2021 17:05:11 -0400 > >> Harry Wentland <harry.wentland@amd.com> wrote: > >> > >>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: > >>> > On Mon, 26 Apr 2021 13:38:49 -0400 > >>> > Harry Wentland <harry.wentland@amd.com> wrote: > >> > >> ... > >> > >>> >> ## Mastering Luminances > >>> >> > >>> >> Now we are able to use the PQ 2084 EOTF to define the luminance of > >>> >> pixels in absolute terms. Unfortunately we're again presented with > >>> >> physical limitations of the display technologies on the market > >>> today. > >>> >> Here are a few examples of luminance ranges of displays. > >>> >> > >>> >> | Display | Luminance range in nits | > >>> >> | ------------------------ | ----------------------- | > >>> >> | Typical PC display | 0.3 - 200 | > >>> >> | Excellent LCD HDTV | 0.3 - 400 | > >>> >> | HDR LCD w/ local dimming | 0.05 - 1,500 | > >>> >> > >>> >> Since no display can currently show the full 0.0005 to 10,000 nits > >>> >> luminance range the display will need to tonemap the HDR content, > >>> i.e > >>> >> to fit the content within a display's capabilities. To assist with > >>> >> tonemapping HDR content is usually accompanied with a metadata that > >>> >> describes (among other things) the minimum and maximum mastering > >>> >> luminance, i.e. the maximum and minimum luminance of the display > >>> that > >>> >> was used to master the HDR content. > >>> >> > >>> >> The HDR metadata is currently defined on the drm_connector via the > >>> >> hdr_output_metadata blob property. > >>> >> > >>> >> It might be useful to define per-plane hdr metadata, as different > >>> >> planes might have been mastered differently. > >>> > > >>> > I don't think this would directly help with the dynamic range > >>> blending > >>> > problem. You still need to establish the mapping between the optical > >>> > values from two different EOTFs and dynamic ranges. Or can you know > >>> > which optical values match the mastering display maximum and minimum > >>> > luminances for not-PQ? > >>> > > >>> > >>> My understanding of this is probably best illustrated by this example: > >>> > >>> Assume HDR was mastered on a display with a maximum white level of 500 > >>> nits and played back on a display that supports a max white level of > >>> 400 > >>> nits. If you know the mastering white level of 500 you know that > >>> this is > >>> the maximum value you need to compress down to 400 nits, allowing > >>> you to > >>> use the full extent of the 400 nits panel. > >> > >> Right, but in the kernel, where do you get these nits values from? > >> > >> hdr_output_metadata blob is infoframe data to the monitor. I think this > >> should be independent of the metadata used for color transformations in > >> the display pipeline before the monitor. > >> > >> EDID may tell us the monitor HDR metadata, but again what is used in > >> the color transformations should be independent, because EDIDs lie, > >> lighting environments change, and users have different preferences. > >> > >> What about black levels? > >> > >> Do you want to do black level adjustment? > >> > >> How exactly should the compression work? > >> > >> Where do you map the mid-tones? > >> > >> What if the end user wants something different? > > > > I suspect that this is not about tone mapping at all. The use cases > > listed always have the display in PQ mode and just assume that no > > content exceeds the PQ limitations. Then you can simply bring all > > content to the color space with a matrix multiplication and then map the > > linear light content somewhere into the PQ range. Tone mapping is > > performed in the display only. The use cases do use the word "desktop" though. Harry, could you expand on this, are you seeking a design that is good for generic desktop compositors too, or one that is more tailored to "embedded" video player systems taking the most advantage of (potentially fixed-function) hardware? What matrix would one choose? Which render intent would it correspond to? If you need to adapt different dynamic ranges into the blending dynamic range, would a simple linear transformation really be enough? > > From a generic wayland compositor point of view this is uninteresting. > > > It a compositor's decision to provide or not the metadata property to > the kernel. The metadata can be available from one or multiple clients > or most likely not available at all. > > Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM > occurs in display ) or NATIVE mode and do not attach any metadata to the > connector and do TM in compositor. > > It is all about user preference or compositor design, or a combination > of both options. Indeed. The thing here is that you cannot just add KMS UAPI, you also need to have the FOSS userspace to go with it. So you need to have your audience defined, userspace patches written and reviewed and agreed to be a good idea. I'm afraid this particular UAPI design would be difficult to justify with Weston. Maybe Kodi is a better audience? But then again, one also needs to consider whether it is enough for a new UAPI to satisfy only part of the possible audience and then need yet another new UAPI to satisfy the rest. Adding new UAPI requires defining the interactions with all existing UAPI as well. Maybe we do need several different UAPIs for the "same" things if the hardware designs are too different to cater with just one. Thanks, pq
On 2021-05-18 3:56 a.m., Pekka Paalanen wrote: > On Mon, 17 May 2021 15:39:03 -0400 > Vitaly Prosyak <vitaly.prosyak@amd.com> wrote: > >> On 2021-05-17 12:48 p.m., Sebastian Wick wrote: >>> On 2021-05-17 10:57, Pekka Paalanen wrote: >>>> On Fri, 14 May 2021 17:05:11 -0400 >>>> Harry Wentland <harry.wentland@amd.com> wrote: >>>> >>>>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: >>>>>> On Mon, 26 Apr 2021 13:38:49 -0400 >>>>>> Harry Wentland <harry.wentland@amd.com> wrote: >>>> >>>> ... >>>> >>>>>>> ## Mastering Luminances >>>>>>> >>>>>>> Now we are able to use the PQ 2084 EOTF to define the luminance of >>>>>>> pixels in absolute terms. Unfortunately we're again presented with >>>>>>> physical limitations of the display technologies on the market >>>>> today. >>>>>>> Here are a few examples of luminance ranges of displays. >>>>>>> >>>>>>> | Display | Luminance range in nits | >>>>>>> | ------------------------ | ----------------------- | >>>>>>> | Typical PC display | 0.3 - 200 | >>>>>>> | Excellent LCD HDTV | 0.3 - 400 | >>>>>>> | HDR LCD w/ local dimming | 0.05 - 1,500 | >>>>>>> >>>>>>> Since no display can currently show the full 0.0005 to 10,000 nits >>>>>>> luminance range the display will need to tonemap the HDR content, >>>>> i.e >>>>>>> to fit the content within a display's capabilities. To assist with >>>>>>> tonemapping HDR content is usually accompanied with a metadata that >>>>>>> describes (among other things) the minimum and maximum mastering >>>>>>> luminance, i.e. the maximum and minimum luminance of the display >>>>> that >>>>>>> was used to master the HDR content. >>>>>>> >>>>>>> The HDR metadata is currently defined on the drm_connector via the >>>>>>> hdr_output_metadata blob property. >>>>>>> >>>>>>> It might be useful to define per-plane hdr metadata, as different >>>>>>> planes might have been mastered differently. >>>>>> >>>>>> I don't think this would directly help with the dynamic range >>>>> blending >>>>>> problem. You still need to establish the mapping between the optical >>>>>> values from two different EOTFs and dynamic ranges. Or can you know >>>>>> which optical values match the mastering display maximum and minimum >>>>>> luminances for not-PQ? >>>>>> >>>>> >>>>> My understanding of this is probably best illustrated by this example: >>>>> >>>>> Assume HDR was mastered on a display with a maximum white level of 500 >>>>> nits and played back on a display that supports a max white level of >>>>> 400 >>>>> nits. If you know the mastering white level of 500 you know that >>>>> this is >>>>> the maximum value you need to compress down to 400 nits, allowing >>>>> you to >>>>> use the full extent of the 400 nits panel. >>>> >>>> Right, but in the kernel, where do you get these nits values from? >>>> >>>> hdr_output_metadata blob is infoframe data to the monitor. I think this >>>> should be independent of the metadata used for color transformations in >>>> the display pipeline before the monitor. >>>> >>>> EDID may tell us the monitor HDR metadata, but again what is used in >>>> the color transformations should be independent, because EDIDs lie, >>>> lighting environments change, and users have different preferences. >>>> >>>> What about black levels? >>>> >>>> Do you want to do black level adjustment? >>>> >>>> How exactly should the compression work? >>>> >>>> Where do you map the mid-tones? >>>> >>>> What if the end user wants something different? >>> >>> I suspect that this is not about tone mapping at all. The use cases >>> listed always have the display in PQ mode and just assume that no >>> content exceeds the PQ limitations. Then you can simply bring all >>> content to the color space with a matrix multiplication and then map the >>> linear light content somewhere into the PQ range. Tone mapping is >>> performed in the display only. > > The use cases do use the word "desktop" though. Harry, could you expand > on this, are you seeking a design that is good for generic desktop > compositors too, or one that is more tailored to "embedded" video > player systems taking the most advantage of (potentially > fixed-function) hardware? > The goal is to enable this on a generic desktop, such as generic Wayland implementations or ChromeOS. We're not looking for a custom solution for some embedded systems, though the solution we end up with should obviously not prevent an implementation on embedded video players. > What matrix would one choose? Which render intent would it > correspond to? > > If you need to adapt different dynamic ranges into the blending dynamic > range, would a simple linear transformation really be enough? > >>> From a generic wayland compositor point of view this is uninteresting. >>> >> It a compositor's decision to provide or not the metadata property to >> the kernel. The metadata can be available from one or multiple clients >> or most likely not available at all. >> >> Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM >> occurs in display ) or NATIVE mode and do not attach any metadata to the >> connector and do TM in compositor. >> >> It is all about user preference or compositor design, or a combination >> of both options. > > Indeed. The thing here is that you cannot just add KMS UAPI, you also > need to have the FOSS userspace to go with it. So you need to have your > audience defined, userspace patches written and reviewed and agreed > to be a good idea. I'm afraid this particular UAPI design would be > difficult to justify with Weston. Maybe Kodi is a better audience? > I'm not sure designing a UAPI for Kodi that's not going to work for Wayland-compositors is the right thing. From a KMS driver maintainer standpoint I don't want an API for each userspace. The idea here is to do design and discussion in public so we can eventually arrive at a UAPI for HDR and CM that works for Wayland and by extension for every other userspace. > But then again, one also needs to consider whether it is enough for a > new UAPI to satisfy only part of the possible audience and then need > yet another new UAPI to satisfy the rest. Adding new UAPI requires > defining the interactions with all existing UAPI as well. > > Maybe we do need several different UAPIs for the "same" things if the > hardware designs are too different to cater with just one. > I feel we should have a section in the RFC that sketches how different HW deals with this currently. It would be good if we can arrive at a UAPI that captures at least the common functionality of various HW. Harry > > Thanks, > pq >
On 2021-05-18 16:19, Harry Wentland wrote: > On 2021-05-18 3:56 a.m., Pekka Paalanen wrote: >> On Mon, 17 May 2021 15:39:03 -0400 >> Vitaly Prosyak <vitaly.prosyak@amd.com> wrote: >> >>> On 2021-05-17 12:48 p.m., Sebastian Wick wrote: >>>> On 2021-05-17 10:57, Pekka Paalanen wrote: >>>>> On Fri, 14 May 2021 17:05:11 -0400 >>>>> Harry Wentland <harry.wentland@amd.com> wrote: >>>>> >>>>>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: >>>>>>> On Mon, 26 Apr 2021 13:38:49 -0400 >>>>>>> Harry Wentland <harry.wentland@amd.com> wrote: >>>>> >>>>> ... >>>>> >>>>>>>> ## Mastering Luminances >>>>>>>> >>>>>>>> Now we are able to use the PQ 2084 EOTF to define the luminance >>>>>>>> of >>>>>>>> pixels in absolute terms. Unfortunately we're again presented >>>>>>>> with >>>>>>>> physical limitations of the display technologies on the market >>>>>> today. >>>>>>>> Here are a few examples of luminance ranges of displays. >>>>>>>> >>>>>>>> | Display | Luminance range in nits | >>>>>>>> | ------------------------ | ----------------------- | >>>>>>>> | Typical PC display | 0.3 - 200 | >>>>>>>> | Excellent LCD HDTV | 0.3 - 400 | >>>>>>>> | HDR LCD w/ local dimming | 0.05 - 1,500 | >>>>>>>> >>>>>>>> Since no display can currently show the full 0.0005 to 10,000 >>>>>>>> nits >>>>>>>> luminance range the display will need to tonemap the HDR >>>>>>>> content, >>>>>> i.e >>>>>>>> to fit the content within a display's capabilities. To assist >>>>>>>> with >>>>>>>> tonemapping HDR content is usually accompanied with a metadata >>>>>>>> that >>>>>>>> describes (among other things) the minimum and maximum mastering >>>>>>>> luminance, i.e. the maximum and minimum luminance of the display >>>>>> that >>>>>>>> was used to master the HDR content. >>>>>>>> >>>>>>>> The HDR metadata is currently defined on the drm_connector via >>>>>>>> the >>>>>>>> hdr_output_metadata blob property. >>>>>>>> >>>>>>>> It might be useful to define per-plane hdr metadata, as >>>>>>>> different >>>>>>>> planes might have been mastered differently. >>>>>>> >>>>>>> I don't think this would directly help with the dynamic range >>>>>> blending >>>>>>> problem. You still need to establish the mapping between the >>>>>>> optical >>>>>>> values from two different EOTFs and dynamic ranges. Or can you >>>>>>> know >>>>>>> which optical values match the mastering display maximum and >>>>>>> minimum >>>>>>> luminances for not-PQ? >>>>>>> >>>>>> >>>>>> My understanding of this is probably best illustrated by this >>>>>> example: >>>>>> >>>>>> Assume HDR was mastered on a display with a maximum white level of >>>>>> 500 >>>>>> nits and played back on a display that supports a max white level >>>>>> of >>>>>> 400 >>>>>> nits. If you know the mastering white level of 500 you know that >>>>>> this is >>>>>> the maximum value you need to compress down to 400 nits, allowing >>>>>> you to >>>>>> use the full extent of the 400 nits panel. >>>>> >>>>> Right, but in the kernel, where do you get these nits values from? >>>>> >>>>> hdr_output_metadata blob is infoframe data to the monitor. I think >>>>> this >>>>> should be independent of the metadata used for color >>>>> transformations in >>>>> the display pipeline before the monitor. >>>>> >>>>> EDID may tell us the monitor HDR metadata, but again what is used >>>>> in >>>>> the color transformations should be independent, because EDIDs lie, >>>>> lighting environments change, and users have different preferences. >>>>> >>>>> What about black levels? >>>>> >>>>> Do you want to do black level adjustment? >>>>> >>>>> How exactly should the compression work? >>>>> >>>>> Where do you map the mid-tones? >>>>> >>>>> What if the end user wants something different? >>>> >>>> I suspect that this is not about tone mapping at all. The use cases >>>> listed always have the display in PQ mode and just assume that no >>>> content exceeds the PQ limitations. Then you can simply bring all >>>> content to the color space with a matrix multiplication and then map >>>> the >>>> linear light content somewhere into the PQ range. Tone mapping is >>>> performed in the display only. >> >> The use cases do use the word "desktop" though. Harry, could you >> expand >> on this, are you seeking a design that is good for generic desktop >> compositors too, or one that is more tailored to "embedded" video >> player systems taking the most advantage of (potentially >> fixed-function) hardware? >> > > The goal is to enable this on a generic desktop, such as generic > Wayland > implementations or ChromeOS. We're not looking for a custom solution > for > some embedded systems, though the solution we end up with should > obviously > not prevent an implementation on embedded video players. > >> What matrix would one choose? Which render intent would it >> correspond to? >> >> If you need to adapt different dynamic ranges into the blending >> dynamic >> range, would a simple linear transformation really be enough? >> >>>> From a generic wayland compositor point of view this is >>>> uninteresting. >>>> >>> It a compositor's decision to provide or not the metadata property to >>> the kernel. The metadata can be available from one or multiple >>> clients >>> or most likely not available at all. >>> >>> Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM >>> occurs in display ) or NATIVE mode and do not attach any metadata to >>> the >>> connector and do TM in compositor. >>> >>> It is all about user preference or compositor design, or a >>> combination >>> of both options. >> >> Indeed. The thing here is that you cannot just add KMS UAPI, you also >> need to have the FOSS userspace to go with it. So you need to have >> your >> audience defined, userspace patches written and reviewed and agreed >> to be a good idea. I'm afraid this particular UAPI design would be >> difficult to justify with Weston. Maybe Kodi is a better audience? >> > > I'm not sure designing a UAPI for Kodi that's not going to work for > Wayland-compositors is the right thing. From a KMS driver maintainer > standpoint I don't want an API for each userspace. > > The idea here is to do design and discussion in public so we can > eventually > arrive at a UAPI for HDR and CM that works for Wayland and by extension > for every other userspace. Eventually we want to be able to drive displays in PQ mode in weston (I think?) where the TF property could be used in some cases. So that property seems good to me. The SDR boost property then might also be useful but it depends on the details of the formula/mechanism that you can come up with. But I want to stress again that we're going to drive all display in their native color space and dynamic range if possible where none of those properties discussed are useful and without some kind of 3D LUT in the plane's pixel pipeline we won't be able to make use of them at all. The Kodi folks can probably give you better feedback and an actual implementation in reasonable time because they want to drive the display in PQ mode. >> But then again, one also needs to consider whether it is enough for a >> new UAPI to satisfy only part of the possible audience and then need >> yet another new UAPI to satisfy the rest. Adding new UAPI requires >> defining the interactions with all existing UAPI as well. >> >> Maybe we do need several different UAPIs for the "same" things if the >> hardware designs are too different to cater with just one. >> > > I feel we should have a section in the RFC that sketches how different > HW > deals with this currently. It would be good if we can arrive at a UAPI > that > captures at least the common functionality of various HW. > > Harry > >> >> Thanks, >> pq >>
On Tue, 18 May 2021 10:19:25 -0400 Harry Wentland <harry.wentland@amd.com> wrote: > On 2021-05-18 3:56 a.m., Pekka Paalanen wrote: > > On Mon, 17 May 2021 15:39:03 -0400 > > Vitaly Prosyak <vitaly.prosyak@amd.com> wrote: > > > >> On 2021-05-17 12:48 p.m., Sebastian Wick wrote: ... > >>> I suspect that this is not about tone mapping at all. The use cases > >>> listed always have the display in PQ mode and just assume that no > >>> content exceeds the PQ limitations. Then you can simply bring all > >>> content to the color space with a matrix multiplication and then map the > >>> linear light content somewhere into the PQ range. Tone mapping is > >>> performed in the display only. > > > > The use cases do use the word "desktop" though. Harry, could you expand > > on this, are you seeking a design that is good for generic desktop > > compositors too, or one that is more tailored to "embedded" video > > player systems taking the most advantage of (potentially > > fixed-function) hardware? > > > > The goal is to enable this on a generic desktop, such as generic Wayland > implementations or ChromeOS. We're not looking for a custom solution for > some embedded systems, though the solution we end up with should obviously > not prevent an implementation on embedded video players. (There is a TL;DR: at the end.) Echoing a little bit what Sebastian already said, I believe there are two sides to this again: - color management in the traditional sense - modern standardised display technology It was perhaps too harsh to say that generic Wayland compositors cannot use enum-based color-related UAPI. Sometimes they could, sometimes it won't be good enough. Traditional color management assumes that no two monitors are the same, even if they are the same make, model, and manufacturing batch, and are driven exactly the same way. Hence, all monitors may require calibration (adjusting monitor knobs), and/or they may require profiling (measuring the light emission with a special hardware device designed for that). Also the viewing environment has an effect. For profiling to be at all meaningful, calibration must be fixed. This means that there must be no dynamic on-the-fly adaptation done in the monitor, in the display hardware, or in the kernel. That is a tall order that I guess is going to be less and less achievable, especially with HDR monitors. The other side is where the end user trusts the standards, and trusts that the drivers and the hardware do what they are specified to do. This is where you can trust that the monitor does the tone-mapping magic right. Weston needs to support both approaches, because we want to prove our new approach to traditional color management, but we also want to support HDR, and if possible, do both at the same time. Doing both at the same time is what we think foremost, because it's also the hardest thing to achieve. If that can be done, then everything else works out too. However, this should not exclude the possibility to trust standards and monitor magic, when the end user wants it. It's also possible that a monitor simply doesn't support a mode that would enable fully color managed HDR, so Weston will need to be able to drive monitors with e.g. BT.2020/PQ data eventually. It's just not the first goal we have. This debate is a little bit ironic. The Wayland approach to traditional color management is that end users should trust the display server to do the right thing, where before people only trusted the individual apps using a specific CMS implementation. The display server was the untrusted one that should just get out of the way and not touch anything. Now I'm arguing that I don't want to trust monitor magic, who knows what atrocities it does to my picture! But take the next logical step, and one would be arguing that end users should trust also monitors to do the right thing. :-) The above has two catches: - Do you actually trust hardware manufacturers and marketers and EDID? Monitors have secret sauce you can't inspect nor change. - You feed a single video stream to a monitor, in a single format, encoding and color space. The display server OTOH gets an arbitrary number of input video streams in arbitrary formats, encodings, and color spaces, and it needs to composite them into one. Composition is hard. It's not enough to know what kind of signals you take in and what kind of signal you must output. You also need to know what the end user wants from the result: the render intent. Even if we trust the monitor magic to do the right thing in interpreting and displaying our output signal, we still need to know what the end user wants from the composition, and we need to control the composition formula to achieve that. TL;DR: I would summarise my comments so far into these: - Telling the kernel the color spaces and letting it come up with whatever color transformation formula from those is not enough, because it puts the render intent policy decision in the kernel. - Telling the kernel what color transformations need to be done is good, if it is clearly defined. - Using an enum-based UAPI to tell the kernel what color transformations needs to be done (e.g. which EOTF or EOTF^-1 to apply at a step in the abstract pipeline) is very likely ok for many Wayland compositors in most cases, but may not be sufficient for all use cases. Of course, one is always bound by what hardware can do, so not a big deal. - You may need to define mutually exclusive KMS properties (referring to my email in another branch of this email tree). - I'm not sure I (we?) can meaningfully review things like "SDR boost" property until we know ourselves how to composite different types of content together. Maybe someone else could. Does this help or raise thoughts? The work on Weston CM&HDR right now is aiming to get it up to a point where we can start nicely testing different compositing approaches and methods and parameters, and I expect that will also feed back into the Wayland CM&HDR protocol design as well. Thanks, pq
On Wed, 19 May 2021 11:53:37 +0300 Pekka Paalanen <ppaalanen@gmail.com> wrote: ... > TL;DR: > > I would summarise my comments so far into these: > > - Telling the kernel the color spaces and letting it come up with > whatever color transformation formula from those is not enough, > because it puts the render intent policy decision in the kernel. > > - Telling the kernel what color transformations need to be done is > good, if it is clearly defined. > > - Using an enum-based UAPI to tell the kernel what color > transformations needs to be done (e.g. which EOTF or EOTF^-1 to apply > at a step in the abstract pipeline) is very likely ok for many > Wayland compositors in most cases, but may not be sufficient for all > use cases. Of course, one is always bound by what hardware can do, so > not a big deal. > > - You may need to define mutually exclusive KMS properties (referring > to my email in another branch of this email tree). > > - I'm not sure I (we?) can meaningfully review things like "SDR boost" > property until we know ourselves how to composite different types of > content together. Maybe someone else could. > > Does this help or raise thoughts? > > The work on Weston CM&HDR right now is aiming to get it up to a point > where we can start nicely testing different compositing approaches and > methods and parameters, and I expect that will also feed back into the > Wayland CM&HDR protocol design as well. I have forgot to mention one important thing: Generic Wayland compositors will be using KMS planes opportunistically. The compositor will be switching between GL and KMS compositing on-demand, refresh by refresh. This means that both GL and KMS compositing must produce identical results, or users will be seeing "color flicks" on switch. This is a practical reason why we really want to know in full detail how the KMS pipeline processes pixels. Thanks, pq