[RFC,01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed

Commit Message

Harry Wentland Sept. 8, 2023, 3:02 p.m. UTC

Signed-off-by: Harry Wentland <[e-mail]>
Cc: Ville Syrjala <[e-mail]>
Cc: Pekka Paalanen <[e-mail]>
Cc: Simon Ser <[e-mail]>
Cc: Harry Wentland <[e-mail]>
Cc: Melissa Wen <[e-mail]>
Cc: Jonas Ådahl <[e-mail]>
Cc: Sebastian Wick <[e-mail]>
Cc: Shashank Sharma <[e-mail]>
Cc: Alexander Goins <[e-mail]>
Cc: Joshua Ashton <[e-mail]>
Cc: Michel Dänzer <[e-mail]>
Cc: Aleix Pol <[e-mail]>
Cc: Xaver Hugl <[e-mail]>
Cc: Victoria Brekenfeld <[e-mail]>
Cc: Daniel Vetter <[e-mail]>
Cc: Uma Shankar <[e-mail]>
Cc: Naseer Ahmed <[e-mail]>
Cc: Christopher Braga <[e-mail]>
---
 Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
 1 file changed, 278 insertions(+)
 create mode 100644 Documentation/gpu/rfc/color_pipeline.rst

Comments

Sebastian Wick Sept. 8, 2023, 7:30 p.m. UTC | #1

Hey Harry,

Thank you and Simon for this great document. Really happy about it, but
obviously I've got a few notes and questions inline.

On Fri, Sep 08, 2023 at 11:02:26AM -0400, Harry Wentland wrote:
> Signed-off-by: Harry Wentland <[e-mail]>
> Cc: Ville Syrjala <[e-mail]>
> Cc: Pekka Paalanen <[e-mail]>
> Cc: Simon Ser <[e-mail]>
> Cc: Harry Wentland <[e-mail]>
> Cc: Melissa Wen <[e-mail]>
> Cc: Jonas Ådahl <[e-mail]>
> Cc: Sebastian Wick <[e-mail]>
> Cc: Shashank Sharma <[e-mail]>
> Cc: Alexander Goins <[e-mail]>
> Cc: Joshua Ashton <[e-mail]>
> Cc: Michel Dänzer <[e-mail]>
> Cc: Aleix Pol <[e-mail]>
> Cc: Xaver Hugl <[e-mail]>
> Cc: Victoria Brekenfeld <[e-mail]>
> Cc: Daniel Vetter <[e-mail]>
> Cc: Uma Shankar <[e-mail]>
> Cc: Naseer Ahmed <[e-mail]>
> Cc: Christopher Braga <[e-mail]>
> ---
>  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
>  1 file changed, 278 insertions(+)
>  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> 
> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
> new file mode 100644
> index 000000000000..bfa4a8f12087
> --- /dev/null
> +++ b/Documentation/gpu/rfc/color_pipeline.rst
> @@ -0,0 +1,278 @@
> +========================
> +Linux Color Pipeline API
> +========================
> +
> +What problem are we solving?
> +============================
> +
> +We would like to support pre-, and post-blending complex color transformations
> +in order to allow for HW-supported HDR use-cases, as well as to provide support
> +to color-managed applications, such as video or image editors.
> +
> +While it is possible to support an HDR output on HW supporting the Colorspace
> +and HDR Metadata drm_connector properties that requires the compositor or
> +application to render and compose the content into one final buffer intended for
> +display. Doing so is costly.
> +
> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
> +operations to support color transformations. These operations are often
> +implemented in fixed-function HW and therefore much more power efficient than
> +performing similar operations via shaders or CPU.
> +
> +We would like to make use of this HW functionality to support complex color
> +transformations with no, or minimal CPU or shader load.
> +
> +
> +How are other OSes solving this problem?
> +========================================
> +
> +The most widely supported use-cases regard HDR content, whether video or
> +gaming.
> +
> +Most OSes will specify the source content format (color gamut, encoding transfer
> +function, and other metadata, such as max and average light levels) to a driver.
> +Drivers will then program their fixed-function HW accordingly to map from a
> +source content buffer's space to a display's space.
> +
> +When fixed-function HW is not available the compositor will assemble a shader to
> +ask the GPU to perform the transformation from the source content format to the
> +display's format.
> +
> +A compositor's mapping function and a driver's mapping function are usually
> +entirely separate concepts. On OSes where a HW vendor has no insight into
> +closed-source compositor code such a vendor will tune their color management
> +code to visually match the compositor's. On other OSes, where both mapping
> +functions are open to an implementer they will ensure both mappings match.
> +
> +
> +Why is Linux different?
> +=======================
> +
> +Unlike other OSes, where there is one compositor for one or more drivers, on
> +Linux we have a many-to-many relationship. Many compositors; many drivers.
> +In addition each compositor vendor or community has their own view of how
> +color management should be done. This is what makes Linux so beautiful.
> +
> +This means that a HW vendor can now no longer tune their driver to one
> +compositor, as tuning it to one will almost inevitably make it look very
> +different from another compositor's color mapping.
> +
> +We need a better solution.
> +
> +
> +Descriptive API
> +===============
> +
> +An API that describes the source and destination colorspaces is a descriptive
> +API. It describes the input and output color spaces but does not describe
> +how precisely they should be mapped. Such a mapping includes many minute
> +design decision that can greatly affect the look of the final result.
> +
> +It is not feasible to describe such mapping with enough detail to ensure the
> +same result from each implementation. In fact, these mappings are a very active
> +research area.
> +
> +
> +Prescriptive API
> +================
> +
> +A prescriptive API describes not the source and destination colorspaces. It
> +instead prescribes a recipe for how to manipulate pixel values to arrive at the
> +desired outcome.
> +
> +This recipe is generally an order straight-forward operations, with clear
> +mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
> +operations that can be described in a precise manner.
> +
> +
> +The Color Pipeline API
> +======================
> +
> +HW color management pipelines can significantly differ between HW
> +vendors in terms of availability, ordering, and capabilities of HW
> +blocks. This makes a common definition of color management blocks and
> +their ordering nigh impossible. Instead we are defining an API that
> +allows user space to discover the HW capabilities.
> +
> +
> +drm_colorop Object & IOCTLs
> +===========================
> +
> +To support the definition of color pipelines we introduce a new DRM core
> +object, a drm_colorop. Individual drm_colorop objects will be chained
> +via the NEXT property of a drm_colorop to constitute a color pipeline.
> +Each drm_colorop object is unique, i.e., even if multiple color
> +pipelines have the same operation they won't share the same drm_colorop
> +object to describe that operation.
> +
> +Just like other DRM objects the drm_colorop objects are discovered via
> +IOCTLs:
> +
> +DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
> +number of all drm_colorop objects.
> +
> +DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
> +It includes the ID for the colorop object, as well as the plane_id of
> +the associated plane. All other values should be registered as
> +properties.
> +
> +Each drm_colorop has three core properties:

We talked a bit about how forwards compatibility will look like. When a
driver introduces a new property that user space doesn't understand yet,
for user space to make use of the pipeline, the property either has to
have a bypass property, or some other kind of standard/core property
that promises that this property is purely informational and the
existance doesn't change the result of the color pipeline. Something
like "INFORMATIONAL" or "INFO" (better suggestions welcome!).

If a property doesn't fall into the two categories, the new pipeline
would be unusable for user space. So, if this new property is added on
an existing piece of hardware, it should be done on an entirely new
pipeline to avoid user space regressions.

I believe these compatibility considerations are very important to
document.

> +
> +TYPE: The type of transformation, such as
> +* enumerated curve
> +* custom (uniform) 1D LUT
> +* 3x3 matrix
> +* 3x4 matrix
> +* 3D LUT
> +* etc.
> +
> +Depending on the type of transformation other properties will describe
> +more details.
> +
> +BYPASS: A boolean property that can be used to easily put a block into
> +bypass mode. While setting other properties might fail atomic check,
> +setting the BYPASS property to true should never fail. This allows DRM
> +clients to fallback to other methods of color management if an atomic
> +check for KMS color operations fails.

This says BYPASS is a prop but in the example below the type property is
a mutable enum with Bypass being an enum variant.

Any reason why you chose to make the type property mutable instead of a
separate BYPASS property? Personally I like the simplicity of immutable
properties and would favor a BYPASS property.

> +
> +NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
> +drm_colorop is the last in the chain.
> +
> +An example of a drm_colorop object might look like one of these::
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> +    ├─ "lut_size": immutable range = 33
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +
> +COLOR_PIPELINE Plane Property
> +=============================
> +
> +Because we don't have existing KMS color properties in the pre-blending
> +portion of display pipelines (i.e. on drm_planes) we are introducing
> +color pipelines here first. Eventually we'll want to use the same
> +concept for the post-blending portion, i.e. drm_crtcs.
> +
> +Color Pipelines are created by a driver and advertised via a new
> +COLOR_PIPELINE enum property on each plane. Values of the property
> +always include '0', which is the default and means all color processing
> +is disabled. Additional values will be the object IDs of the first

Regarding the requirement to never fail atomic checks which set a
colorop to bypass: I think for user space it would be sufficient if
setting the entire pipeline to bypass would not fail the atomic check.

I'm also wondering if there is/will be hardware out there which always
does some kind of transformation which cannot be turned off. How would
they implement the '0' pipeline?

> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> +or more possible color pipelines. A DRM client will select a color
> +pipeline by setting the COLOR PIPELINE to the respective value.
> +
> +In the case where drivers have custom support for pre-blending color
> +processing those drivers shall reject atomic commits that are trying to
> +set both the custom color properties, as well as the COLOR_PIPELINE
> +property.

By 'not setting the COLOR_PIPELINE property' you mean any value other
than '0' I guess? Should be fine to a proper user space.

> +
> +An example of a COLOR_PIPELINE property on a plane might look like this::
> +
> +    Plane 10
> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> +    ├─ …
> +    └─ "color_pipeline": enum {0, 42, 52} = 0
> +
> +
> +Color Pipeline Discovery
> +========================
> +
> +A DRM client wanting color management on a drm_plane will:
> +
> +1. Read all drm_colorop objects
> +2. Get the COLOR_PIPELINE property of the plane
> +3. iterate all COLOR_PIPELINE enum values
> +4. for each enum value walk the color pipeline (via the NEXT pointers)
> +   and see if the available color operations are suitable for the
> +   desired color management operations
> +
> +An example of chained properties to define an AMD pre-blending color
> +pipeline might look like this::
> +
> +    Plane 10
> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> +    └─ "color_pipeline": enum {0, 42} = 0
> +    Color operation 42 (input CSC)
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 43
> +    Color operation 43
> +    ├─ "type": enum {Scaling} = Scaling
> +    └─ "next": immutable color operation ID = 44
> +    Color operation 44 (DeGamma)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> +    └─ "next": immutable color operation ID = 45
> +    Color operation 45 (gamut remap)
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 46
> +    Color operation 46 (shaper LUT RAM)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 47
> +    Color operation 47 (3D LUT RAM)
> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> +    ├─ "lut_size": immutable range = 17
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 48
> +    Color operation 48 (blend gamma)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 0
> +
> +
> +Color Pipeline Programming
> +==========================
> +
> +Once a DRM client has found a suitable pipeline it will:
> +
> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
> +   drm_colorop object of the desired pipeline
> +2. Set the properties for all drm_colorop objects in the pipeline to the
> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
> +   and false for enabled drm_colorop blocks
> +3. Perform atomic_check/commit as desired
> +
> +To configure the pipeline for an HDR10 PQ plane and blending in linear
> +space, a compositor might perform an atomic commit with the following
> +property values::
> +
> +    Plane 10
> +    └─ "color_pipeline" = 42
> +    Color operation 42 (input CSC)
> +    └─ "bypass" = true
> +    Color operation 44 (DeGamma)
> +    └─ "bypass" = true
> +    Color operation 45 (gamut remap)
> +    └─ "bypasse" = true
> +    Color operation 46 (shaper LUT RAM)
> +    └─ "bypass" = true
> +    Color operation 47 (3D LUT RAM)
> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
> +    Color operation 48 (blend gamma)
> +    └─ "1d_curve_type" = PQ inverse EOTF
> +
> +
> +References
> +==========
> +
> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> \ No newline at end of file
> -- 
> 2.42.0
>

Harry Wentland Sept. 8, 2023, 8:38 p.m. UTC | #2

On 2023-09-08 15:30, Sebastian Wick wrote:
> Hey Harry,
> 
> Thank you and Simon for this great document. Really happy about it, but
> obviously I've got a few notes and questions inline.
> 
> On Fri, Sep 08, 2023 at 11:02:26AM -0400, Harry Wentland wrote:
>> Signed-off-by: Harry Wentland <[e-mail]>
>> Cc: Ville Syrjala <[e-mail]>
>> Cc: Pekka Paalanen <[e-mail]>
>> Cc: Simon Ser <[e-mail]>
>> Cc: Harry Wentland <[e-mail]>
>> Cc: Melissa Wen <[e-mail]>
>> Cc: Jonas Ådahl <[e-mail]>
>> Cc: Sebastian Wick <[e-mail]>
>> Cc: Shashank Sharma <[e-mail]>
>> Cc: Alexander Goins <[e-mail]>
>> Cc: Joshua Ashton <[e-mail]>
>> Cc: Michel Dänzer <[e-mail]>
>> Cc: Aleix Pol <[e-mail]>
>> Cc: Xaver Hugl <[e-mail]>
>> Cc: Victoria Brekenfeld <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Uma Shankar <[e-mail]>
>> Cc: Naseer Ahmed <[e-mail]>
>> Cc: Christopher Braga <[e-mail]>
>> ---
>>   Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
>>   1 file changed, 278 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
>>
>> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
>> new file mode 100644
>> index 000000000000..bfa4a8f12087
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/color_pipeline.rst
>> @@ -0,0 +1,278 @@
>> +========================
>> +Linux Color Pipeline API
>> +========================
>> +
>> +What problem are we solving?
>> +============================
>> +
>> +We would like to support pre-, and post-blending complex color transformations
>> +in order to allow for HW-supported HDR use-cases, as well as to provide support
>> +to color-managed applications, such as video or image editors.
>> +
>> +While it is possible to support an HDR output on HW supporting the Colorspace
>> +and HDR Metadata drm_connector properties that requires the compositor or
>> +application to render and compose the content into one final buffer intended for
>> +display. Doing so is costly.
>> +
>> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
>> +operations to support color transformations. These operations are often
>> +implemented in fixed-function HW and therefore much more power efficient than
>> +performing similar operations via shaders or CPU.
>> +
>> +We would like to make use of this HW functionality to support complex color
>> +transformations with no, or minimal CPU or shader load.
>> +
>> +
>> +How are other OSes solving this problem?
>> +========================================
>> +
>> +The most widely supported use-cases regard HDR content, whether video or
>> +gaming.
>> +
>> +Most OSes will specify the source content format (color gamut, encoding transfer
>> +function, and other metadata, such as max and average light levels) to a driver.
>> +Drivers will then program their fixed-function HW accordingly to map from a
>> +source content buffer's space to a display's space.
>> +
>> +When fixed-function HW is not available the compositor will assemble a shader to
>> +ask the GPU to perform the transformation from the source content format to the
>> +display's format.
>> +
>> +A compositor's mapping function and a driver's mapping function are usually
>> +entirely separate concepts. On OSes where a HW vendor has no insight into
>> +closed-source compositor code such a vendor will tune their color management
>> +code to visually match the compositor's. On other OSes, where both mapping
>> +functions are open to an implementer they will ensure both mappings match.
>> +
>> +
>> +Why is Linux different?
>> +=======================
>> +
>> +Unlike other OSes, where there is one compositor for one or more drivers, on
>> +Linux we have a many-to-many relationship. Many compositors; many drivers.
>> +In addition each compositor vendor or community has their own view of how
>> +color management should be done. This is what makes Linux so beautiful.
>> +
>> +This means that a HW vendor can now no longer tune their driver to one
>> +compositor, as tuning it to one will almost inevitably make it look very
>> +different from another compositor's color mapping.
>> +
>> +We need a better solution.
>> +
>> +
>> +Descriptive API
>> +===============
>> +
>> +An API that describes the source and destination colorspaces is a descriptive
>> +API. It describes the input and output color spaces but does not describe
>> +how precisely they should be mapped. Such a mapping includes many minute
>> +design decision that can greatly affect the look of the final result.
>> +
>> +It is not feasible to describe such mapping with enough detail to ensure the
>> +same result from each implementation. In fact, these mappings are a very active
>> +research area.
>> +
>> +
>> +Prescriptive API
>> +================
>> +
>> +A prescriptive API describes not the source and destination colorspaces. It
>> +instead prescribes a recipe for how to manipulate pixel values to arrive at the
>> +desired outcome.
>> +
>> +This recipe is generally an order straight-forward operations, with clear
>> +mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
>> +operations that can be described in a precise manner.
>> +
>> +
>> +The Color Pipeline API
>> +======================
>> +
>> +HW color management pipelines can significantly differ between HW
>> +vendors in terms of availability, ordering, and capabilities of HW
>> +blocks. This makes a common definition of color management blocks and
>> +their ordering nigh impossible. Instead we are defining an API that
>> +allows user space to discover the HW capabilities.
>> +
>> +
>> +drm_colorop Object & IOCTLs
>> +===========================
>> +
>> +To support the definition of color pipelines we introduce a new DRM core
>> +object, a drm_colorop. Individual drm_colorop objects will be chained
>> +via the NEXT property of a drm_colorop to constitute a color pipeline.
>> +Each drm_colorop object is unique, i.e., even if multiple color
>> +pipelines have the same operation they won't share the same drm_colorop
>> +object to describe that operation.
>> +
>> +Just like other DRM objects the drm_colorop objects are discovered via
>> +IOCTLs:
>> +
>> +DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
>> +number of all drm_colorop objects.
>> +
>> +DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
>> +It includes the ID for the colorop object, as well as the plane_id of
>> +the associated plane. All other values should be registered as
>> +properties.
>> +
>> +Each drm_colorop has three core properties:
> 
> We talked a bit about how forwards compatibility will look like. When a
> driver introduces a new property that user space doesn't understand yet,
> for user space to make use of the pipeline, the property either has to
> have a bypass property, or some other kind of standard/core property
> that promises that this property is purely informational and the
> existance doesn't change the result of the color pipeline. Something
> like "INFORMATIONAL" or "INFO" (better suggestions welcome!).
> 
> If a property doesn't fall into the two categories, the new pipeline
> would be unusable for user space. So, if this new property is added on
> an existing piece of hardware, it should be done on an entirely new
> pipeline to avoid user space regressions.
> 
> I believe these compatibility considerations are very important to
> document.
> 

Should a property have a "bypass" or would it be enough if a colorop
has a "bypass"?

I might need to digest what you said a bit more... Not sure I fully
understand yet, in particular how one would bypass a particular
property.

>> +
>> +TYPE: The type of transformation, such as
>> +* enumerated curve
>> +* custom (uniform) 1D LUT
>> +* 3x3 matrix
>> +* 3x4 matrix
>> +* 3D LUT
>> +* etc.
>> +
>> +Depending on the type of transformation other properties will describe
>> +more details.
>> +
>> +BYPASS: A boolean property that can be used to easily put a block into
>> +bypass mode. While setting other properties might fail atomic check,
>> +setting the BYPASS property to true should never fail. This allows DRM
>> +clients to fallback to other methods of color management if an atomic
>> +check for KMS color operations fails.
> 
> This says BYPASS is a prop but in the example below the type property is
> a mutable enum with Bypass being an enum variant.
> 
> Any reason why you chose to make the type property mutable instead of a
> separate BYPASS property? Personally I like the simplicity of immutable
> properties and would favor a BYPASS property.
> 

My bad, I copy-pasted from Simon's draft but my actual implementation
already looks different. Type is immutable. BYPASS is a separate mutable 
property.

>> +
>> +NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
>> +drm_colorop is the last in the chain.
>> +
>> +An example of a drm_colorop object might look like one of these::
>> +
>> +    Color operation 42
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +
>> +    Color operation 42
>> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
>> +    ├─ "lut_size": immutable range = 33
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +
>> +    Color operation 42
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +
>> +
>> +COLOR_PIPELINE Plane Property
>> +=============================
>> +
>> +Because we don't have existing KMS color properties in the pre-blending
>> +portion of display pipelines (i.e. on drm_planes) we are introducing
>> +color pipelines here first. Eventually we'll want to use the same
>> +concept for the post-blending portion, i.e. drm_crtcs.
>> +
>> +Color Pipelines are created by a driver and advertised via a new
>> +COLOR_PIPELINE enum property on each plane. Values of the property
>> +always include '0', which is the default and means all color processing
>> +is disabled. Additional values will be the object IDs of the first
> 
> Regarding the requirement to never fail atomic checks which set a
> colorop to bypass: I think for user space it would be sufficient if
> setting the entire pipeline to bypass would not fail the atomic check.
> 
> I'm also wondering if there is/will be hardware out there which always
> does some kind of transformation which cannot be turned off. How would
> they implement the '0' pipeline?
> 

Interesting thought. Not sure if that exists. Maybe in that case
COLOR_PIPELINE doesn't ever expose 0 and we use an (immutable) 
drm_colorop to describe the operation.

>> +drm_colorop in a pipeline. A driver can create and advertise none, one,
>> +or more possible color pipelines. A DRM client will select a color
>> +pipeline by setting the COLOR PIPELINE to the respective value.
>> +
>> +In the case where drivers have custom support for pre-blending color
>> +processing those drivers shall reject atomic commits that are trying to
>> +set both the custom color properties, as well as the COLOR_PIPELINE
>> +property.
> 
> By 'not setting the COLOR_PIPELINE property' you mean any value other
> than '0' I guess? Should be fine to a proper user space.
> 

Correct, I should tighten up my language here.

Harry

>> +
>> +An example of a COLOR_PIPELINE property on a plane might look like this::
>> +
>> +    Plane 10
>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>> +    ├─ …
>> +    └─ "color_pipeline": enum {0, 42, 52} = 0
>> +
>> +
>> +Color Pipeline Discovery
>> +========================
>> +
>> +A DRM client wanting color management on a drm_plane will:
>> +
>> +1. Read all drm_colorop objects
>> +2. Get the COLOR_PIPELINE property of the plane
>> +3. iterate all COLOR_PIPELINE enum values
>> +4. for each enum value walk the color pipeline (via the NEXT pointers)
>> +   and see if the available color operations are suitable for the
>> +   desired color management operations
>> +
>> +An example of chained properties to define an AMD pre-blending color
>> +pipeline might look like this::
>> +
>> +    Plane 10
>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>> +    └─ "color_pipeline": enum {0, 42} = 0
>> +    Color operation 42 (input CSC)
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +    Color operation 43
>> +    ├─ "type": enum {Scaling} = Scaling
>> +    └─ "next": immutable color operation ID = 44
>> +    Color operation 44 (DeGamma)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
>> +    └─ "next": immutable color operation ID = 45
>> +    Color operation 45 (gamut remap)
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 46
>> +    Color operation 46 (shaper LUT RAM)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 47
>> +    Color operation 47 (3D LUT RAM)
>> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
>> +    ├─ "lut_size": immutable range = 17
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 48
>> +    Color operation 48 (blend gamma)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 0
>> +
>> +
>> +Color Pipeline Programming
>> +==========================
>> +
>> +Once a DRM client has found a suitable pipeline it will:
>> +
>> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
>> +   drm_colorop object of the desired pipeline
>> +2. Set the properties for all drm_colorop objects in the pipeline to the
>> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
>> +   and false for enabled drm_colorop blocks
>> +3. Perform atomic_check/commit as desired
>> +
>> +To configure the pipeline for an HDR10 PQ plane and blending in linear
>> +space, a compositor might perform an atomic commit with the following
>> +property values::
>> +
>> +    Plane 10
>> +    └─ "color_pipeline" = 42
>> +    Color operation 42 (input CSC)
>> +    └─ "bypass" = true
>> +    Color operation 44 (DeGamma)
>> +    └─ "bypass" = true
>> +    Color operation 45 (gamut remap)
>> +    └─ "bypasse" = true
>> +    Color operation 46 (shaper LUT RAM)
>> +    └─ "bypass" = true
>> +    Color operation 47 (3D LUT RAM)
>> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
>> +    Color operation 48 (blend gamma)
>> +    └─ "1d_curve_type" = PQ inverse EOTF
>> +
>> +
>> +References
>> +==========
>> +
>> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
>> \ No newline at end of file
>> -- 
>> 2.42.0
>>
>

Pekka Paalanen Sept. 13, 2023, 11:29 a.m. UTC | #3

On Fri, 8 Sep 2023 11:02:26 -0400
Harry Wentland <[e-mail]> wrote:

> Signed-off-by: Harry Wentland <[e-mail]>
> Cc: Ville Syrjala <[e-mail]>
> Cc: Pekka Paalanen <[e-mail]>
> Cc: Simon Ser <[e-mail]>
> Cc: Harry Wentland <[e-mail]>
> Cc: Melissa Wen <[e-mail]>
> Cc: Jonas Ådahl <[e-mail]>
> Cc: Sebastian Wick <[e-mail]>
> Cc: Shashank Sharma <[e-mail]>
> Cc: Alexander Goins <[e-mail]>
> Cc: Joshua Ashton <[e-mail]>
> Cc: Michel Dänzer <[e-mail]>
> Cc: Aleix Pol <[e-mail]>
> Cc: Xaver Hugl <[e-mail]>
> Cc: Victoria Brekenfeld <[e-mail]>
> Cc: Daniel Vetter <[e-mail]>
> Cc: Uma Shankar <[e-mail]>
> Cc: Naseer Ahmed <[e-mail]>
> Cc: Christopher Braga <[e-mail]>
> ---
>  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
>  1 file changed, 278 insertions(+)
>  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst

Hi Harry,

it's really nice to see this!

Sebastian started on the backward/forward compatibility, so I'll
comment on everything else here, and leave the compatibility for that
thread.

> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
> new file mode 100644
> index 000000000000..bfa4a8f12087
> --- /dev/null
> +++ b/Documentation/gpu/rfc/color_pipeline.rst
> @@ -0,0 +1,278 @@
> +========================
> +Linux Color Pipeline API
> +========================
> +
> +What problem are we solving?
> +============================
> +
> +We would like to support pre-, and post-blending complex color transformations

+in display controller hardware

> +in order to allow for HW-supported HDR use-cases, as well as to provide support
> +to color-managed applications, such as video or image editors.
> +
> +While it is possible to support an HDR output on HW supporting the Colorspace
> +and HDR Metadata drm_connector properties that requires the compositor or
> +application to render and compose the content into one final buffer intended for
> +display. Doing so is costly.

I think a tiny re-wording would make it easier to read:

+~While i~*I*t is possible to support an HDR output on HW supporting the Colorspace
+and HDR Metadata drm_connector properties*, but* that requires the compositor or
+application to render and compose the content into one final buffer intended for
+display. Doing so is costly.

deletion ~~
addition **


> +
> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
> +operations to support color transformations. These operations are often
> +implemented in fixed-function HW and therefore much more power efficient than
> +performing similar operations via shaders or CPU.
> +
> +We would like to make use of this HW functionality to support complex color
> +transformations with no, or minimal CPU or shader load.
> +
> +
> +How are other OSes solving this problem?
> +========================================
> +
> +The most widely supported use-cases regard HDR content, whether video or
> +gaming.
> +
> +Most OSes will specify the source content format (color gamut, encoding transfer
> +function, and other metadata, such as max and average light levels) to a driver.
> +Drivers will then program their fixed-function HW accordingly to map from a
> +source content buffer's space to a display's space.
> +
> +When fixed-function HW is not available the compositor will assemble a shader to
> +ask the GPU to perform the transformation from the source content format to the
> +display's format.
> +
> +A compositor's mapping function and a driver's mapping function are usually
> +entirely separate concepts. On OSes where a HW vendor has no insight into
> +closed-source compositor code such a vendor will tune their color management
> +code to visually match the compositor's. On other OSes, where both mapping
> +functions are open to an implementer they will ensure both mappings match.
> +

I'd add, assuming it's true:

This results in mapping algorithm lock-in, meaning that no-one alone can
experiment with or introduce new mapping algorithms and achieve
consistent results regardless of which implementation path is taken.

> +
> +Why is Linux different?
> +=======================
> +
> +Unlike other OSes, where there is one compositor for one or more drivers, on
> +Linux we have a many-to-many relationship. Many compositors; many drivers.
> +In addition each compositor vendor or community has their own view of how
> +color management should be done. This is what makes Linux so beautiful.
> +
> +This means that a HW vendor can now no longer tune their driver to one
> +compositor, as tuning it to one will almost inevitably make it look very
> +different from another compositor's color mapping.

This is easy to misunderstand as "all Linux desktops will get your
colors different so you cannot have a consistent look in an app". That
might trigger a few flamewars, even though it is a true goal in essence.

Maybe "almost inevitably" should be worded much more uncertain. Maybe
just "could make it look fairly different".

Much of color management is about user preferences. Different desktops
may have different sets of tunables, like different monitors and TV
have different tunables and color modes. It is still also an active
research area, and new image formats and new ways of driving displays
will surely emerge.

Different use cases have different fundamental goals with color
management, which warrants the "different colors". Linux should be
applicable to a wide range of use cases.

If these thoughts give you ideas how to rewrite this section, go for
it, but it could also be enough to change the couple words I suggested.


> +
> +We need a better solution.
> +
> +
> +Descriptive API
> +===============
> +
> +An API that describes the source and destination colorspaces is a descriptive
> +API. It describes the input and output color spaces but does not describe
> +how precisely they should be mapped. Such a mapping includes many minute
> +design decision that can greatly affect the look of the final result.
> +
> +It is not feasible to describe such mapping with enough detail to ensure the
> +same result from each implementation. In fact, these mappings are a very active
> +research area.
> +
> +
> +Prescriptive API
> +================
> +
> +A prescriptive API describes not the source and destination colorspaces. It
> +instead prescribes a recipe for how to manipulate pixel values to arrive at the
> +desired outcome.
> +
> +This recipe is generally an order straight-forward operations, with clear

Is this line missing some words?

> +mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
> +operations that can be described in a precise manner.
> +
> +
> +The Color Pipeline API
> +======================
> +
> +HW color management pipelines can significantly differ between HW
> +vendors in terms of availability, ordering, and capabilities of HW
> +blocks. This makes a common definition of color management blocks and
> +their ordering nigh impossible. Instead we are defining an API that
> +allows user space to discover the HW capabilities.

+in a generic manner, agnostic of specific drivers and hardware.

> +
> +
> +drm_colorop Object & IOCTLs
> +===========================
> +
> +To support the definition of color pipelines we introduce a new DRM core

"we define the DRM core object type drm_colorop."

I think that reads better after being merged upstream.

> +object, a drm_colorop. Individual drm_colorop objects will be chained
> +via the NEXT property of a drm_colorop to constitute a color pipeline.
> +Each drm_colorop object is unique, i.e., even if multiple color
> +pipelines have the same operation they won't share the same drm_colorop
> +object to describe that operation.

Maybe add some words here how drivers are not expected to map a
drm_colorop object statically to a specific HW block?

If someone was to assume the contrary, they would be wondering how they
can ever expose multiple pipelines.

The mapping between drm_colorop objects and HW blocks is completely a
driver internal detail, and can be as dynamic or static as the driver
needs it to be.

Speaking of drivers, I remember recently writing an email about what
would be a good way to expose HW functionality as drm_colorops.
Something about that would be good to document for driver writers,
maybe in a section of its own:

- Try to expose pipelines that use already defined colorops, even if
  your hardware pipeline is split differently. This allows existing
  userspace to immediately take advantage of the hardware.

- Additionally, try to expose your actual hardware blocks as colorops.
  Define new colorop types where you believe it can offer significant
  benefits if userspace learns to program them.

- Avoid defining new colorops for compound operations with very narrow
  scope. If you have a hardware block for a special operation that
  cannot be split further, you can expose that as a new colorop type.
  However, try to not define colorops for "use cases", especially if
  they require you to combine multiple hardware blocks.

- Design new colorops as prescriptive, not descriptive; by the
  mathematical formula, not by the assumed input and output.

A defined colorop type must be deterministic. Its operation can depend
only on its properties (and input?) and nothing else, allowed error
tolerance notwithstanding.

(By input I'm thinking of a block that maintains some state from
previous frame color statistics, and adjusts its behaviour.)

> +
> +Just like other DRM objects the drm_colorop objects are discovered via
> +IOCTLs:
> +
> +DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
> +number of all drm_colorop objects.

What is this useful for? Isn't the COLOR_PIPELINE plane property enough
to discover everything?

> +DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
> +It includes the ID for the colorop object, as well as the plane_id of
> +the associated plane. All other values should be registered as
> +properties.

Why should plane_id not be a property? Could we remove GETCOLOROP
completely?

> +
> +Each drm_colorop has three core properties:
> +
> +TYPE: The type of transformation, such as
> +* enumerated curve
> +* custom (uniform) 1D LUT
> +* 3x3 matrix
> +* 3x4 matrix
> +* 3D LUT
> +* etc.
> +
> +Depending on the type of transformation other properties will describe
> +more details.
> +
> +BYPASS: A boolean property that can be used to easily put a block into
> +bypass mode. While setting other properties might fail atomic check,
> +setting the BYPASS property to true should never fail. This allows DRM
> +clients to fallback to other methods of color management if an atomic
> +check for KMS color operations fails.
> +
> +NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
> +drm_colorop is the last in the chain.
> +
> +An example of a drm_colorop object might look like one of these::
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> +    ├─ "lut_size": immutable range = 33
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +
> +COLOR_PIPELINE Plane Property
> +=============================
> +
> +Because we don't have existing KMS color properties in the pre-blending
> +portion of display pipelines (i.e. on drm_planes) we are introducing
> +color pipelines here first. Eventually we'll want to use the same
> +concept for the post-blending portion, i.e. drm_crtcs.

This paragraph might fit better in a cover letter.

> +
> +Color Pipelines are created by a driver and advertised via a new
> +COLOR_PIPELINE enum property on each plane. Values of the property
> +always include '0', which is the default and means all color processing
> +is disabled. Additional values will be the object IDs of the first
> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> +or more possible color pipelines. A DRM client will select a color
> +pipeline by setting the COLOR PIPELINE to the respective value.
> +
> +In the case where drivers have custom support for pre-blending color
> +processing those drivers shall reject atomic commits that are trying to
> +set both the custom color properties, as well as the COLOR_PIPELINE

s/set/use/ because one of them could be carried-over state from
previous commits while not literally set in this one.

> +property.
> +
> +An example of a COLOR_PIPELINE property on a plane might look like this::
> +
> +    Plane 10
> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> +    ├─ …
> +    └─ "color_pipeline": enum {0, 42, 52} = 0

Enum values are string names. I presume the intention here is that the
strings will never need to be parsed, and the uint64_t is always equal
to the string representation, right?

That needs a statement here. It differs from all previous uses of
enums, and e.g. requires a little bit of new API in libweston's
DRM-backend to handle since it has its own enums referring to the
string names that get mapped to the uint64_t per owning KMS object.

> +
> +
> +Color Pipeline Discovery
> +========================
> +
> +A DRM client wanting color management on a drm_plane will:
> +
> +1. Read all drm_colorop objects

What does this do?

> +2. Get the COLOR_PIPELINE property of the plane
> +3. iterate all COLOR_PIPELINE enum values
> +4. for each enum value walk the color pipeline (via the NEXT pointers)
> +   and see if the available color operations are suitable for the
> +   desired color management operations
> +
> +An example of chained properties to define an AMD pre-blending color
> +pipeline might look like this::
> +
> +    Plane 10
> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> +    └─ "color_pipeline": enum {0, 42} = 0
> +    Color operation 42 (input CSC)

I presume the string "(input CSC)" does not come from KMS, and is
actually just a comment added here by hand?


Thanks,
pq

> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 43
> +    Color operation 43
> +    ├─ "type": enum {Scaling} = Scaling
> +    └─ "next": immutable color operation ID = 44
> +    Color operation 44 (DeGamma)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> +    └─ "next": immutable color operation ID = 45
> +    Color operation 45 (gamut remap)
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 46
> +    Color operation 46 (shaper LUT RAM)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 47
> +    Color operation 47 (3D LUT RAM)
> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> +    ├─ "lut_size": immutable range = 17
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 48
> +    Color operation 48 (blend gamma)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 0
> +
> +
> +Color Pipeline Programming
> +==========================
> +
> +Once a DRM client has found a suitable pipeline it will:
> +
> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
> +   drm_colorop object of the desired pipeline
> +2. Set the properties for all drm_colorop objects in the pipeline to the
> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
> +   and false for enabled drm_colorop blocks
> +3. Perform atomic_check/commit as desired
> +
> +To configure the pipeline for an HDR10 PQ plane and blending in linear
> +space, a compositor might perform an atomic commit with the following
> +property values::
> +
> +    Plane 10
> +    └─ "color_pipeline" = 42
> +    Color operation 42 (input CSC)
> +    └─ "bypass" = true
> +    Color operation 44 (DeGamma)
> +    └─ "bypass" = true
> +    Color operation 45 (gamut remap)
> +    └─ "bypasse" = true
> +    Color operation 46 (shaper LUT RAM)
> +    └─ "bypass" = true
> +    Color operation 47 (3D LUT RAM)
> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
> +    Color operation 48 (blend gamma)
> +    └─ "1d_curve_type" = PQ inverse EOTF
> +
> +
> +References
> +==========
> +
> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> \ No newline at end of file

Pekka Paalanen Sept. 13, 2023, 11:53 a.m. UTC | #4

On Fri, 8 Sep 2023 16:38:44 -0400
Harry Wentland <[e-mail]> wrote:

> On 2023-09-08 15:30, Sebastian Wick wrote:
> > Hey Harry,
> > 
> > Thank you and Simon for this great document. Really happy about it, but
> > obviously I've got a few notes and questions inline.
> > 
> > On Fri, Sep 08, 2023 at 11:02:26AM -0400, Harry Wentland wrote:  
> >> Signed-off-by: Harry Wentland <[e-mail]>
> >> Cc: Ville Syrjala <[e-mail]>
> >> Cc: Pekka Paalanen <[e-mail]>
> >> Cc: Simon Ser <[e-mail]>
> >> Cc: Harry Wentland <[e-mail]>
> >> Cc: Melissa Wen <[e-mail]>
> >> Cc: Jonas Ådahl <[e-mail]>
> >> Cc: Sebastian Wick <[e-mail]>
> >> Cc: Shashank Sharma <[e-mail]>
> >> Cc: Alexander Goins <[e-mail]>
> >> Cc: Joshua Ashton <[e-mail]>
> >> Cc: Michel Dänzer <[e-mail]>
> >> Cc: Aleix Pol <[e-mail]>
> >> Cc: Xaver Hugl <[e-mail]>
> >> Cc: Victoria Brekenfeld <[e-mail]>
> >> Cc: Daniel Vetter <[e-mail]>
> >> Cc: Uma Shankar <[e-mail]>
> >> Cc: Naseer Ahmed <[e-mail]>
> >> Cc: Christopher Braga <[e-mail]>
> >> ---
> >>   Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
> >>   1 file changed, 278 insertions(+)
> >>   create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> >>
> >> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
> >> new file mode 100644
> >> index 000000000000..bfa4a8f12087
> >> --- /dev/null
> >> +++ b/Documentation/gpu/rfc/color_pipeline.rst
> >> @@ -0,0 +1,278 @@
> >> +========================
> >> +Linux Color Pipeline API
> >> +========================
> >> +
> >> +What problem are we solving?
> >> +============================
> >> +
> >> +We would like to support pre-, and post-blending complex color transformations
> >> +in order to allow for HW-supported HDR use-cases, as well as to provide support
> >> +to color-managed applications, such as video or image editors.
> >> +
> >> +While it is possible to support an HDR output on HW supporting the Colorspace
> >> +and HDR Metadata drm_connector properties that requires the compositor or
> >> +application to render and compose the content into one final buffer intended for
> >> +display. Doing so is costly.
> >> +
> >> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
> >> +operations to support color transformations. These operations are often
> >> +implemented in fixed-function HW and therefore much more power efficient than
> >> +performing similar operations via shaders or CPU.
> >> +
> >> +We would like to make use of this HW functionality to support complex color
> >> +transformations with no, or minimal CPU or shader load.
> >> +
> >> +
> >> +How are other OSes solving this problem?
> >> +========================================
> >> +
> >> +The most widely supported use-cases regard HDR content, whether video or
> >> +gaming.
> >> +
> >> +Most OSes will specify the source content format (color gamut, encoding transfer
> >> +function, and other metadata, such as max and average light levels) to a driver.
> >> +Drivers will then program their fixed-function HW accordingly to map from a
> >> +source content buffer's space to a display's space.
> >> +
> >> +When fixed-function HW is not available the compositor will assemble a shader to
> >> +ask the GPU to perform the transformation from the source content format to the
> >> +display's format.
> >> +
> >> +A compositor's mapping function and a driver's mapping function are usually
> >> +entirely separate concepts. On OSes where a HW vendor has no insight into
> >> +closed-source compositor code such a vendor will tune their color management
> >> +code to visually match the compositor's. On other OSes, where both mapping
> >> +functions are open to an implementer they will ensure both mappings match.
> >> +
> >> +
> >> +Why is Linux different?
> >> +=======================
> >> +
> >> +Unlike other OSes, where there is one compositor for one or more drivers, on
> >> +Linux we have a many-to-many relationship. Many compositors; many drivers.
> >> +In addition each compositor vendor or community has their own view of how
> >> +color management should be done. This is what makes Linux so beautiful.
> >> +
> >> +This means that a HW vendor can now no longer tune their driver to one
> >> +compositor, as tuning it to one will almost inevitably make it look very
> >> +different from another compositor's color mapping.
> >> +
> >> +We need a better solution.
> >> +
> >> +
> >> +Descriptive API
> >> +===============
> >> +
> >> +An API that describes the source and destination colorspaces is a descriptive
> >> +API. It describes the input and output color spaces but does not describe
> >> +how precisely they should be mapped. Such a mapping includes many minute
> >> +design decision that can greatly affect the look of the final result.
> >> +
> >> +It is not feasible to describe such mapping with enough detail to ensure the
> >> +same result from each implementation. In fact, these mappings are a very active
> >> +research area.
> >> +
> >> +
> >> +Prescriptive API
> >> +================
> >> +
> >> +A prescriptive API describes not the source and destination colorspaces. It
> >> +instead prescribes a recipe for how to manipulate pixel values to arrive at the
> >> +desired outcome.
> >> +
> >> +This recipe is generally an order straight-forward operations, with clear
> >> +mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
> >> +operations that can be described in a precise manner.
> >> +
> >> +
> >> +The Color Pipeline API
> >> +======================
> >> +
> >> +HW color management pipelines can significantly differ between HW
> >> +vendors in terms of availability, ordering, and capabilities of HW
> >> +blocks. This makes a common definition of color management blocks and
> >> +their ordering nigh impossible. Instead we are defining an API that
> >> +allows user space to discover the HW capabilities.
> >> +
> >> +
> >> +drm_colorop Object & IOCTLs
> >> +===========================
> >> +
> >> +To support the definition of color pipelines we introduce a new DRM core
> >> +object, a drm_colorop. Individual drm_colorop objects will be chained
> >> +via the NEXT property of a drm_colorop to constitute a color pipeline.
> >> +Each drm_colorop object is unique, i.e., even if multiple color
> >> +pipelines have the same operation they won't share the same drm_colorop
> >> +object to describe that operation.
> >> +
> >> +Just like other DRM objects the drm_colorop objects are discovered via
> >> +IOCTLs:
> >> +
> >> +DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
> >> +number of all drm_colorop objects.
> >> +
> >> +DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
> >> +It includes the ID for the colorop object, as well as the plane_id of
> >> +the associated plane. All other values should be registered as
> >> +properties.
> >> +
> >> +Each drm_colorop has three core properties:  
> > 
> > We talked a bit about how forwards compatibility will look like. When a
> > driver introduces a new property that user space doesn't understand yet,
> > for user space to make use of the pipeline, the property either has to

A new property or colorop?

> > have a bypass property, or some other kind of standard/core property
> > that promises that this property is purely informational and the
> > existance doesn't change the result of the color pipeline. Something
> > like "INFORMATIONAL" or "INFO" (better suggestions welcome!).
> > 
> > If a property doesn't fall into the two categories, the new pipeline
> > would be unusable for user space. So, if this new property is added on
> > an existing piece of hardware, it should be done on an entirely new
> > pipeline to avoid user space regressions.
> > 
> > I believe these compatibility considerations are very important to
> > document.
> >   
> 
> Should a property have a "bypass" or would it be enough if a colorop
> has a "bypass"?
> 
> I might need to digest what you said a bit more... Not sure I fully
> understand yet, in particular how one would bypass a particular
> property.

I suspect new colorops in an old pipeline need different handling than
a new property in an old colorop.

What if bypass is not what happened before a new colorop was added in a
pipeline?

Let's have an example: the CRTC automatic RGB->YUV conversion to the
wire format applied as needed. Previously it was all implicit,
automatic, and not exposed. Now one wants to expose it as a new colorop
in the pipeline. Let's say you also expose the old pipeline without
this.

How will userspace choose between a pipeline without the colorop, and a
pipeline with the colorop set to bypass?

The definition of bypass is that it does not alter the values passing
through, but that's not what happens without the colorop.

Maybe this colorop must not have the standard bypass property.

Maybe "bypass" should be "default" instead. But then the color pipeline
results become undefined from userspace perspective, so that doesn't
work.

Maybe we need a rule that everything that modifies pixel values in a
pipeline MUST be exposed from the very moment the COLOR_PIPELINE
property is added to a KMS object. HW blocks that a driver hardcodes to
pass-through and so do not modify any values do not need to be exposed;
they can be trivially exposed later with "bypass" property.

That last "maybe" is actually a hard requirement for the whole color
pipeline design to work, because otherwise the results are unknown to
userspace.

What should a driver writer do if hardware implicitly clamps values at
some point? That potentially modifies values, so it must be taken into
account, especially when it is possible to exceed the nominal unit
range [0.0, 1.0].

What if a driver writer misses something or makes a mistake? Can that
be fixed a few releases later without regressing userspace? I feel like
that last "maybe" might be difficult to ensure.

> >> +
> >> +TYPE: The type of transformation, such as
> >> +* enumerated curve
> >> +* custom (uniform) 1D LUT
> >> +* 3x3 matrix
> >> +* 3x4 matrix
> >> +* 3D LUT
> >> +* etc.
> >> +
> >> +Depending on the type of transformation other properties will describe
> >> +more details.
> >> +
> >> +BYPASS: A boolean property that can be used to easily put a block into
> >> +bypass mode. While setting other properties might fail atomic check,
> >> +setting the BYPASS property to true should never fail. This allows DRM
> >> +clients to fallback to other methods of color management if an atomic
> >> +check for KMS color operations fails.  
> > 
> > This says BYPASS is a prop but in the example below the type property is
> > a mutable enum with Bypass being an enum variant.
> > 
> > Any reason why you chose to make the type property mutable instead of a
> > separate BYPASS property? Personally I like the simplicity of immutable
> > properties and would favor a BYPASS property.
> >   
> 
> My bad, I copy-pasted from Simon's draft but my actual implementation
> already looks different. Type is immutable. BYPASS is a separate mutable 
> property.
> 
> >> +
> >> +NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
> >> +drm_colorop is the last in the chain.
> >> +
> >> +An example of a drm_colorop object might look like one of these::
> >> +
> >> +    Color operation 42
> >> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> >> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
> >> +    ├─ "lut_size": immutable range = 4096
> >> +    ├─ "lut_data": blob
> >> +    └─ "next": immutable color operation ID = 43
> >> +
> >> +    Color operation 42
> >> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> >> +    ├─ "lut_size": immutable range = 33
> >> +    ├─ "lut_data": blob
> >> +    └─ "next": immutable color operation ID = 43
> >> +
> >> +    Color operation 42
> >> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> >> +    ├─ "matrix_data": blob
> >> +    └─ "next": immutable color operation ID = 43
> >> +
> >> +
> >> +COLOR_PIPELINE Plane Property
> >> +=============================
> >> +
> >> +Because we don't have existing KMS color properties in the pre-blending
> >> +portion of display pipelines (i.e. on drm_planes) we are introducing
> >> +color pipelines here first. Eventually we'll want to use the same
> >> +concept for the post-blending portion, i.e. drm_crtcs.
> >> +
> >> +Color Pipelines are created by a driver and advertised via a new
> >> +COLOR_PIPELINE enum property on each plane. Values of the property
> >> +always include '0', which is the default and means all color processing
> >> +is disabled. Additional values will be the object IDs of the first  
> > 
> > Regarding the requirement to never fail atomic checks which set a
> > colorop to bypass: I think for user space it would be sufficient if
> > setting the entire pipeline to bypass would not fail the atomic check.

I agree.

> > 
> > I'm also wondering if there is/will be hardware out there which always
> > does some kind of transformation which cannot be turned off. How would
> > they implement the '0' pipeline?
> >   
> 
> Interesting thought. Not sure if that exists. Maybe in that case
> COLOR_PIPELINE doesn't ever expose 0 and we use an (immutable) 
> drm_colorop to describe the operation.

Right.


Thanks,
pq

> 
> >> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> >> +or more possible color pipelines. A DRM client will select a color
> >> +pipeline by setting the COLOR PIPELINE to the respective value.
> >> +
> >> +In the case where drivers have custom support for pre-blending color
> >> +processing those drivers shall reject atomic commits that are trying to
> >> +set both the custom color properties, as well as the COLOR_PIPELINE
> >> +property.  
> > 
> > By 'not setting the COLOR_PIPELINE property' you mean any value other
> > than '0' I guess? Should be fine to a proper user space.
> >   
> 
> Correct, I should tighten up my language here.
> 
> Harry
> 
> >> +
> >> +An example of a COLOR_PIPELINE property on a plane might look like this::
> >> +
> >> +    Plane 10
> >> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> >> +    ├─ …
> >> +    └─ "color_pipeline": enum {0, 42, 52} = 0
> >> +
> >> +
> >> +Color Pipeline Discovery
> >> +========================
> >> +
> >> +A DRM client wanting color management on a drm_plane will:
> >> +
> >> +1. Read all drm_colorop objects
> >> +2. Get the COLOR_PIPELINE property of the plane
> >> +3. iterate all COLOR_PIPELINE enum values
> >> +4. for each enum value walk the color pipeline (via the NEXT pointers)
> >> +   and see if the available color operations are suitable for the
> >> +   desired color management operations
> >> +
> >> +An example of chained properties to define an AMD pre-blending color
> >> +pipeline might look like this::
> >> +
> >> +    Plane 10
> >> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> >> +    └─ "color_pipeline": enum {0, 42} = 0
> >> +    Color operation 42 (input CSC)
> >> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> >> +    ├─ "matrix_data": blob
> >> +    └─ "next": immutable color operation ID = 43
> >> +    Color operation 43
> >> +    ├─ "type": enum {Scaling} = Scaling
> >> +    └─ "next": immutable color operation ID = 44
> >> +    Color operation 44 (DeGamma)
> >> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> >> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> >> +    └─ "next": immutable color operation ID = 45
> >> +    Color operation 45 (gamut remap)
> >> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> >> +    ├─ "matrix_data": blob
> >> +    └─ "next": immutable color operation ID = 46
> >> +    Color operation 46 (shaper LUT RAM)
> >> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> >> +    ├─ "1d_curve_type": enum {LUT} = LUT
> >> +    ├─ "lut_size": immutable range = 4096
> >> +    ├─ "lut_data": blob
> >> +    └─ "next": immutable color operation ID = 47
> >> +    Color operation 47 (3D LUT RAM)
> >> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> >> +    ├─ "lut_size": immutable range = 17
> >> +    ├─ "lut_data": blob
> >> +    └─ "next": immutable color operation ID = 48
> >> +    Color operation 48 (blend gamma)
> >> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> >> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> >> +    ├─ "lut_size": immutable range = 4096
> >> +    ├─ "lut_data": blob
> >> +    └─ "next": immutable color operation ID = 0
> >> +
> >> +
> >> +Color Pipeline Programming
> >> +==========================
> >> +
> >> +Once a DRM client has found a suitable pipeline it will:
> >> +
> >> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
> >> +   drm_colorop object of the desired pipeline
> >> +2. Set the properties for all drm_colorop objects in the pipeline to the
> >> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
> >> +   and false for enabled drm_colorop blocks
> >> +3. Perform atomic_check/commit as desired
> >> +
> >> +To configure the pipeline for an HDR10 PQ plane and blending in linear
> >> +space, a compositor might perform an atomic commit with the following
> >> +property values::
> >> +
> >> +    Plane 10
> >> +    └─ "color_pipeline" = 42
> >> +    Color operation 42 (input CSC)
> >> +    └─ "bypass" = true
> >> +    Color operation 44 (DeGamma)
> >> +    └─ "bypass" = true
> >> +    Color operation 45 (gamut remap)
> >> +    └─ "bypasse" = true
> >> +    Color operation 46 (shaper LUT RAM)
> >> +    └─ "bypass" = true
> >> +    Color operation 47 (3D LUT RAM)
> >> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
> >> +    Color operation 48 (blend gamma)
> >> +    └─ "1d_curve_type" = PQ inverse EOTF
> >> +
> >> +
> >> +References
> >> +==========
> >> +
> >> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> >> \ No newline at end of file
> >> -- 
> >> 2.42.0
> >>  
> >

Melissa Wen Oct. 10, 2023, 4:13 p.m. UTC | #5

O 09/08, Harry Wentland wrote:
> Signed-off-by: Harry Wentland <[e-mail]>
> Cc: Ville Syrjala <[e-mail]>
> Cc: Pekka Paalanen <[e-mail]>
> Cc: Simon Ser <[e-mail]>
> Cc: Harry Wentland <[e-mail]>
> Cc: Melissa Wen <[e-mail]>
> Cc: Jonas Ådahl <[e-mail]>
> Cc: Sebastian Wick <[e-mail]>
> Cc: Shashank Sharma <[e-mail]>
> Cc: Alexander Goins <[e-mail]>
> Cc: Joshua Ashton <[e-mail]>
> Cc: Michel Dänzer <[e-mail]>
> Cc: Aleix Pol <[e-mail]>
> Cc: Xaver Hugl <[e-mail]>
> Cc: Victoria Brekenfeld <[e-mail]>
> Cc: Daniel Vetter <[e-mail]>
> Cc: Uma Shankar <[e-mail]>
> Cc: Naseer Ahmed <[e-mail]>
> Cc: Christopher Braga <[e-mail]>
> ---
>  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
>  1 file changed, 278 insertions(+)
>  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> 
> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
> new file mode 100644
> index 000000000000..bfa4a8f12087
> --- /dev/null
> +++ b/Documentation/gpu/rfc/color_pipeline.rst
> @@ -0,0 +1,278 @@
> +========================
> +Linux Color Pipeline API
> +========================
> +
> +What problem are we solving?
> +============================
> +
> +We would like to support pre-, and post-blending complex color transformations
> +in order to allow for HW-supported HDR use-cases, as well as to provide support
> +to color-managed applications, such as video or image editors.
> +
> +While it is possible to support an HDR output on HW supporting the Colorspace
> +and HDR Metadata drm_connector properties that requires the compositor or
> +application to render and compose the content into one final buffer intended for
> +display. Doing so is costly.
> +
> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
> +operations to support color transformations. These operations are often
> +implemented in fixed-function HW and therefore much more power efficient than
> +performing similar operations via shaders or CPU.
> +
> +We would like to make use of this HW functionality to support complex color
> +transformations with no, or minimal CPU or shader load.
> +
> +
> +How are other OSes solving this problem?
> +========================================
> +
> +The most widely supported use-cases regard HDR content, whether video or
> +gaming.
> +
> +Most OSes will specify the source content format (color gamut, encoding transfer
> +function, and other metadata, such as max and average light levels) to a driver.
> +Drivers will then program their fixed-function HW accordingly to map from a
> +source content buffer's space to a display's space.
> +
> +When fixed-function HW is not available the compositor will assemble a shader to
> +ask the GPU to perform the transformation from the source content format to the
> +display's format.
> +
> +A compositor's mapping function and a driver's mapping function are usually
> +entirely separate concepts. On OSes where a HW vendor has no insight into
> +closed-source compositor code such a vendor will tune their color management
> +code to visually match the compositor's. On other OSes, where both mapping
> +functions are open to an implementer they will ensure both mappings match.
> +
> +
> +Why is Linux different?
> +=======================
> +
> +Unlike other OSes, where there is one compositor for one or more drivers, on
> +Linux we have a many-to-many relationship. Many compositors; many drivers.
> +In addition each compositor vendor or community has their own view of how
> +color management should be done. This is what makes Linux so beautiful.
> +
> +This means that a HW vendor can now no longer tune their driver to one
> +compositor, as tuning it to one will almost inevitably make it look very
> +different from another compositor's color mapping.
> +
> +We need a better solution.
> +
> +
> +Descriptive API
> +===============
> +
> +An API that describes the source and destination colorspaces is a descriptive
> +API. It describes the input and output color spaces but does not describe
> +how precisely they should be mapped. Such a mapping includes many minute
> +design decision that can greatly affect the look of the final result.
> +
> +It is not feasible to describe such mapping with enough detail to ensure the
> +same result from each implementation. In fact, these mappings are a very active
> +research area.
> +
> +
> +Prescriptive API
> +================
> +
> +A prescriptive API describes not the source and destination colorspaces. It
> +instead prescribes a recipe for how to manipulate pixel values to arrive at the
> +desired outcome.
> +
> +This recipe is generally an order straight-forward operations, with clear
> +mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
> +operations that can be described in a precise manner.
> +
> +
> +The Color Pipeline API
> +======================
> +
> +HW color management pipelines can significantly differ between HW
> +vendors in terms of availability, ordering, and capabilities of HW
> +blocks. This makes a common definition of color management blocks and
> +their ordering nigh impossible. Instead we are defining an API that
> +allows user space to discover the HW capabilities.
> +
> +
> +drm_colorop Object & IOCTLs
> +===========================
> +
> +To support the definition of color pipelines we introduce a new DRM core
> +object, a drm_colorop. Individual drm_colorop objects will be chained
> +via the NEXT property of a drm_colorop to constitute a color pipeline.
> +Each drm_colorop object is unique, i.e., even if multiple color
> +pipelines have the same operation they won't share the same drm_colorop
> +object to describe that operation.
> +
> +Just like other DRM objects the drm_colorop objects are discovered via
> +IOCTLs:
> +
> +DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
> +number of all drm_colorop objects.
> +
> +DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
> +It includes the ID for the colorop object, as well as the plane_id of
> +the associated plane. All other values should be registered as
> +properties.
> +
> +Each drm_colorop has three core properties:
> +
> +TYPE: The type of transformation, such as
> +* enumerated curve
> +* custom (uniform) 1D LUT
> +* 3x3 matrix
> +* 3x4 matrix
> +* 3D LUT
> +* etc.
> +
> +Depending on the type of transformation other properties will describe
> +more details.
> +
> +BYPASS: A boolean property that can be used to easily put a block into
> +bypass mode. While setting other properties might fail atomic check,
> +setting the BYPASS property to true should never fail. This allows DRM
> +clients to fallback to other methods of color management if an atomic
> +check for KMS color operations fails.
> +
> +NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
> +drm_colorop is the last in the chain.
> +
> +An example of a drm_colorop object might look like one of these::
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> +    ├─ "lut_size": immutable range = 33
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +    Color operation 42
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 43
> +
> +
> +COLOR_PIPELINE Plane Property
> +=============================
> +
> +Because we don't have existing KMS color properties in the pre-blending
> +portion of display pipelines (i.e. on drm_planes) we are introducing
> +color pipelines here first. Eventually we'll want to use the same
> +concept for the post-blending portion, i.e. drm_crtcs.
> +
> +Color Pipelines are created by a driver and advertised via a new
> +COLOR_PIPELINE enum property on each plane. Values of the property
> +always include '0', which is the default and means all color processing
> +is disabled. Additional values will be the object IDs of the first
> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> +or more possible color pipelines. A DRM client will select a color
> +pipeline by setting the COLOR PIPELINE to the respective value.
> +
> +In the case where drivers have custom support for pre-blending color
> +processing those drivers shall reject atomic commits that are trying to
> +set both the custom color properties, as well as the COLOR_PIPELINE
> +property.
> +
> +An example of a COLOR_PIPELINE property on a plane might look like this::
> +
> +    Plane 10
> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> +    ├─ …
> +    └─ "color_pipeline": enum {0, 42, 52} = 0
> +
> +
> +Color Pipeline Discovery
> +========================
> +
> +A DRM client wanting color management on a drm_plane will:
> +
> +1. Read all drm_colorop objects
> +2. Get the COLOR_PIPELINE property of the plane
> +3. iterate all COLOR_PIPELINE enum values
> +4. for each enum value walk the color pipeline (via the NEXT pointers)
> +   and see if the available color operations are suitable for the
> +   desired color management operations
> +
> +An example of chained properties to define an AMD pre-blending color
> +pipeline might look like this::

Hi Harry,

Thanks for sharing this proposal. Overall I think it's very aligned with
Simon's description of the generic KMS color API. I think it's a good
start point and we can refine over iterations. My general questions have
already been pointed out by Sebastian and Pekka (mainly regarding the
BYPASS property).

I still have some doubts on how to fit these set of colorops with some
AMD corners cases as below:

> +
> +    Plane 10
> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> +    └─ "color_pipeline": enum {0, 42} = 0
> +    Color operation 42 (input CSC)
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 43

IIUC, for input CSC, there are currently two possiblities, or we use
`drm_plane_color_encoding` and `drm_plane_color range` to get
pre-defined coefficients or we set a custom matrix here, right? If so, I
think we need some kind of pre-defined matrix option?

Also, with this new plane API in place, I understand that we will
already need think on how to deal with the mixing between old drm color
properties (color encoding and color range) and these new way of setting
plane color properties. IIUC, Pekka asked a related question about it
when talking about CRTC automatic RGB->YUV (?) 

> +    Color operation 43
> +    ├─ "type": enum {Scaling} = Scaling
> +    └─ "next": immutable color operation ID = 44
> +    Color operation 44 (DeGamma)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> +    └─ "next": immutable color operation ID = 45
> +    Color operation 45 (gamut remap)
> +    ├─ "type": enum {Bypass, Matrix} = Matrix
> +    ├─ "matrix_data": blob
> +    └─ "next": immutable color operation ID = 46
> +    Color operation 46 (shaper LUT RAM)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 47

For shaper and blend LUT RAM, that the driver supports pre-defined
curves and custom LUT at the same time but the resulted LUT is a
combination of those, how to make this behavior clear? Could this
behavior be described by two colorop in a row: for example, one for
shaper TF and,just after, another for shaper LUT or would it not be the
right representation?

> +    Color operation 47 (3D LUT RAM)
> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> +    ├─ "lut_size": immutable range = 17
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 48
> +    Color operation 48 (blend gamma)
> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> +    ├─ "lut_size": immutable range = 4096
> +    ├─ "lut_data": blob
> +    └─ "next": immutable color operation ID = 0
> +
> +
> +Color Pipeline Programming
> +==========================
> +
> +Once a DRM client has found a suitable pipeline it will:
> +
> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
> +   drm_colorop object of the desired pipeline
> +2. Set the properties for all drm_colorop objects in the pipeline to the
> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
> +   and false for enabled drm_colorop blocks
> +3. Perform atomic_check/commit as desired
> +
> +To configure the pipeline for an HDR10 PQ plane and blending in linear
> +space, a compositor might perform an atomic commit with the following
> +property values::
> +
> +    Plane 10
> +    └─ "color_pipeline" = 42
> +    Color operation 42 (input CSC)
> +    └─ "bypass" = true
> +    Color operation 44 (DeGamma)
> +    └─ "bypass" = true
> +    Color operation 45 (gamut remap)
> +    └─ "bypasse" = true
> +    Color operation 46 (shaper LUT RAM)
> +    └─ "bypass" = true
> +    Color operation 47 (3D LUT RAM)
> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
> +    Color operation 48 (blend gamma)
> +    └─ "1d_curve_type" = PQ inverse EOTF

Isn't it a PQ EOTF for blend gamma?

Best Regards,

Melissa

> +
> +
> +References
> +==========
> +
> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> \ No newline at end of file
> -- 
> 2.42.0
>

Pekka Paalanen Oct. 11, 2023, 10:20 a.m. UTC | #6

On Tue, 10 Oct 2023 15:13:46 -0100
Melissa Wen <[e-mail]> wrote:

> O 09/08, Harry Wentland wrote:
> > Signed-off-by: Harry Wentland <[e-mail]>
> > Cc: Ville Syrjala <[e-mail]>
> > Cc: Pekka Paalanen <[e-mail]>
> > Cc: Simon Ser <[e-mail]>
> > Cc: Harry Wentland <[e-mail]>
> > Cc: Melissa Wen <[e-mail]>
> > Cc: Jonas Ådahl <[e-mail]>
> > Cc: Sebastian Wick <[e-mail]>
> > Cc: Shashank Sharma <[e-mail]>
> > Cc: Alexander Goins <[e-mail]>
> > Cc: Joshua Ashton <[e-mail]>
> > Cc: Michel Dänzer <[e-mail]>
> > Cc: Aleix Pol <[e-mail]>
> > Cc: Xaver Hugl <[e-mail]>
> > Cc: Victoria Brekenfeld <[e-mail]>
> > Cc: Daniel Vetter <[e-mail]>
> > Cc: Uma Shankar <[e-mail]>
> > Cc: Naseer Ahmed <[e-mail]>
> > Cc: Christopher Braga <[e-mail]>
> > ---
> >  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
> >  1 file changed, 278 insertions(+)
> >  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> > 
> > diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
> > new file mode 100644
> > index 000000000000..bfa4a8f12087
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/color_pipeline.rst

...

> > +Color Pipeline Discovery
> > +========================
> > +
> > +A DRM client wanting color management on a drm_plane will:
> > +
> > +1. Read all drm_colorop objects
> > +2. Get the COLOR_PIPELINE property of the plane
> > +3. iterate all COLOR_PIPELINE enum values
> > +4. for each enum value walk the color pipeline (via the NEXT pointers)
> > +   and see if the available color operations are suitable for the
> > +   desired color management operations
> > +
> > +An example of chained properties to define an AMD pre-blending color
> > +pipeline might look like this::  
> 
> Hi Harry,
> 
> Thanks for sharing this proposal. Overall I think it's very aligned with
> Simon's description of the generic KMS color API. I think it's a good
> start point and we can refine over iterations. My general questions have
> already been pointed out by Sebastian and Pekka (mainly regarding the
> BYPASS property).
> 
> I still have some doubts on how to fit these set of colorops with some
> AMD corners cases as below:
> 
> > +
> > +    Plane 10
> > +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> > +    └─ "color_pipeline": enum {0, 42} = 0
> > +    Color operation 42 (input CSC)
> > +    ├─ "type": enum {Bypass, Matrix} = Matrix
> > +    ├─ "matrix_data": blob
> > +    └─ "next": immutable color operation ID = 43  
> 
> IIUC, for input CSC, there are currently two possiblities, or we use
> `drm_plane_color_encoding` and `drm_plane_color range` to get
> pre-defined coefficients or we set a custom matrix here, right? If so, I
> think we need some kind of pre-defined matrix option?
> 
> Also, with this new plane API in place, I understand that we will
> already need think on how to deal with the mixing between old drm color
> properties (color encoding and color range) and these new way of setting
> plane color properties. IIUC, Pekka asked a related question about it
> when talking about CRTC automatic RGB->YUV (?) 

I didn't realize color encoding and color range KMS plane properties
even existed. There is even color space on rockchip!

https://drmdb.emersion.fr/properties?object-type=4008636142

That list has even more conflicts: DEGAMMA_MODE, EOTF, FEATURE,
NV_INPUT_COLORSPACE, SCALING_FILTER, WATERMARK, alpha, GLOBAL_ALPHA,
brightness, colorkey, contrast, and more. I hope most of them are
actually from downstream drivers.

I think they should be forbidden to be used together with the new
pipeline UAPI. Mixing does not work in the long run, it would be
undefined at what position do the old properties apply in a pipeline.

Apparently, we already need a DRM client cap for the new color pipeline
UAPI to hide these legacy things.


This is different from "CRTC automatic RGB->YUV", because the CRTC
thing is chosen silently by the driver and there is nothing after it.
Those old plane properties are explicitly programmed by userspace.


Thanks,
pq

> > +    Color operation 43
> > +    ├─ "type": enum {Scaling} = Scaling
> > +    └─ "next": immutable color operation ID = 44
> > +    Color operation 44 (DeGamma)
> > +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> > +    └─ "next": immutable color operation ID = 45
> > +    Color operation 45 (gamut remap)
> > +    ├─ "type": enum {Bypass, Matrix} = Matrix
> > +    ├─ "matrix_data": blob
> > +    └─ "next": immutable color operation ID = 46
> > +    Color operation 46 (shaper LUT RAM)
> > +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > +    ├─ "1d_curve_type": enum {LUT} = LUT
> > +    ├─ "lut_size": immutable range = 4096
> > +    ├─ "lut_data": blob
> > +    └─ "next": immutable color operation ID = 47  
> 
> For shaper and blend LUT RAM, that the driver supports pre-defined
> curves and custom LUT at the same time but the resulted LUT is a
> combination of those, how to make this behavior clear? Could this
> behavior be described by two colorop in a row: for example, one for
> shaper TF and,just after, another for shaper LUT or would it not be the
> right representation?
> 
> > +    Color operation 47 (3D LUT RAM)
> > +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> > +    ├─ "lut_size": immutable range = 17
> > +    ├─ "lut_data": blob
> > +    └─ "next": immutable color operation ID = 48
> > +    Color operation 48 (blend gamma)
> > +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> > +    ├─ "lut_size": immutable range = 4096
> > +    ├─ "lut_data": blob
> > +    └─ "next": immutable color operation ID = 0
> > +
> > +
> > +Color Pipeline Programming
> > +==========================
> > +
> > +Once a DRM client has found a suitable pipeline it will:
> > +
> > +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
> > +   drm_colorop object of the desired pipeline
> > +2. Set the properties for all drm_colorop objects in the pipeline to the
> > +   desired values, setting BYPASS to true for unused drm_colorop blocks,
> > +   and false for enabled drm_colorop blocks
> > +3. Perform atomic_check/commit as desired
> > +
> > +To configure the pipeline for an HDR10 PQ plane and blending in linear
> > +space, a compositor might perform an atomic commit with the following
> > +property values::
> > +
> > +    Plane 10
> > +    └─ "color_pipeline" = 42
> > +    Color operation 42 (input CSC)
> > +    └─ "bypass" = true
> > +    Color operation 44 (DeGamma)
> > +    └─ "bypass" = true
> > +    Color operation 45 (gamut remap)
> > +    └─ "bypasse" = true
> > +    Color operation 46 (shaper LUT RAM)
> > +    └─ "bypass" = true
> > +    Color operation 47 (3D LUT RAM)
> > +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
> > +    Color operation 48 (blend gamma)
> > +    └─ "1d_curve_type" = PQ inverse EOTF  
> 
> Isn't it a PQ EOTF for blend gamma?
> 
> Best Regards,
> 
> Melissa
> 
> > +
> > +
> > +References
> > +==========
> > +
> > +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> > \ No newline at end of file
> > -- 
> > 2.42.0
> >

Sebastian Wick Oct. 11, 2023, 7:12 p.m. UTC | #7

On Wed, Oct 11, 2023 at 12:20 PM Pekka Paalanen <[e-mail]> wrote:
>
> On Tue, 10 Oct 2023 15:13:46 -0100
> Melissa Wen <[e-mail]> wrote:
>
> > O 09/08, Harry Wentland wrote:
> > > Signed-off-by: Harry Wentland <[e-mail]>
> > > Cc: Ville Syrjala <[e-mail]>
> > > Cc: Pekka Paalanen <[e-mail]>
> > > Cc: Simon Ser <[e-mail]>
> > > Cc: Harry Wentland <[e-mail]>
> > > Cc: Melissa Wen <[e-mail]>
> > > Cc: Jonas Ådahl <[e-mail]>
> > > Cc: Sebastian Wick <[e-mail]>
> > > Cc: Shashank Sharma <[e-mail]>
> > > Cc: Alexander Goins <[e-mail]>
> > > Cc: Joshua Ashton <[e-mail]>
> > > Cc: Michel Dänzer <[e-mail]>
> > > Cc: Aleix Pol <[e-mail]>
> > > Cc: Xaver Hugl <[e-mail]>
> > > Cc: Victoria Brekenfeld <[e-mail]>
> > > Cc: Daniel Vetter <[e-mail]>
> > > Cc: Uma Shankar <[e-mail]>
> > > Cc: Naseer Ahmed <[e-mail]>
> > > Cc: Christopher Braga <[e-mail]>
> > > ---
> > >  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
> > >  1 file changed, 278 insertions(+)
> > >  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> > >
> > > diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
> > > new file mode 100644
> > > index 000000000000..bfa4a8f12087
> > > --- /dev/null
> > > +++ b/Documentation/gpu/rfc/color_pipeline.rst
>
> ...
>
> > > +Color Pipeline Discovery
> > > +========================
> > > +
> > > +A DRM client wanting color management on a drm_plane will:
> > > +
> > > +1. Read all drm_colorop objects
> > > +2. Get the COLOR_PIPELINE property of the plane
> > > +3. iterate all COLOR_PIPELINE enum values
> > > +4. for each enum value walk the color pipeline (via the NEXT pointers)
> > > +   and see if the available color operations are suitable for the
> > > +   desired color management operations
> > > +
> > > +An example of chained properties to define an AMD pre-blending color
> > > +pipeline might look like this::
> >
> > Hi Harry,
> >
> > Thanks for sharing this proposal. Overall I think it's very aligned with
> > Simon's description of the generic KMS color API. I think it's a good
> > start point and we can refine over iterations. My general questions have
> > already been pointed out by Sebastian and Pekka (mainly regarding the
> > BYPASS property).
> >
> > I still have some doubts on how to fit these set of colorops with some
> > AMD corners cases as below:
> >
> > > +
> > > +    Plane 10
> > > +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> > > +    └─ "color_pipeline": enum {0, 42} = 0
> > > +    Color operation 42 (input CSC)
> > > +    ├─ "type": enum {Bypass, Matrix} = Matrix
> > > +    ├─ "matrix_data": blob
> > > +    └─ "next": immutable color operation ID = 43
> >
> > IIUC, for input CSC, there are currently two possiblities, or we use
> > `drm_plane_color_encoding` and `drm_plane_color range` to get
> > pre-defined coefficients or we set a custom matrix here, right? If so, I
> > think we need some kind of pre-defined matrix option?

Seems reasonable. If they are mutually exclusive you might want to
expose 2 different pipelines for it.

> > Also, with this new plane API in place, I understand that we will
> > already need think on how to deal with the mixing between old drm color
> > properties (color encoding and color range) and these new way of setting
> > plane color properties. IIUC, Pekka asked a related question about it
> > when talking about CRTC automatic RGB->YUV (?)

Indeed, good catch! I listed some of them in my proposal more than one
year ago but completely forgot about them already.

>
> I didn't realize color encoding and color range KMS plane properties
> even existed. There is even color space on rockchip!
>
> https://drmdb.emersion.fr/properties?object-type=4008636142
>
> That list has even more conflicts: DEGAMMA_MODE, EOTF, FEATURE,
> NV_INPUT_COLORSPACE, SCALING_FILTER, WATERMARK, alpha, GLOBAL_ALPHA,
> brightness, colorkey, contrast, and more. I hope most of them are
> actually from downstream drivers.
>
> I think they should be forbidden to be used together with the new
> pipeline UAPI. Mixing does not work in the long run, it would be
> undefined at what position do the old properties apply in a pipeline.
>
> Apparently, we already need a DRM client cap for the new color pipeline
> UAPI to hide these legacy things.

Agreed. We'll need one cap for planes and one in the future for CRTCs then.

>
> This is different from "CRTC automatic RGB->YUV", because the CRTC
> thing is chosen silently by the driver and there is nothing after it.
> Those old plane properties are explicitly programmed by userspace.
>
>
> Thanks,
> pq
>
> > > +    Color operation 43
> > > +    ├─ "type": enum {Scaling} = Scaling
> > > +    └─ "next": immutable color operation ID = 44
> > > +    Color operation 44 (DeGamma)
> > > +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > > +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> > > +    └─ "next": immutable color operation ID = 45
> > > +    Color operation 45 (gamut remap)
> > > +    ├─ "type": enum {Bypass, Matrix} = Matrix
> > > +    ├─ "matrix_data": blob
> > > +    └─ "next": immutable color operation ID = 46
> > > +    Color operation 46 (shaper LUT RAM)
> > > +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > > +    ├─ "1d_curve_type": enum {LUT} = LUT
> > > +    ├─ "lut_size": immutable range = 4096
> > > +    ├─ "lut_data": blob
> > > +    └─ "next": immutable color operation ID = 47
> >
> > For shaper and blend LUT RAM, that the driver supports pre-defined
> > curves and custom LUT at the same time but the resulted LUT is a
> > combination of those, how to make this behavior clear? Could this
> > behavior be described by two colorop in a row: for example, one for
> > shaper TF and,just after, another for shaper LUT or would it not be the
> > right representation?
> >
> > > +    Color operation 47 (3D LUT RAM)
> > > +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
> > > +    ├─ "lut_size": immutable range = 17
> > > +    ├─ "lut_data": blob
> > > +    └─ "next": immutable color operation ID = 48
> > > +    Color operation 48 (blend gamma)
> > > +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > > +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
> > > +    ├─ "lut_size": immutable range = 4096
> > > +    ├─ "lut_data": blob
> > > +    └─ "next": immutable color operation ID = 0
> > > +
> > > +
> > > +Color Pipeline Programming
> > > +==========================
> > > +
> > > +Once a DRM client has found a suitable pipeline it will:
> > > +
> > > +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
> > > +   drm_colorop object of the desired pipeline
> > > +2. Set the properties for all drm_colorop objects in the pipeline to the
> > > +   desired values, setting BYPASS to true for unused drm_colorop blocks,
> > > +   and false for enabled drm_colorop blocks
> > > +3. Perform atomic_check/commit as desired
> > > +
> > > +To configure the pipeline for an HDR10 PQ plane and blending in linear
> > > +space, a compositor might perform an atomic commit with the following
> > > +property values::
> > > +
> > > +    Plane 10
> > > +    └─ "color_pipeline" = 42
> > > +    Color operation 42 (input CSC)
> > > +    └─ "bypass" = true
> > > +    Color operation 44 (DeGamma)
> > > +    └─ "bypass" = true
> > > +    Color operation 45 (gamut remap)
> > > +    └─ "bypasse" = true
> > > +    Color operation 46 (shaper LUT RAM)
> > > +    └─ "bypass" = true
> > > +    Color operation 47 (3D LUT RAM)
> > > +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
> > > +    Color operation 48 (blend gamma)
> > > +    └─ "1d_curve_type" = PQ inverse EOTF
> >
> > Isn't it a PQ EOTF for blend gamma?
> >
> > Best Regards,
> >
> > Melissa
> >
> > > +
> > > +
> > > +References
> > > +==========
> > > +
> > > +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> > > \ No newline at end of file
> > > --
> > > 2.42.0
> > >
>

Harry Wentland Oct. 19, 2023, 2:56 p.m. UTC | #8

On 2023-09-13 07:29, Pekka Paalanen wrote:
> On Fri, 8 Sep 2023 11:02:26 -0400
> Harry Wentland <[e-mail]> wrote:
> 
>> Signed-off-by: Harry Wentland <[e-mail]>
>> Cc: Ville Syrjala <[e-mail]>
>> Cc: Pekka Paalanen <[e-mail]>
>> Cc: Simon Ser <[e-mail]>
>> Cc: Harry Wentland <[e-mail]>
>> Cc: Melissa Wen <[e-mail]>
>> Cc: Jonas Ådahl <[e-mail]>
>> Cc: Sebastian Wick <[e-mail]>
>> Cc: Shashank Sharma <[e-mail]>
>> Cc: Alexander Goins <[e-mail]>
>> Cc: Joshua Ashton <[e-mail]>
>> Cc: Michel Dänzer <[e-mail]>
>> Cc: Aleix Pol <[e-mail]>
>> Cc: Xaver Hugl <[e-mail]>
>> Cc: Victoria Brekenfeld <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Uma Shankar <[e-mail]>
>> Cc: Naseer Ahmed <[e-mail]>
>> Cc: Christopher Braga <[e-mail]>
>> ---
>>  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
>>  1 file changed, 278 insertions(+)
>>  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> 
> Hi Harry,
> 
> it's really nice to see this!
> 

Thanks for the feedback. I'm just putting a v2 together with comments
(partially) addressed.

> Sebastian started on the backward/forward compatibility, so I'll
> comment on everything else here, and leave the compatibility for that
> thread.
> 
>> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
>> new file mode 100644
>> index 000000000000..bfa4a8f12087
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/color_pipeline.rst
...
>> +COLOR_PIPELINE Plane Property
>> +=============================
>> +
>> +Because we don't have existing KMS color properties in the pre-blending
>> +portion of display pipelines (i.e. on drm_planes) we are introducing
>> +color pipelines here first. Eventually we'll want to use the same
>> +concept for the post-blending portion, i.e. drm_crtcs.
> 
> This paragraph might fit better in a cover letter.
> 
>> +
>> +Color Pipelines are created by a driver and advertised via a new
>> +COLOR_PIPELINE enum property on each plane. Values of the property
>> +always include '0', which is the default and means all color processing
>> +is disabled. Additional values will be the object IDs of the first
>> +drm_colorop in a pipeline. A driver can create and advertise none, one,
>> +or more possible color pipelines. A DRM client will select a color
>> +pipeline by setting the COLOR PIPELINE to the respective value.
>> +
>> +In the case where drivers have custom support for pre-blending color
>> +processing those drivers shall reject atomic commits that are trying to
>> +set both the custom color properties, as well as the COLOR_PIPELINE
> 
> s/set/use/ because one of them could be carried-over state from
> previous commits while not literally set in this one.
> 
>> +property.
>> +
>> +An example of a COLOR_PIPELINE property on a plane might look like this::
>> +
>> +    Plane 10
>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>> +    ├─ …
>> +    └─ "color_pipeline": enum {0, 42, 52} = 0
> 
> Enum values are string names. I presume the intention here is that the
> strings will never need to be parsed, and the uint64_t is always equal
> to the string representation, right?
> 
> That needs a statement here. It differs from all previous uses of
> enums, and e.g. requires a little bit of new API in libweston's
> DRM-backend to handle since it has its own enums referring to the
> string names that get mapped to the uint64_t per owning KMS object.
> 

I'm currently putting the DRM object ID in the "value" and use the
"name" as a descriptive name.

> struct drm_mode_property_enum {
> 	__u64 value;
> 	char name[DRM_PROP_NAME_LEN];
> };

This works well in IGT and gives us a nice descriptive name for
debugging, but I could consider changing this if it'd simplify
people's lives.

>> +
>> +
>> +Color Pipeline Discovery
>> +========================
>> +
>> +A DRM client wanting color management on a drm_plane will:
>> +
>> +1. Read all drm_colorop objects
> 
> What does this do?

We probably don't need this, and with it we probably don't need
the new IOCTLs. I added this to align with IGT's current init
procedure where it reads all DRM core objects, like planes, etc.,
before using them. But realistically we can just look at the
colorop ID from the COLOR_PIPELINE property and then retrieve
the other colorops through the NEXT pointer.

> 
>> +2. Get the COLOR_PIPELINE property of the plane
>> +3. iterate all COLOR_PIPELINE enum values
>> +4. for each enum value walk the color pipeline (via the NEXT pointers)
>> +   and see if the available color operations are suitable for the
>> +   desired color management operations
>> +
>> +An example of chained properties to define an AMD pre-blending color
>> +pipeline might look like this::
>> +
>> +    Plane 10
>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>> +    └─ "color_pipeline": enum {0, 42} = 0
>> +    Color operation 42 (input CSC)
> 
> I presume the string "(input CSC)" does not come from KMS, and is
> actually just a comment added here by hand?
> 

Exactly. It only exists as a comment here. I'll remove it.

Harry

> 
> Thanks,
> pq
> 
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +    Color operation 43
>> +    ├─ "type": enum {Scaling} = Scaling
>> +    └─ "next": immutable color operation ID = 44
>> +    Color operation 44 (DeGamma)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
>> +    └─ "next": immutable color operation ID = 45
>> +    Color operation 45 (gamut remap)
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 46
>> +    Color operation 46 (shaper LUT RAM)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 47
>> +    Color operation 47 (3D LUT RAM)
>> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
>> +    ├─ "lut_size": immutable range = 17
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 48
>> +    Color operation 48 (blend gamma)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 0
>> +
>> +
>> +Color Pipeline Programming
>> +==========================
>> +
>> +Once a DRM client has found a suitable pipeline it will:
>> +
>> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
>> +   drm_colorop object of the desired pipeline
>> +2. Set the properties for all drm_colorop objects in the pipeline to the
>> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
>> +   and false for enabled drm_colorop blocks
>> +3. Perform atomic_check/commit as desired
>> +
>> +To configure the pipeline for an HDR10 PQ plane and blending in linear
>> +space, a compositor might perform an atomic commit with the following
>> +property values::
>> +
>> +    Plane 10
>> +    └─ "color_pipeline" = 42
>> +    Color operation 42 (input CSC)
>> +    └─ "bypass" = true
>> +    Color operation 44 (DeGamma)
>> +    └─ "bypass" = true
>> +    Color operation 45 (gamut remap)
>> +    └─ "bypasse" = true
>> +    Color operation 46 (shaper LUT RAM)
>> +    └─ "bypass" = true
>> +    Color operation 47 (3D LUT RAM)
>> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
>> +    Color operation 48 (blend gamma)
>> +    └─ "1d_curve_type" = PQ inverse EOTF
>> +
>> +
>> +References
>> +==========
>> +
>> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
>> \ No newline at end of file
>

Harry Wentland Oct. 19, 2023, 2:56 p.m. UTC | #9

On 2023-10-10 12:13, Melissa Wen wrote:
> O 09/08, Harry Wentland wrote:
>> Signed-off-by: Harry Wentland <[e-mail]>
>> Cc: Ville Syrjala <[e-mail]>
>> Cc: Pekka Paalanen <[e-mail]>
>> Cc: Simon Ser <[e-mail]>
>> Cc: Harry Wentland <[e-mail]>
>> Cc: Melissa Wen <[e-mail]>
>> Cc: Jonas Ådahl <[e-mail]>
>> Cc: Sebastian Wick <[e-mail]>
>> Cc: Shashank Sharma <[e-mail]>
>> Cc: Alexander Goins <[e-mail]>
>> Cc: Joshua Ashton <[e-mail]>
>> Cc: Michel Dänzer <[e-mail]>
>> Cc: Aleix Pol <[e-mail]>
>> Cc: Xaver Hugl <[e-mail]>
>> Cc: Victoria Brekenfeld <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Uma Shankar <[e-mail]>
>> Cc: Naseer Ahmed <[e-mail]>
>> Cc: Christopher Braga <[e-mail]>
>> ---
>>  Documentation/gpu/rfc/color_pipeline.rst | 278 +++++++++++++++++++++++
>>  1 file changed, 278 insertions(+)
>>  create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
>>
>> diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
>> new file mode 100644
>> index 000000000000..bfa4a8f12087
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/color_pipeline.rst
>> @@ -0,0 +1,278 @@
>> +========================
>> +Linux Color Pipeline API
>> +========================
>> +
>> +What problem are we solving?
>> +============================
>> +
>> +We would like to support pre-, and post-blending complex color transformations
>> +in order to allow for HW-supported HDR use-cases, as well as to provide support
>> +to color-managed applications, such as video or image editors.
>> +
>> +While it is possible to support an HDR output on HW supporting the Colorspace
>> +and HDR Metadata drm_connector properties that requires the compositor or
>> +application to render and compose the content into one final buffer intended for
>> +display. Doing so is costly.
>> +
>> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
>> +operations to support color transformations. These operations are often
>> +implemented in fixed-function HW and therefore much more power efficient than
>> +performing similar operations via shaders or CPU.
>> +
>> +We would like to make use of this HW functionality to support complex color
>> +transformations with no, or minimal CPU or shader load.
>> +
>> +
>> +How are other OSes solving this problem?
>> +========================================
>> +
>> +The most widely supported use-cases regard HDR content, whether video or
>> +gaming.
>> +
>> +Most OSes will specify the source content format (color gamut, encoding transfer
>> +function, and other metadata, such as max and average light levels) to a driver.
>> +Drivers will then program their fixed-function HW accordingly to map from a
>> +source content buffer's space to a display's space.
>> +
>> +When fixed-function HW is not available the compositor will assemble a shader to
>> +ask the GPU to perform the transformation from the source content format to the
>> +display's format.
>> +
>> +A compositor's mapping function and a driver's mapping function are usually
>> +entirely separate concepts. On OSes where a HW vendor has no insight into
>> +closed-source compositor code such a vendor will tune their color management
>> +code to visually match the compositor's. On other OSes, where both mapping
>> +functions are open to an implementer they will ensure both mappings match.
>> +
>> +
>> +Why is Linux different?
>> +=======================
>> +
>> +Unlike other OSes, where there is one compositor for one or more drivers, on
>> +Linux we have a many-to-many relationship. Many compositors; many drivers.
>> +In addition each compositor vendor or community has their own view of how
>> +color management should be done. This is what makes Linux so beautiful.
>> +
>> +This means that a HW vendor can now no longer tune their driver to one
>> +compositor, as tuning it to one will almost inevitably make it look very
>> +different from another compositor's color mapping.
>> +
>> +We need a better solution.
>> +
>> +
>> +Descriptive API
>> +===============
>> +
>> +An API that describes the source and destination colorspaces is a descriptive
>> +API. It describes the input and output color spaces but does not describe
>> +how precisely they should be mapped. Such a mapping includes many minute
>> +design decision that can greatly affect the look of the final result.
>> +
>> +It is not feasible to describe such mapping with enough detail to ensure the
>> +same result from each implementation. In fact, these mappings are a very active
>> +research area.
>> +
>> +
>> +Prescriptive API
>> +================
>> +
>> +A prescriptive API describes not the source and destination colorspaces. It
>> +instead prescribes a recipe for how to manipulate pixel values to arrive at the
>> +desired outcome.
>> +
>> +This recipe is generally an order straight-forward operations, with clear
>> +mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
>> +operations that can be described in a precise manner.
>> +
>> +
>> +The Color Pipeline API
>> +======================
>> +
>> +HW color management pipelines can significantly differ between HW
>> +vendors in terms of availability, ordering, and capabilities of HW
>> +blocks. This makes a common definition of color management blocks and
>> +their ordering nigh impossible. Instead we are defining an API that
>> +allows user space to discover the HW capabilities.
>> +
>> +
>> +drm_colorop Object & IOCTLs
>> +===========================
>> +
>> +To support the definition of color pipelines we introduce a new DRM core
>> +object, a drm_colorop. Individual drm_colorop objects will be chained
>> +via the NEXT property of a drm_colorop to constitute a color pipeline.
>> +Each drm_colorop object is unique, i.e., even if multiple color
>> +pipelines have the same operation they won't share the same drm_colorop
>> +object to describe that operation.
>> +
>> +Just like other DRM objects the drm_colorop objects are discovered via
>> +IOCTLs:
>> +
>> +DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
>> +number of all drm_colorop objects.
>> +
>> +DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
>> +It includes the ID for the colorop object, as well as the plane_id of
>> +the associated plane. All other values should be registered as
>> +properties.
>> +
>> +Each drm_colorop has three core properties:
>> +
>> +TYPE: The type of transformation, such as
>> +* enumerated curve
>> +* custom (uniform) 1D LUT
>> +* 3x3 matrix
>> +* 3x4 matrix
>> +* 3D LUT
>> +* etc.
>> +
>> +Depending on the type of transformation other properties will describe
>> +more details.
>> +
>> +BYPASS: A boolean property that can be used to easily put a block into
>> +bypass mode. While setting other properties might fail atomic check,
>> +setting the BYPASS property to true should never fail. This allows DRM
>> +clients to fallback to other methods of color management if an atomic
>> +check for KMS color operations fails.
>> +
>> +NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
>> +drm_colorop is the last in the chain.
>> +
>> +An example of a drm_colorop object might look like one of these::
>> +
>> +    Color operation 42
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +
>> +    Color operation 42
>> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
>> +    ├─ "lut_size": immutable range = 33
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +
>> +    Color operation 42
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 43
>> +
>> +
>> +COLOR_PIPELINE Plane Property
>> +=============================
>> +
>> +Because we don't have existing KMS color properties in the pre-blending
>> +portion of display pipelines (i.e. on drm_planes) we are introducing
>> +color pipelines here first. Eventually we'll want to use the same
>> +concept for the post-blending portion, i.e. drm_crtcs.
>> +
>> +Color Pipelines are created by a driver and advertised via a new
>> +COLOR_PIPELINE enum property on each plane. Values of the property
>> +always include '0', which is the default and means all color processing
>> +is disabled. Additional values will be the object IDs of the first
>> +drm_colorop in a pipeline. A driver can create and advertise none, one,
>> +or more possible color pipelines. A DRM client will select a color
>> +pipeline by setting the COLOR PIPELINE to the respective value.
>> +
>> +In the case where drivers have custom support for pre-blending color
>> +processing those drivers shall reject atomic commits that are trying to
>> +set both the custom color properties, as well as the COLOR_PIPELINE
>> +property.
>> +
>> +An example of a COLOR_PIPELINE property on a plane might look like this::
>> +
>> +    Plane 10
>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>> +    ├─ …
>> +    └─ "color_pipeline": enum {0, 42, 52} = 0
>> +
>> +
>> +Color Pipeline Discovery
>> +========================
>> +
>> +A DRM client wanting color management on a drm_plane will:
>> +
>> +1. Read all drm_colorop objects
>> +2. Get the COLOR_PIPELINE property of the plane
>> +3. iterate all COLOR_PIPELINE enum values
>> +4. for each enum value walk the color pipeline (via the NEXT pointers)
>> +   and see if the available color operations are suitable for the
>> +   desired color management operations
>> +
>> +An example of chained properties to define an AMD pre-blending color
>> +pipeline might look like this::
> 
> Hi Harry,
> 
> Thanks for sharing this proposal. Overall I think it's very aligned with
> Simon's description of the generic KMS color API. I think it's a good
> start point and we can refine over iterations. My general questions have
> already been pointed out by Sebastian and Pekka (mainly regarding the
> BYPASS property).
> 
> I still have some doubts on how to fit these set of colorops with some
> AMD corners cases as below:
> 

These aren't the final ones. This is intentionally presented as how
an AMD pre-blending color pipeline "might look like". An actual one
will look different and problem align a bit more to your AMD driver-
specific properties.

>> +
>> +    Plane 10
>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>> +    └─ "color_pipeline": enum {0, 42} = 0
>> +    Color operation 42 (input CSC)
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 43
> 
> IIUC, for input CSC, there are currently two possiblities, or we use
> `drm_plane_color_encoding` and `drm_plane_color range` to get
> pre-defined coefficients or we set a custom matrix here, right? If so, I
> think we need some kind of pre-defined matrix option?
> 

Agreed.

> Also, with this new plane API in place, I understand that we will
> already need think on how to deal with the mixing between old drm color
> properties (color encoding and color range) and these new way of setting
> plane color properties. IIUC, Pekka asked a related question about it
> when talking about CRTC automatic RGB->YUV (?) 
> 

We'll still need to confirm whether we'll want to deprecate these
existing properties. If we do that we'd want a client prop. Things
should still work without deprecating them, if drivers just pick up
after the initial encoding and range CSC.

But realistically it might be better to deprecate them and turn them
into explicit colorops.

>> +    Color operation 43
>> +    ├─ "type": enum {Scaling} = Scaling
>> +    └─ "next": immutable color operation ID = 44
>> +    Color operation 44 (DeGamma)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
>> +    └─ "next": immutable color operation ID = 45
>> +    Color operation 45 (gamut remap)
>> +    ├─ "type": enum {Bypass, Matrix} = Matrix
>> +    ├─ "matrix_data": blob
>> +    └─ "next": immutable color operation ID = 46
>> +    Color operation 46 (shaper LUT RAM)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 47
> 
> For shaper and blend LUT RAM, that the driver supports pre-defined
> curves and custom LUT at the same time but the resulted LUT is a
> combination of those, how to make this behavior clear? Could this
> behavior be described by two colorop in a row: for example, one for
> shaper TF and,just after, another for shaper LUT or would it not be the
> right representation?
> 

Yes. Again, this is only a (simplified) example in order to show how
things could look like on real HW.

>> +    Color operation 47 (3D LUT RAM)
>> +    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
>> +    ├─ "lut_size": immutable range = 17
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 48
>> +    Color operation 48 (blend gamma)
>> +    ├─ "type": enum {Bypass, 1D curve} = 1D curve
>> +    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
>> +    ├─ "lut_size": immutable range = 4096
>> +    ├─ "lut_data": blob
>> +    └─ "next": immutable color operation ID = 0
>> +
>> +
>> +Color Pipeline Programming
>> +==========================
>> +
>> +Once a DRM client has found a suitable pipeline it will:
>> +
>> +1. Set the COLOR_PIPELINE enum value to the one pointing at the first
>> +   drm_colorop object of the desired pipeline
>> +2. Set the properties for all drm_colorop objects in the pipeline to the
>> +   desired values, setting BYPASS to true for unused drm_colorop blocks,
>> +   and false for enabled drm_colorop blocks
>> +3. Perform atomic_check/commit as desired
>> +
>> +To configure the pipeline for an HDR10 PQ plane and blending in linear
>> +space, a compositor might perform an atomic commit with the following
>> +property values::
>> +
>> +    Plane 10
>> +    └─ "color_pipeline" = 42
>> +    Color operation 42 (input CSC)
>> +    └─ "bypass" = true
>> +    Color operation 44 (DeGamma)
>> +    └─ "bypass" = true
>> +    Color operation 45 (gamut remap)
>> +    └─ "bypasse" = true
>> +    Color operation 46 (shaper LUT RAM)
>> +    └─ "bypass" = true
>> +    Color operation 47 (3D LUT RAM)
>> +    └─ "lut_data" = Gamut mapping + tone mapping + night mode
>> +    Color operation 48 (blend gamma)
>> +    └─ "1d_curve_type" = PQ inverse EOTF
> 
> Isn't it a PQ EOTF for blend gamma?
> 

Of course.

Harry

> Best Regards,
> 
> Melissa
> 
>> +
>> +
>> +References
>> +==========
>> +
>> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
>> \ No newline at end of file
>> -- 
>> 2.42.0
>>

Pekka Paalanen Oct. 20, 2023, 10:17 a.m. UTC | #10

On Thu, 19 Oct 2023 10:56:29 -0400
Harry Wentland <[e-mail]> wrote:

> On 2023-09-13 07:29, Pekka Paalanen wrote:
> > On Fri, 8 Sep 2023 11:02:26 -0400
> > Harry Wentland <[e-mail]> wrote:
> >   
> >> Signed-off-by: Harry Wentland <[e-mail]>

...

> >> +COLOR_PIPELINE Plane Property
> >> +=============================
> >> +
> >> +Because we don't have existing KMS color properties in the pre-blending
> >> +portion of display pipelines (i.e. on drm_planes) we are introducing
> >> +color pipelines here first. Eventually we'll want to use the same
> >> +concept for the post-blending portion, i.e. drm_crtcs.  
> > 
> > This paragraph might fit better in a cover letter.
> >   
> >> +
> >> +Color Pipelines are created by a driver and advertised via a new
> >> +COLOR_PIPELINE enum property on each plane. Values of the property
> >> +always include '0', which is the default and means all color processing
> >> +is disabled. Additional values will be the object IDs of the first
> >> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> >> +or more possible color pipelines. A DRM client will select a color
> >> +pipeline by setting the COLOR PIPELINE to the respective value.
> >> +
> >> +In the case where drivers have custom support for pre-blending color
> >> +processing those drivers shall reject atomic commits that are trying to
> >> +set both the custom color properties, as well as the COLOR_PIPELINE  
> > 
> > s/set/use/ because one of them could be carried-over state from
> > previous commits while not literally set in this one.
> >   
> >> +property.
> >> +
> >> +An example of a COLOR_PIPELINE property on a plane might look like this::
> >> +
> >> +    Plane 10
> >> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> >> +    ├─ …
> >> +    └─ "color_pipeline": enum {0, 42, 52} = 0  
> > 
> > Enum values are string names. I presume the intention here is that the
> > strings will never need to be parsed, and the uint64_t is always equal
> > to the string representation, right?
> > 
> > That needs a statement here. It differs from all previous uses of
> > enums, and e.g. requires a little bit of new API in libweston's
> > DRM-backend to handle since it has its own enums referring to the
> > string names that get mapped to the uint64_t per owning KMS object.
> >   
> 
> I'm currently putting the DRM object ID in the "value" and use the
> "name" as a descriptive name.

Would that string name be UAPI? I mean, if userspace hardcodes and
looks for that name, will that keep working? If it's descriptive then I
would assume not, but for every enum existing so far the string name is
UAPI.

> > struct drm_mode_property_enum {
> > 	__u64 value;
> > 	char name[DRM_PROP_NAME_LEN];
> > };  
> 
> This works well in IGT and gives us a nice descriptive name for
> debugging, but I could consider changing this if it'd simplify
> people's lives.

It's nice if we can have a description string for each pipeline, but
according to KMS UAPI conventions so far, userspace would look for the
string name. I'm worried that could backfire to the kernel.

Or, maybe we do want to make the string UAPI and accept that some
userspace might look for it rather than an arrangement of colorops?

Matching colorop sequences is "hard". A developer looking at pipelines,
picking one, and hardcoding its name as a preferred choice would be too
easy. "Works for my cards." IOW, if there is a useful looking string
name, we can be sure that someone will depend on it.


Thanks,
pq

> >> +References
> >> +==========
> >> +
> >> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> >> \ No newline at end of file  
> >   
>

Pekka Paalanen Oct. 20, 2023, 10:36 a.m. UTC | #11

On Thu, 19 Oct 2023 10:56:40 -0400
Harry Wentland <[e-mail]> wrote:

> On 2023-10-10 12:13, Melissa Wen wrote:
> > O 09/08, Harry Wentland wrote:  
> >> Signed-off-by: Harry Wentland <[e-mail]>

...

> > Also, with this new plane API in place, I understand that we will
> > already need think on how to deal with the mixing between old drm color
> > properties (color encoding and color range) and these new way of setting
> > plane color properties. IIUC, Pekka asked a related question about it
> > when talking about CRTC automatic RGB->YUV (?) 
> >   
> 
> We'll still need to confirm whether we'll want to deprecate these
> existing properties. If we do that we'd want a client prop. Things
> should still work without deprecating them, if drivers just pick up
> after the initial encoding and range CSC.
> 
> But realistically it might be better to deprecate them and turn them
> into explicit colorops.

The existing properties would need to be explicitly reflected in the
new pipelines anyway, otherwise there would always be doubt at which
point of a pipeline the old properties apply, and they might even
need to change positions between pipelines.

I think it is simply easier to just hide all old color related
properties when userspace sets the client-cap to enable pipelines. The
problem is to make sure to hide all old properties on all drivers that
support the client-cap.

As a pipeline must be complete (describe everything that happens to
pixel values), it's going to be a flag day per driver.

Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
as well. Maybe it's purely informative and non-configurable, keyed by
FB pixel format, but still.

We also need a colorop to represent sample filtering, e.g. bilinear,
like I think Sebastian may have mentioned in the past. Everything
before the sample filter happens "per tap" as Joshua Ashton put it, and
everything after it happens on the sample that was computed as a
weighted average of the filter tap inputs (texels).

There could be colorops other than sample filtering that operate on
more than one sample at a time, like blur or sharpness. There could
even be colorops that change the image size like adding padding that
the following colorop hardware requires, and then yet another colorop
that clips that padding away. For an example, see
https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html

If that padding and its color can affect the pipeline results of the
pixels near the padding (e.g. some convolution is applied with them,
which may be the reason why padding is necessary to begin with), then
it would be best to model it.


Thanks,
pq

Harry Wentland Nov. 6, 2023, 4:19 p.m. UTC | #12

On 2023-10-20 06:36, Pekka Paalanen wrote:
> On Thu, 19 Oct 2023 10:56:40 -0400
> Harry Wentland <[e-mail]> wrote:
> 
>> On 2023-10-10 12:13, Melissa Wen wrote:
>>> O 09/08, Harry Wentland wrote:  
>>>> Signed-off-by: Harry Wentland <[e-mail]>
> 
> ...
> 
>>> Also, with this new plane API in place, I understand that we will
>>> already need think on how to deal with the mixing between old drm color
>>> properties (color encoding and color range) and these new way of setting
>>> plane color properties. IIUC, Pekka asked a related question about it
>>> when talking about CRTC automatic RGB->YUV (?) 
>>>   
>>
>> We'll still need to confirm whether we'll want to deprecate these
>> existing properties. If we do that we'd want a client prop. Things
>> should still work without deprecating them, if drivers just pick up
>> after the initial encoding and range CSC.
>>
>> But realistically it might be better to deprecate them and turn them
>> into explicit colorops.
> 
> The existing properties would need to be explicitly reflected in the
> new pipelines anyway, otherwise there would always be doubt at which
> point of a pipeline the old properties apply, and they might even
> need to change positions between pipelines.
> 
> I think it is simply easier to just hide all old color related
> properties when userspace sets the client-cap to enable pipelines. The
> problem is to make sure to hide all old properties on all drivers that
> support the client-cap.
> 
> As a pipeline must be complete (describe everything that happens to
> pixel values), it's going to be a flag day per driver.
> 
> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
> as well. Maybe it's purely informative and non-configurable, keyed by
> FB pixel format, but still.
> 
> We also need a colorop to represent sample filtering, e.g. bilinear,
> like I think Sebastian may have mentioned in the past. Everything
> before the sample filter happens "per tap" as Joshua Ashton put it, and
> everything after it happens on the sample that was computed as a
> weighted average of the filter tap inputs (texels).
> 
> There could be colorops other than sample filtering that operate on
> more than one sample at a time, like blur or sharpness. There could
> even be colorops that change the image size like adding padding that
> the following colorop hardware requires, and then yet another colorop
> that clips that padding away. For an example, see
> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
> 
> If that padding and its color can affect the pipeline results of the
> pixels near the padding (e.g. some convolution is applied with them,
> which may be the reason why padding is necessary to begin with), then
> it would be best to model it.
> 

I hear you but I'm also somewhat shying away from defining this at this point.
There are already too many things that need to happen and I will focus on the
actual color blocks (LUTs, matrices) first. We'll always be able to add a new
(read-only) colorop type to define scaling and tap behavior at any point and
a client is free to ignore a color pipeline if it doesn't find any tap/scale
info in it.

Harry

> 
> Thanks,
> pq

Harry Wentland Nov. 6, 2023, 4:24 p.m. UTC | #13

On 2023-10-20 06:17, Pekka Paalanen wrote:
> On Thu, 19 Oct 2023 10:56:29 -0400
> Harry Wentland <[e-mail]> wrote:
> 
>> On 2023-09-13 07:29, Pekka Paalanen wrote:
>>> On Fri, 8 Sep 2023 11:02:26 -0400
>>> Harry Wentland <[e-mail]> wrote:
>>>   
>>>> Signed-off-by: Harry Wentland <[e-mail]>
> 
> ...
> 
>>>> +COLOR_PIPELINE Plane Property
>>>> +=============================
>>>> +
>>>> +Because we don't have existing KMS color properties in the pre-blending
>>>> +portion of display pipelines (i.e. on drm_planes) we are introducing
>>>> +color pipelines here first. Eventually we'll want to use the same
>>>> +concept for the post-blending portion, i.e. drm_crtcs.  
>>>
>>> This paragraph might fit better in a cover letter.
>>>   
>>>> +
>>>> +Color Pipelines are created by a driver and advertised via a new
>>>> +COLOR_PIPELINE enum property on each plane. Values of the property
>>>> +always include '0', which is the default and means all color processing
>>>> +is disabled. Additional values will be the object IDs of the first
>>>> +drm_colorop in a pipeline. A driver can create and advertise none, one,
>>>> +or more possible color pipelines. A DRM client will select a color
>>>> +pipeline by setting the COLOR PIPELINE to the respective value.
>>>> +
>>>> +In the case where drivers have custom support for pre-blending color
>>>> +processing those drivers shall reject atomic commits that are trying to
>>>> +set both the custom color properties, as well as the COLOR_PIPELINE  
>>>
>>> s/set/use/ because one of them could be carried-over state from
>>> previous commits while not literally set in this one.
>>>   
>>>> +property.
>>>> +
>>>> +An example of a COLOR_PIPELINE property on a plane might look like this::
>>>> +
>>>> +    Plane 10
>>>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>>>> +    ├─ …
>>>> +    └─ "color_pipeline": enum {0, 42, 52} = 0  
>>>
>>> Enum values are string names. I presume the intention here is that the
>>> strings will never need to be parsed, and the uint64_t is always equal
>>> to the string representation, right?
>>>
>>> That needs a statement here. It differs from all previous uses of
>>> enums, and e.g. requires a little bit of new API in libweston's
>>> DRM-backend to handle since it has its own enums referring to the
>>> string names that get mapped to the uint64_t per owning KMS object.
>>>   
>>
>> I'm currently putting the DRM object ID in the "value" and use the
>> "name" as a descriptive name.
> 
> Would that string name be UAPI? I mean, if userspace hardcodes and
> looks for that name, will that keep working? If it's descriptive then I
> would assume not, but for every enum existing so far the string name is
> UAPI.
> 

Yes, it's UAPI, as that's how userspace will set the property. The value
is still important to be able to find out which is the first colorop in
the pipeline.

Harry

>>> struct drm_mode_property_enum {
>>> 	__u64 value;
>>> 	char name[DRM_PROP_NAME_LEN];
>>> };  
>>
>> This works well in IGT and gives us a nice descriptive name for
>> debugging, but I could consider changing this if it'd simplify
>> people's lives.
> 
> It's nice if we can have a description string for each pipeline, but
> according to KMS UAPI conventions so far, userspace would look for the
> string name. I'm worried that could backfire to the kernel.
> 
> Or, maybe we do want to make the string UAPI and accept that some
> userspace might look for it rather than an arrangement of colorops?
> 
> Matching colorop sequences is "hard". A developer looking at pipelines,
> picking one, and hardcoding its name as a preferred choice would be too
> easy. "Works for my cards." IOW, if there is a useful looking string
> name, we can be sure that someone will depend on it.
> 
> 
> Thanks,
> pq
> 
>>>> +References
>>>> +==========
>>>> +
>>>> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
>>>> \ No newline at end of file  
>>>   
>>
>

Pekka Paalanen Nov. 7, 2023, 9:38 a.m. UTC | #14

On Mon, 6 Nov 2023 11:24:50 -0500
Harry Wentland <[e-mail]> wrote:

> On 2023-10-20 06:17, Pekka Paalanen wrote:
> > On Thu, 19 Oct 2023 10:56:29 -0400
> > Harry Wentland <[e-mail]> wrote:
> >   
> >> On 2023-09-13 07:29, Pekka Paalanen wrote:  
> >>> On Fri, 8 Sep 2023 11:02:26 -0400
> >>> Harry Wentland <[e-mail]> wrote:
> >>>     
> >>>> Signed-off-by: Harry Wentland <[e-mail]>  
> > 
> > ...
> >   
> >>>> +COLOR_PIPELINE Plane Property
> >>>> +=============================
> >>>> +
> >>>> +Because we don't have existing KMS color properties in the pre-blending
> >>>> +portion of display pipelines (i.e. on drm_planes) we are introducing
> >>>> +color pipelines here first. Eventually we'll want to use the same
> >>>> +concept for the post-blending portion, i.e. drm_crtcs.    
> >>>
> >>> This paragraph might fit better in a cover letter.
> >>>     
> >>>> +
> >>>> +Color Pipelines are created by a driver and advertised via a new
> >>>> +COLOR_PIPELINE enum property on each plane. Values of the property
> >>>> +always include '0', which is the default and means all color processing
> >>>> +is disabled. Additional values will be the object IDs of the first
> >>>> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> >>>> +or more possible color pipelines. A DRM client will select a color
> >>>> +pipeline by setting the COLOR PIPELINE to the respective value.
> >>>> +
> >>>> +In the case where drivers have custom support for pre-blending color
> >>>> +processing those drivers shall reject atomic commits that are trying to
> >>>> +set both the custom color properties, as well as the COLOR_PIPELINE    
> >>>
> >>> s/set/use/ because one of them could be carried-over state from
> >>> previous commits while not literally set in this one.
> >>>     
> >>>> +property.
> >>>> +
> >>>> +An example of a COLOR_PIPELINE property on a plane might look like this::
> >>>> +
> >>>> +    Plane 10
> >>>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> >>>> +    ├─ …
> >>>> +    └─ "color_pipeline": enum {0, 42, 52} = 0    
> >>>
> >>> Enum values are string names. I presume the intention here is that the
> >>> strings will never need to be parsed, and the uint64_t is always equal
> >>> to the string representation, right?
> >>>
> >>> That needs a statement here. It differs from all previous uses of
> >>> enums, and e.g. requires a little bit of new API in libweston's
> >>> DRM-backend to handle since it has its own enums referring to the
> >>> string names that get mapped to the uint64_t per owning KMS object.
> >>>     
> >>
> >> I'm currently putting the DRM object ID in the "value" and use the
> >> "name" as a descriptive name.  
> > 
> > Would that string name be UAPI? I mean, if userspace hardcodes and
> > looks for that name, will that keep working? If it's descriptive then I
> > would assume not, but for every enum existing so far the string name is
> > UAPI.
> >   
> 
> Yes, it's UAPI, as that's how userspace will set the property. The value
> is still important to be able to find out which is the first colorop in
> the pipeline.

Userspace will hardcode string names, look up the KMS uint64_t
corresponding to it, and then use the uint64_t to program KMS.

But for color pipeline objects, the initial idea was that we expect
userspace to look through all available pipelines and see if any of
them can express what userspace wants. This does not need the string
name to be UAPI per se.

Of course, it is easier if userspace can be hardcoded for a specific
color pipeline, so all that matching and searching is avoided, but as a
driver developer, do you want that?

Or maybe the practical end result is the same regardless, because if a
driver removes a pipeline on specific hardware and userspace cannot
find another, that would be a kernel regression anyway.

Then again, if userspace doesn't do the matching and searching, it'll
likely struggle to work on different hardware. Driver developers would
get requests to expose this and that specific pipeline. Is that an ok
prospect?


Thanks,
pq


> >>> struct drm_mode_property_enum {
> >>> 	__u64 value;
> >>> 	char name[DRM_PROP_NAME_LEN];
> >>> };    
> >>
> >> This works well in IGT and gives us a nice descriptive name for
> >> debugging, but I could consider changing this if it'd simplify
> >> people's lives.  
> > 
> > It's nice if we can have a description string for each pipeline, but
> > according to KMS UAPI conventions so far, userspace would look for the
> > string name. I'm worried that could backfire to the kernel.
> > 
> > Or, maybe we do want to make the string UAPI and accept that some
> > userspace might look for it rather than an arrangement of colorops?
> > 
> > Matching colorop sequences is "hard". A developer looking at pipelines,
> > picking one, and hardcoding its name as a preferred choice would be too
> > easy. "Works for my cards." IOW, if there is a useful looking string
> > name, we can be sure that someone will depend on it.
> > 
> > 
> > Thanks,
> > pq
> >   
> >>>> +References
> >>>> +==========
> >>>> +
> >>>> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> >>>> \ No newline at end of file    
> >>>     
> >>  
> >   
>

Pekka Paalanen Nov. 7, 2023, 9:55 a.m. UTC | #15

On Mon, 6 Nov 2023 11:19:27 -0500
Harry Wentland <[e-mail]> wrote:

> On 2023-10-20 06:36, Pekka Paalanen wrote:
> > On Thu, 19 Oct 2023 10:56:40 -0400
> > Harry Wentland <[e-mail]> wrote:
> >   
> >> On 2023-10-10 12:13, Melissa Wen wrote:  
> >>> O 09/08, Harry Wentland wrote:    
> >>>> Signed-off-by: Harry Wentland <[e-mail]>  
> > 
> > ...
> >   
> >>> Also, with this new plane API in place, I understand that we will
> >>> already need think on how to deal with the mixing between old drm color
> >>> properties (color encoding and color range) and these new way of setting
> >>> plane color properties. IIUC, Pekka asked a related question about it
> >>> when talking about CRTC automatic RGB->YUV (?) 
> >>>     
> >>
> >> We'll still need to confirm whether we'll want to deprecate these
> >> existing properties. If we do that we'd want a client prop. Things
> >> should still work without deprecating them, if drivers just pick up
> >> after the initial encoding and range CSC.
> >>
> >> But realistically it might be better to deprecate them and turn them
> >> into explicit colorops.  
> > 
> > The existing properties would need to be explicitly reflected in the
> > new pipelines anyway, otherwise there would always be doubt at which
> > point of a pipeline the old properties apply, and they might even
> > need to change positions between pipelines.
> > 
> > I think it is simply easier to just hide all old color related
> > properties when userspace sets the client-cap to enable pipelines. The
> > problem is to make sure to hide all old properties on all drivers that
> > support the client-cap.
> > 
> > As a pipeline must be complete (describe everything that happens to
> > pixel values), it's going to be a flag day per driver.
> > 
> > Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
> > as well. Maybe it's purely informative and non-configurable, keyed by
> > FB pixel format, but still.
> > 
> > We also need a colorop to represent sample filtering, e.g. bilinear,
> > like I think Sebastian may have mentioned in the past. Everything
> > before the sample filter happens "per tap" as Joshua Ashton put it, and
> > everything after it happens on the sample that was computed as a
> > weighted average of the filter tap inputs (texels).
> > 
> > There could be colorops other than sample filtering that operate on
> > more than one sample at a time, like blur or sharpness. There could
> > even be colorops that change the image size like adding padding that
> > the following colorop hardware requires, and then yet another colorop
> > that clips that padding away. For an example, see
> > https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
> > 
> > If that padding and its color can affect the pipeline results of the
> > pixels near the padding (e.g. some convolution is applied with them,
> > which may be the reason why padding is necessary to begin with), then
> > it would be best to model it.
> >   
> 
> I hear you but I'm also somewhat shying away from defining this at this point.

Would you define them before the new UAPI is released though?

I agree there is no need to have them in this patch series, but I think
we'd hit the below problems if the UAPI is released without them.

> There are already too many things that need to happen and I will focus on the
> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
> (read-only) colorop type to define scaling and tap behavior at any point and
> a client is free to ignore a color pipeline if it doesn't find any tap/scale
> info in it.

How would userspace know to look for tap/scale info, if there is no
upstream definition even on paper?

And the opposite case, if someone writes userspace without tap/scale
colorops, and then drivers add those, and there is no pipeline without
them, because they always exist. Would that userspace disregard all
those pipelines because it does not understand tap/scale colorops,
leaving no usable pipelines? Would that not be kernel regressing
userspace?

If the kernel keeps on exposing pipelines without the colorops, it
fails the basic promise of the whole design: that all pixel value
affecting operations are at least listed if not controllable.

How will we avoid painting ourselves in a corner?

Maybe we need a colorop for "here be dragons" documented as having
unknown and unreliable effects, until driver authors are sure that
everything has been modelled in the pipeline and there are no unknowns?
Or a flag on the pipelines, if we can have that. Then we can at least
tell when the pipeline does not fulfil the basic promise.


Thanks,
pq

Harry Wentland Nov. 7, 2023, 3:39 p.m. UTC | #16

On 2023-11-07 04:38, Pekka Paalanen wrote:
> On Mon, 6 Nov 2023 11:24:50 -0500
> Harry Wentland <[e-mail]> wrote:
> 
>> On 2023-10-20 06:17, Pekka Paalanen wrote:
>>> On Thu, 19 Oct 2023 10:56:29 -0400
>>> Harry Wentland <[e-mail]> wrote:
>>>   
>>>> On 2023-09-13 07:29, Pekka Paalanen wrote:  
>>>>> On Fri, 8 Sep 2023 11:02:26 -0400
>>>>> Harry Wentland <[e-mail]> wrote:
>>>>>     
>>>>>> Signed-off-by: Harry Wentland <[e-mail]>  
>>>
>>> ...
>>>   
>>>>>> +COLOR_PIPELINE Plane Property
>>>>>> +=============================
>>>>>> +
>>>>>> +Because we don't have existing KMS color properties in the pre-blending
>>>>>> +portion of display pipelines (i.e. on drm_planes) we are introducing
>>>>>> +color pipelines here first. Eventually we'll want to use the same
>>>>>> +concept for the post-blending portion, i.e. drm_crtcs.    
>>>>>
>>>>> This paragraph might fit better in a cover letter.
>>>>>     
>>>>>> +
>>>>>> +Color Pipelines are created by a driver and advertised via a new
>>>>>> +COLOR_PIPELINE enum property on each plane. Values of the property
>>>>>> +always include '0', which is the default and means all color processing
>>>>>> +is disabled. Additional values will be the object IDs of the first
>>>>>> +drm_colorop in a pipeline. A driver can create and advertise none, one,
>>>>>> +or more possible color pipelines. A DRM client will select a color
>>>>>> +pipeline by setting the COLOR PIPELINE to the respective value.
>>>>>> +
>>>>>> +In the case where drivers have custom support for pre-blending color
>>>>>> +processing those drivers shall reject atomic commits that are trying to
>>>>>> +set both the custom color properties, as well as the COLOR_PIPELINE    
>>>>>
>>>>> s/set/use/ because one of them could be carried-over state from
>>>>> previous commits while not literally set in this one.
>>>>>     
>>>>>> +property.
>>>>>> +
>>>>>> +An example of a COLOR_PIPELINE property on a plane might look like this::
>>>>>> +
>>>>>> +    Plane 10
>>>>>> +    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
>>>>>> +    ├─ …
>>>>>> +    └─ "color_pipeline": enum {0, 42, 52} = 0    
>>>>>
>>>>> Enum values are string names. I presume the intention here is that the
>>>>> strings will never need to be parsed, and the uint64_t is always equal
>>>>> to the string representation, right?
>>>>>
>>>>> That needs a statement here. It differs from all previous uses of
>>>>> enums, and e.g. requires a little bit of new API in libweston's
>>>>> DRM-backend to handle since it has its own enums referring to the
>>>>> string names that get mapped to the uint64_t per owning KMS object.
>>>>>     
>>>>
>>>> I'm currently putting the DRM object ID in the "value" and use the
>>>> "name" as a descriptive name.  
>>>
>>> Would that string name be UAPI? I mean, if userspace hardcodes and
>>> looks for that name, will that keep working? If it's descriptive then I
>>> would assume not, but for every enum existing so far the string name is
>>> UAPI.
>>>   
>>
>> Yes, it's UAPI, as that's how userspace will set the property. The value
>> is still important to be able to find out which is the first colorop in
>> the pipeline.
> 
> Userspace will hardcode string names, look up the KMS uint64_t
> corresponding to it, and then use the uint64_t to program KMS.
> 
> But for color pipeline objects, the initial idea was that we expect
> userspace to look through all available pipelines and see if any of
> them can express what userspace wants. This does not need the string
> name to be UAPI per se.
> 
> Of course, it is easier if userspace can be hardcoded for a specific
> color pipeline, so all that matching and searching is avoided, but as a
> driver developer, do you want that?
> 
> Or maybe the practical end result is the same regardless, because if a
> driver removes a pipeline on specific hardware and userspace cannot
> find another, that would be a kernel regression anyway.
> 
> Then again, if userspace doesn't do the matching and searching, it'll
> likely struggle to work on different hardware. Driver developers would
> get requests to expose this and that specific pipeline. Is that an ok
> prospect?
> 

IMO in Linux land there will always be two types of DRM clients.

Ones that aim to work on all systems (within reasons) and use as
much of the DRM functionality in a driver-agnostic fashion. These
are things like Weston, Gnome, KDE, sway, etc., aka your "canonical
upstream project[s]" as mentioned in the "Userland Interfaces"
section of the "GPU Driver Developer's Guide".

Then there are people that build custom solutions for custom systems
with no desire or incentive to have them work in a driver-agnostic
fashion. These are often built on top of existing projects, such as
Weston, with customizations that are never designed to go upstream.

In the latter case I think it's entirely fair to hard-code the desired
color pipelines. In the former case I think that would be extremely
short-sighted and go against the driver-agnostic design philosophies
of these compositors.

As a driver developer I would like canonical upstream projects to have
an implementation that does not break with each new HW generation,
meaning these projects should parse the color pipelines.

Harry

> 
> Thanks,
> pq
> 
> 
>>>>> struct drm_mode_property_enum {
>>>>> 	__u64 value;
>>>>> 	char name[DRM_PROP_NAME_LEN];
>>>>> };    
>>>>
>>>> This works well in IGT and gives us a nice descriptive name for
>>>> debugging, but I could consider changing this if it'd simplify
>>>> people's lives.  
>>>
>>> It's nice if we can have a description string for each pipeline, but
>>> according to KMS UAPI conventions so far, userspace would look for the
>>> string name. I'm worried that could backfire to the kernel.
>>>
>>> Or, maybe we do want to make the string UAPI and accept that some
>>> userspace might look for it rather than an arrangement of colorops?
>>>
>>> Matching colorop sequences is "hard". A developer looking at pipelines,
>>> picking one, and hardcoding its name as a preferred choice would be too
>>> easy. "Works for my cards." IOW, if there is a useful looking string
>>> name, we can be sure that someone will depend on it.
>>>
>>>
>>> Thanks,
>>> pq
>>>   
>>>>>> +References
>>>>>> +==========
>>>>>> +
>>>>>> +1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
>>>>>> \ No newline at end of file    
>>>>>     
>>>>  
>>>   
>>
>

Harry Wentland Nov. 7, 2023, 4:58 p.m. UTC | #17

On 2023-11-07 04:55, Pekka Paalanen wrote:
> On Mon, 6 Nov 2023 11:19:27 -0500
> Harry Wentland <[e-mail]> wrote:
> 
>> On 2023-10-20 06:36, Pekka Paalanen wrote:
>>> On Thu, 19 Oct 2023 10:56:40 -0400
>>> Harry Wentland <[e-mail]> wrote:
>>>   
>>>> On 2023-10-10 12:13, Melissa Wen wrote:  
>>>>> O 09/08, Harry Wentland wrote:    
>>>>>> Signed-off-by: Harry Wentland <[e-mail]>  
>>>
>>> ...
>>>   
>>>>> Also, with this new plane API in place, I understand that we will
>>>>> already need think on how to deal with the mixing between old drm color
>>>>> properties (color encoding and color range) and these new way of setting
>>>>> plane color properties. IIUC, Pekka asked a related question about it
>>>>> when talking about CRTC automatic RGB->YUV (?) 
>>>>>     
>>>>
>>>> We'll still need to confirm whether we'll want to deprecate these
>>>> existing properties. If we do that we'd want a client prop. Things
>>>> should still work without deprecating them, if drivers just pick up
>>>> after the initial encoding and range CSC.
>>>>
>>>> But realistically it might be better to deprecate them and turn them
>>>> into explicit colorops.  
>>>
>>> The existing properties would need to be explicitly reflected in the
>>> new pipelines anyway, otherwise there would always be doubt at which
>>> point of a pipeline the old properties apply, and they might even
>>> need to change positions between pipelines.
>>>
>>> I think it is simply easier to just hide all old color related
>>> properties when userspace sets the client-cap to enable pipelines. The
>>> problem is to make sure to hide all old properties on all drivers that
>>> support the client-cap.
>>>
>>> As a pipeline must be complete (describe everything that happens to
>>> pixel values), it's going to be a flag day per driver.
>>>
>>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
>>> as well. Maybe it's purely informative and non-configurable, keyed by
>>> FB pixel format, but still.
>>>
>>> We also need a colorop to represent sample filtering, e.g. bilinear,
>>> like I think Sebastian may have mentioned in the past. Everything
>>> before the sample filter happens "per tap" as Joshua Ashton put it, and
>>> everything after it happens on the sample that was computed as a
>>> weighted average of the filter tap inputs (texels).
>>>
>>> There could be colorops other than sample filtering that operate on
>>> more than one sample at a time, like blur or sharpness. There could
>>> even be colorops that change the image size like adding padding that
>>> the following colorop hardware requires, and then yet another colorop
>>> that clips that padding away. For an example, see
>>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
>>>
>>> If that padding and its color can affect the pipeline results of the
>>> pixels near the padding (e.g. some convolution is applied with them,
>>> which may be the reason why padding is necessary to begin with), then
>>> it would be best to model it.
>>>   
>>
>> I hear you but I'm also somewhat shying away from defining this at this point.
> 
> Would you define them before the new UAPI is released though?
> 
> I agree there is no need to have them in this patch series, but I think
> we'd hit the below problems if the UAPI is released without them.
> 
>> There are already too many things that need to happen and I will focus on the
>> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
>> (read-only) colorop type to define scaling and tap behavior at any point and
>> a client is free to ignore a color pipeline if it doesn't find any tap/scale
>> info in it.
> 
> How would userspace know to look for tap/scale info, if there is no
> upstream definition even on paper?
> 

So far OSes did not care about this. Whether that's good or bad is
something everyone can answer for themselves.

If you write a compositor and really need this you can just ignore
color pipelines that don't have this, i.e., you'll probably want
to wait with implementing color pipeline support until you have what
you need from DRM/KMS.

This is not to say I don't want to have support for Weston. But I'm
wondering if we place too much importance on getting every little
thing figured out whereas we could be making forward progress and
address more aspects of a color pipeline in the future. There is a
reason gamescope has made a huge difference in driving the color
management work forward.

> And the opposite case, if someone writes userspace without tap/scale
> colorops, and then drivers add those, and there is no pipeline without
> them, because they always exist. Would that userspace disregard all
> those pipelines because it does not understand tap/scale colorops,
> leaving no usable pipelines? Would that not be kernel regressing
> userspace?
> 

The simple solution is to leave previously advertised pipelines
untouched and add a new version that does include scaling information.

> If the kernel keeps on exposing pipelines without the colorops, it
> fails the basic promise of the whole design: that all pixel value
> affecting operations are at least listed if not controllable.
> 
> How will we avoid painting ourselves in a corner?
> 
> Maybe we need a colorop for "here be dragons" documented as having
> unknown and unreliable effects, until driver authors are sure that
> everything has been modelled in the pipeline and there are no unknowns?
> Or a flag on the pipelines, if we can have that. Then we can at least
> tell when the pipeline does not fulfil the basic promise.
> 

The will always be dragons at some level.

Harry

> 
> Thanks,
> pq

Pekka Paalanen Nov. 8, 2023, 10:16 a.m. UTC | #18

On Tue, 7 Nov 2023 11:58:26 -0500
Harry Wentland <[e-mail]> wrote:

> On 2023-11-07 04:55, Pekka Paalanen wrote:
> > On Mon, 6 Nov 2023 11:19:27 -0500
> > Harry Wentland <[e-mail]> wrote:
> >   
> >> On 2023-10-20 06:36, Pekka Paalanen wrote:  
> >>> On Thu, 19 Oct 2023 10:56:40 -0400
> >>> Harry Wentland <[e-mail]> wrote:
> >>>     
> >>>> On 2023-10-10 12:13, Melissa Wen wrote:    
> >>>>> O 09/08, Harry Wentland wrote:      
> >>>>>> Signed-off-by: Harry Wentland <[e-mail]>    
> >>>
> >>> ...
> >>>     
> >>>>> Also, with this new plane API in place, I understand that we will
> >>>>> already need think on how to deal with the mixing between old drm color
> >>>>> properties (color encoding and color range) and these new way of setting
> >>>>> plane color properties. IIUC, Pekka asked a related question about it
> >>>>> when talking about CRTC automatic RGB->YUV (?) 
> >>>>>       
> >>>>
> >>>> We'll still need to confirm whether we'll want to deprecate these
> >>>> existing properties. If we do that we'd want a client prop. Things
> >>>> should still work without deprecating them, if drivers just pick up
> >>>> after the initial encoding and range CSC.
> >>>>
> >>>> But realistically it might be better to deprecate them and turn them
> >>>> into explicit colorops.    
> >>>
> >>> The existing properties would need to be explicitly reflected in the
> >>> new pipelines anyway, otherwise there would always be doubt at which
> >>> point of a pipeline the old properties apply, and they might even
> >>> need to change positions between pipelines.
> >>>
> >>> I think it is simply easier to just hide all old color related
> >>> properties when userspace sets the client-cap to enable pipelines. The
> >>> problem is to make sure to hide all old properties on all drivers that
> >>> support the client-cap.
> >>>
> >>> As a pipeline must be complete (describe everything that happens to
> >>> pixel values), it's going to be a flag day per driver.
> >>>
> >>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
> >>> as well. Maybe it's purely informative and non-configurable, keyed by
> >>> FB pixel format, but still.
> >>>
> >>> We also need a colorop to represent sample filtering, e.g. bilinear,
> >>> like I think Sebastian may have mentioned in the past. Everything
> >>> before the sample filter happens "per tap" as Joshua Ashton put it, and
> >>> everything after it happens on the sample that was computed as a
> >>> weighted average of the filter tap inputs (texels).
> >>>
> >>> There could be colorops other than sample filtering that operate on
> >>> more than one sample at a time, like blur or sharpness. There could
> >>> even be colorops that change the image size like adding padding that
> >>> the following colorop hardware requires, and then yet another colorop
> >>> that clips that padding away. For an example, see
> >>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
> >>>
> >>> If that padding and its color can affect the pipeline results of the
> >>> pixels near the padding (e.g. some convolution is applied with them,
> >>> which may be the reason why padding is necessary to begin with), then
> >>> it would be best to model it.
> >>>     
> >>
> >> I hear you but I'm also somewhat shying away from defining this at this point.  
> > 
> > Would you define them before the new UAPI is released though?
> > 
> > I agree there is no need to have them in this patch series, but I think
> > we'd hit the below problems if the UAPI is released without them.
> >   
> >> There are already too many things that need to happen and I will focus on the
> >> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
> >> (read-only) colorop type to define scaling and tap behavior at any point and
> >> a client is free to ignore a color pipeline if it doesn't find any tap/scale
> >> info in it.  
> > 
> > How would userspace know to look for tap/scale info, if there is no
> > upstream definition even on paper?
> >   
> 
> So far OSes did not care about this. Whether that's good or bad is
> something everyone can answer for themselves.
> 
> If you write a compositor and really need this you can just ignore
> color pipelines that don't have this, i.e., you'll probably want
> to wait with implementing color pipeline support until you have what
> you need from DRM/KMS.
> 
> This is not to say I don't want to have support for Weston. But I'm
> wondering if we place too much importance on getting every little
> thing figured out whereas we could be making forward progress and
> address more aspects of a color pipeline in the future. There is a
> reason gamescope has made a huge difference in driving the color
> management work forward.
> 
> > And the opposite case, if someone writes userspace without tap/scale
> > colorops, and then drivers add those, and there is no pipeline without
> > them, because they always exist. Would that userspace disregard all
> > those pipelines because it does not understand tap/scale colorops,
> > leaving no usable pipelines? Would that not be kernel regressing
> > userspace?
> >   
> 
> The simple solution is to leave previously advertised pipelines
> untouched and add a new version that does include scaling information.
> 
> > If the kernel keeps on exposing pipelines without the colorops, it
> > fails the basic promise of the whole design: that all pixel value
> > affecting operations are at least listed if not controllable.
> > 
> > How will we avoid painting ourselves in a corner?
> > 
> > Maybe we need a colorop for "here be dragons" documented as having
> > unknown and unreliable effects, until driver authors are sure that
> > everything has been modelled in the pipeline and there are no unknowns?
> > Or a flag on the pipelines, if we can have that. Then we can at least
> > tell when the pipeline does not fulfil the basic promise.
> >   
> 
> The will always be dragons at some level.

Do I understand right that the goal of fully understood color pipelines
is a lost cause?

That every pipeline might always have something unknown and there is no
way for userspace to know if it does? Maybe because driver developers
don't know either?

By something unknown I refer to anything outside of basic precision
issues. Doing interpolation or mixing of inputs on the wrong side of a
known non-linear colorop, for example.

An incremental UAPI development approach is fine by me, meaning that
pipelines might not be complete at first, but I believe that requires
telling userspace whether the driver developers consider the pipeline
complete (no undescribed operations that would significantly change
results from the expected results given the UAPI exposed pipeline).

The prime example of what I would like to know is that if a FB
contains PQ-encoded image and I use a color pipeline to scale that
image up, will the interpolation happen before or after the non-linear
colorop that decodes PQ. That is a significant difference as pointed
out by Joshua.


Thanks,
pq

Sebastian Wick Nov. 8, 2023, 11:40 a.m. UTC | #19

On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:
>
> On Tue, 7 Nov 2023 11:58:26 -0500
> Harry Wentland <[e-mail]> wrote:
>
> > On 2023-11-07 04:55, Pekka Paalanen wrote:
> > > On Mon, 6 Nov 2023 11:19:27 -0500
> > > Harry Wentland <[e-mail]> wrote:
> > >
> > >> On 2023-10-20 06:36, Pekka Paalanen wrote:
> > >>> On Thu, 19 Oct 2023 10:56:40 -0400
> > >>> Harry Wentland <[e-mail]> wrote:
> > >>>
> > >>>> On 2023-10-10 12:13, Melissa Wen wrote:
> > >>>>> O 09/08, Harry Wentland wrote:
> > >>>>>> Signed-off-by: Harry Wentland <[e-mail]>
> > >>>
> > >>> ...
> > >>>
> > >>>>> Also, with this new plane API in place, I understand that we will
> > >>>>> already need think on how to deal with the mixing between old drm color
> > >>>>> properties (color encoding and color range) and these new way of setting
> > >>>>> plane color properties. IIUC, Pekka asked a related question about it
> > >>>>> when talking about CRTC automatic RGB->YUV (?)
> > >>>>>
> > >>>>
> > >>>> We'll still need to confirm whether we'll want to deprecate these
> > >>>> existing properties. If we do that we'd want a client prop. Things
> > >>>> should still work without deprecating them, if drivers just pick up
> > >>>> after the initial encoding and range CSC.
> > >>>>
> > >>>> But realistically it might be better to deprecate them and turn them
> > >>>> into explicit colorops.
> > >>>
> > >>> The existing properties would need to be explicitly reflected in the
> > >>> new pipelines anyway, otherwise there would always be doubt at which
> > >>> point of a pipeline the old properties apply, and they might even
> > >>> need to change positions between pipelines.
> > >>>
> > >>> I think it is simply easier to just hide all old color related
> > >>> properties when userspace sets the client-cap to enable pipelines. The
> > >>> problem is to make sure to hide all old properties on all drivers that
> > >>> support the client-cap.
> > >>>
> > >>> As a pipeline must be complete (describe everything that happens to
> > >>> pixel values), it's going to be a flag day per driver.
> > >>>
> > >>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
> > >>> as well. Maybe it's purely informative and non-configurable, keyed by
> > >>> FB pixel format, but still.
> > >>>
> > >>> We also need a colorop to represent sample filtering, e.g. bilinear,
> > >>> like I think Sebastian may have mentioned in the past. Everything
> > >>> before the sample filter happens "per tap" as Joshua Ashton put it, and
> > >>> everything after it happens on the sample that was computed as a
> > >>> weighted average of the filter tap inputs (texels).
> > >>>
> > >>> There could be colorops other than sample filtering that operate on
> > >>> more than one sample at a time, like blur or sharpness. There could
> > >>> even be colorops that change the image size like adding padding that
> > >>> the following colorop hardware requires, and then yet another colorop
> > >>> that clips that padding away. For an example, see
> > >>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
> > >>>
> > >>> If that padding and its color can affect the pipeline results of the
> > >>> pixels near the padding (e.g. some convolution is applied with them,
> > >>> which may be the reason why padding is necessary to begin with), then
> > >>> it would be best to model it.
> > >>>
> > >>
> > >> I hear you but I'm also somewhat shying away from defining this at this point.
> > >
> > > Would you define them before the new UAPI is released though?
> > >
> > > I agree there is no need to have them in this patch series, but I think
> > > we'd hit the below problems if the UAPI is released without them.
> > >
> > >> There are already too many things that need to happen and I will focus on the
> > >> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
> > >> (read-only) colorop type to define scaling and tap behavior at any point and
> > >> a client is free to ignore a color pipeline if it doesn't find any tap/scale
> > >> info in it.
> > >
> > > How would userspace know to look for tap/scale info, if there is no
> > > upstream definition even on paper?
> > >
> >
> > So far OSes did not care about this. Whether that's good or bad is
> > something everyone can answer for themselves.
> >
> > If you write a compositor and really need this you can just ignore
> > color pipelines that don't have this, i.e., you'll probably want
> > to wait with implementing color pipeline support until you have what
> > you need from DRM/KMS.
> >
> > This is not to say I don't want to have support for Weston. But I'm
> > wondering if we place too much importance on getting every little
> > thing figured out whereas we could be making forward progress and
> > address more aspects of a color pipeline in the future. There is a
> > reason gamescope has made a huge difference in driving the color
> > management work forward.
> >
> > > And the opposite case, if someone writes userspace without tap/scale
> > > colorops, and then drivers add those, and there is no pipeline without
> > > them, because they always exist. Would that userspace disregard all
> > > those pipelines because it does not understand tap/scale colorops,
> > > leaving no usable pipelines? Would that not be kernel regressing
> > > userspace?
> > >
> >
> > The simple solution is to leave previously advertised pipelines
> > untouched and add a new version that does include scaling information.
> >
> > > If the kernel keeps on exposing pipelines without the colorops, it
> > > fails the basic promise of the whole design: that all pixel value
> > > affecting operations are at least listed if not controllable.
> > >
> > > How will we avoid painting ourselves in a corner?
> > >
> > > Maybe we need a colorop for "here be dragons" documented as having
> > > unknown and unreliable effects, until driver authors are sure that
> > > everything has been modelled in the pipeline and there are no unknowns?
> > > Or a flag on the pipelines, if we can have that. Then we can at least
> > > tell when the pipeline does not fulfil the basic promise.
> > >
> >
> > The will always be dragons at some level.
>
> Do I understand right that the goal of fully understood color pipelines
> is a lost cause?
>
> That every pipeline might always have something unknown and there is no
> way for userspace to know if it does? Maybe because driver developers
> don't know either?
>
> By something unknown I refer to anything outside of basic precision
> issues. Doing interpolation or mixing of inputs on the wrong side of a
> known non-linear colorop, for example.

I don't think that's the case. Hardware vendors should understand the
hardware and exposing everything that affects the values is the goal
here. There will be a transitional period where the pipelines might
not expose every detail yet but that's fine. It's better than what we
have now and should become even better with time. It would maybe be
helpful in the future to have a cap, or property, or whatever, to
indicate that the pipelines are "complete" descriptions of what
happens to the values but we can discuss it when it becomes relevant.

> An incremental UAPI development approach is fine by me, meaning that
> pipelines might not be complete at first, but I believe that requires
> telling userspace whether the driver developers consider the pipeline
> complete (no undescribed operations that would significantly change
> results from the expected results given the UAPI exposed pipeline).
>
> The prime example of what I would like to know is that if a FB
> contains PQ-encoded image and I use a color pipeline to scale that
> image up, will the interpolation happen before or after the non-linear
> colorop that decodes PQ. That is a significant difference as pointed
> out by Joshua.
>
>
> Thanks,
> pq

Harry Wentland Nov. 8, 2023, 2:31 p.m. UTC | #20

On 2023-11-08 06:40, Sebastian Wick wrote:
> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:
>>
>> On Tue, 7 Nov 2023 11:58:26 -0500
>> Harry Wentland <[e-mail]> wrote:
>>
>>> On 2023-11-07 04:55, Pekka Paalanen wrote:
>>>> On Mon, 6 Nov 2023 11:19:27 -0500
>>>> Harry Wentland <[e-mail]> wrote:
>>>>
>>>>> On 2023-10-20 06:36, Pekka Paalanen wrote:
>>>>>> On Thu, 19 Oct 2023 10:56:40 -0400
>>>>>> Harry Wentland <[e-mail]> wrote:
>>>>>>
>>>>>>> On 2023-10-10 12:13, Melissa Wen wrote:
>>>>>>>> O 09/08, Harry Wentland wrote:
>>>>>>>>> Signed-off-by: Harry Wentland <[e-mail]>
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>>> Also, with this new plane API in place, I understand that we will
>>>>>>>> already need think on how to deal with the mixing between old drm color
>>>>>>>> properties (color encoding and color range) and these new way of setting
>>>>>>>> plane color properties. IIUC, Pekka asked a related question about it
>>>>>>>> when talking about CRTC automatic RGB->YUV (?)
>>>>>>>>
>>>>>>>
>>>>>>> We'll still need to confirm whether we'll want to deprecate these
>>>>>>> existing properties. If we do that we'd want a client prop. Things
>>>>>>> should still work without deprecating them, if drivers just pick up
>>>>>>> after the initial encoding and range CSC.
>>>>>>>
>>>>>>> But realistically it might be better to deprecate them and turn them
>>>>>>> into explicit colorops.
>>>>>>
>>>>>> The existing properties would need to be explicitly reflected in the
>>>>>> new pipelines anyway, otherwise there would always be doubt at which
>>>>>> point of a pipeline the old properties apply, and they might even
>>>>>> need to change positions between pipelines.
>>>>>>
>>>>>> I think it is simply easier to just hide all old color related
>>>>>> properties when userspace sets the client-cap to enable pipelines. The
>>>>>> problem is to make sure to hide all old properties on all drivers that
>>>>>> support the client-cap.
>>>>>>
>>>>>> As a pipeline must be complete (describe everything that happens to
>>>>>> pixel values), it's going to be a flag day per driver.
>>>>>>
>>>>>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
>>>>>> as well. Maybe it's purely informative and non-configurable, keyed by
>>>>>> FB pixel format, but still.
>>>>>>
>>>>>> We also need a colorop to represent sample filtering, e.g. bilinear,
>>>>>> like I think Sebastian may have mentioned in the past. Everything
>>>>>> before the sample filter happens "per tap" as Joshua Ashton put it, and
>>>>>> everything after it happens on the sample that was computed as a
>>>>>> weighted average of the filter tap inputs (texels).
>>>>>>
>>>>>> There could be colorops other than sample filtering that operate on
>>>>>> more than one sample at a time, like blur or sharpness. There could
>>>>>> even be colorops that change the image size like adding padding that
>>>>>> the following colorop hardware requires, and then yet another colorop
>>>>>> that clips that padding away. For an example, see
>>>>>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
>>>>>>
>>>>>> If that padding and its color can affect the pipeline results of the
>>>>>> pixels near the padding (e.g. some convolution is applied with them,
>>>>>> which may be the reason why padding is necessary to begin with), then
>>>>>> it would be best to model it.
>>>>>>
>>>>>
>>>>> I hear you but I'm also somewhat shying away from defining this at this point.
>>>>
>>>> Would you define them before the new UAPI is released though?
>>>>
>>>> I agree there is no need to have them in this patch series, but I think
>>>> we'd hit the below problems if the UAPI is released without them.
>>>>
>>>>> There are already too many things that need to happen and I will focus on the
>>>>> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
>>>>> (read-only) colorop type to define scaling and tap behavior at any point and
>>>>> a client is free to ignore a color pipeline if it doesn't find any tap/scale
>>>>> info in it.
>>>>
>>>> How would userspace know to look for tap/scale info, if there is no
>>>> upstream definition even on paper?
>>>>
>>>
>>> So far OSes did not care about this. Whether that's good or bad is
>>> something everyone can answer for themselves.
>>>
>>> If you write a compositor and really need this you can just ignore
>>> color pipelines that don't have this, i.e., you'll probably want
>>> to wait with implementing color pipeline support until you have what
>>> you need from DRM/KMS.
>>>
>>> This is not to say I don't want to have support for Weston. But I'm
>>> wondering if we place too much importance on getting every little
>>> thing figured out whereas we could be making forward progress and
>>> address more aspects of a color pipeline in the future. There is a
>>> reason gamescope has made a huge difference in driving the color
>>> management work forward.
>>>
>>>> And the opposite case, if someone writes userspace without tap/scale
>>>> colorops, and then drivers add those, and there is no pipeline without
>>>> them, because they always exist. Would that userspace disregard all
>>>> those pipelines because it does not understand tap/scale colorops,
>>>> leaving no usable pipelines? Would that not be kernel regressing
>>>> userspace?
>>>>
>>>
>>> The simple solution is to leave previously advertised pipelines
>>> untouched and add a new version that does include scaling information.
>>>
>>>> If the kernel keeps on exposing pipelines without the colorops, it
>>>> fails the basic promise of the whole design: that all pixel value
>>>> affecting operations are at least listed if not controllable.
>>>>
>>>> How will we avoid painting ourselves in a corner?
>>>>
>>>> Maybe we need a colorop for "here be dragons" documented as having
>>>> unknown and unreliable effects, until driver authors are sure that
>>>> everything has been modelled in the pipeline and there are no unknowns?
>>>> Or a flag on the pipelines, if we can have that. Then we can at least
>>>> tell when the pipeline does not fulfil the basic promise.
>>>>
>>>
>>> The will always be dragons at some level.
>>
>> Do I understand right that the goal of fully understood color pipelines
>> is a lost cause?
>>
>> That every pipeline might always have something unknown and there is no
>> way for userspace to know if it does? Maybe because driver developers
>> don't know either?
>>
>> By something unknown I refer to anything outside of basic precision
>> issues. Doing interpolation or mixing of inputs on the wrong side of a
>> known non-linear colorop, for example.
> 
> I don't think that's the case. Hardware vendors should understand the
> hardware and exposing everything that affects the values is the goal
> here. There will be a transitional period where the pipelines might
> not expose every detail yet but that's fine. It's better than what we
> have now and should become even better with time. It would maybe be
> helpful in the future to have a cap, or property, or whatever, to
> indicate that the pipelines are "complete" descriptions of what
> happens to the values but we can discuss it when it becomes relevant.
> 

I agree, for the most part. But how do you then define "complete" if
you exclude precision issues?

>> An incremental UAPI development approach is fine by me, meaning that
>> pipelines might not be complete at first, but I believe that requires
>> telling userspace whether the driver developers consider the pipeline
>> complete (no undescribed operations that would significantly change
>> results from the expected results given the UAPI exposed pipeline).
>>
>> The prime example of what I would like to know is that if a FB
>> contains PQ-encoded image and I use a color pipeline to scale that
>> image up, will the interpolation happen before or after the non-linear
>> colorop that decodes PQ. That is a significant difference as pointed
>> out by Joshua.
>>

That's fair and I want to give that to you. My concern stems from
the sentiment that I hear that any pipeline that doesn't explicitly
advertise this is useless. I don't agree there. Let's not let perfect
be the enemy of good.

Harry

>>
>> Thanks,
>> pq
>

Pekka Paalanen Nov. 8, 2023, 4:19 p.m. UTC | #21

On Wed, 8 Nov 2023 09:31:17 -0500
Harry Wentland <[e-mail]> wrote:

> On 2023-11-08 06:40, Sebastian Wick wrote:
> > On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:  
> >>
> >> On Tue, 7 Nov 2023 11:58:26 -0500
> >> Harry Wentland <[e-mail]> wrote:
> >>  
> >>> On 2023-11-07 04:55, Pekka Paalanen wrote:  
> >>>> On Mon, 6 Nov 2023 11:19:27 -0500
> >>>> Harry Wentland <[e-mail]> wrote:
> >>>>  
> >>>>> On 2023-10-20 06:36, Pekka Paalanen wrote:  
> >>>>>> On Thu, 19 Oct 2023 10:56:40 -0400
> >>>>>> Harry Wentland <[e-mail]> wrote:
> >>>>>>  
> >>>>>>> On 2023-10-10 12:13, Melissa Wen wrote:  
> >>>>>>>> O 09/08, Harry Wentland wrote:  
> >>>>>>>>> Signed-off-by: Harry Wentland <[e-mail]>  
> >>>>>>
> >>>>>> ...
> >>>>>>  
> >>>>>>>> Also, with this new plane API in place, I understand that we will
> >>>>>>>> already need think on how to deal with the mixing between old drm color
> >>>>>>>> properties (color encoding and color range) and these new way of setting
> >>>>>>>> plane color properties. IIUC, Pekka asked a related question about it
> >>>>>>>> when talking about CRTC automatic RGB->YUV (?)
> >>>>>>>>  
> >>>>>>>
> >>>>>>> We'll still need to confirm whether we'll want to deprecate these
> >>>>>>> existing properties. If we do that we'd want a client prop. Things
> >>>>>>> should still work without deprecating them, if drivers just pick up
> >>>>>>> after the initial encoding and range CSC.
> >>>>>>>
> >>>>>>> But realistically it might be better to deprecate them and turn them
> >>>>>>> into explicit colorops.  
> >>>>>>
> >>>>>> The existing properties would need to be explicitly reflected in the
> >>>>>> new pipelines anyway, otherwise there would always be doubt at which
> >>>>>> point of a pipeline the old properties apply, and they might even
> >>>>>> need to change positions between pipelines.
> >>>>>>
> >>>>>> I think it is simply easier to just hide all old color related
> >>>>>> properties when userspace sets the client-cap to enable pipelines. The
> >>>>>> problem is to make sure to hide all old properties on all drivers that
> >>>>>> support the client-cap.
> >>>>>>
> >>>>>> As a pipeline must be complete (describe everything that happens to
> >>>>>> pixel values), it's going to be a flag day per driver.
> >>>>>>
> >>>>>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
> >>>>>> as well. Maybe it's purely informative and non-configurable, keyed by
> >>>>>> FB pixel format, but still.
> >>>>>>
> >>>>>> We also need a colorop to represent sample filtering, e.g. bilinear,
> >>>>>> like I think Sebastian may have mentioned in the past. Everything
> >>>>>> before the sample filter happens "per tap" as Joshua Ashton put it, and
> >>>>>> everything after it happens on the sample that was computed as a
> >>>>>> weighted average of the filter tap inputs (texels).
> >>>>>>
> >>>>>> There could be colorops other than sample filtering that operate on
> >>>>>> more than one sample at a time, like blur or sharpness. There could
> >>>>>> even be colorops that change the image size like adding padding that
> >>>>>> the following colorop hardware requires, and then yet another colorop
> >>>>>> that clips that padding away. For an example, see
> >>>>>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
> >>>>>>
> >>>>>> If that padding and its color can affect the pipeline results of the
> >>>>>> pixels near the padding (e.g. some convolution is applied with them,
> >>>>>> which may be the reason why padding is necessary to begin with), then
> >>>>>> it would be best to model it.
> >>>>>>  
> >>>>>
> >>>>> I hear you but I'm also somewhat shying away from defining this at this point.  
> >>>>
> >>>> Would you define them before the new UAPI is released though?
> >>>>
> >>>> I agree there is no need to have them in this patch series, but I think
> >>>> we'd hit the below problems if the UAPI is released without them.
> >>>>  
> >>>>> There are already too many things that need to happen and I will focus on the
> >>>>> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
> >>>>> (read-only) colorop type to define scaling and tap behavior at any point and
> >>>>> a client is free to ignore a color pipeline if it doesn't find any tap/scale
> >>>>> info in it.  
> >>>>
> >>>> How would userspace know to look for tap/scale info, if there is no
> >>>> upstream definition even on paper?
> >>>>  
> >>>
> >>> So far OSes did not care about this. Whether that's good or bad is
> >>> something everyone can answer for themselves.
> >>>
> >>> If you write a compositor and really need this you can just ignore
> >>> color pipelines that don't have this, i.e., you'll probably want
> >>> to wait with implementing color pipeline support until you have what
> >>> you need from DRM/KMS.
> >>>
> >>> This is not to say I don't want to have support for Weston. But I'm
> >>> wondering if we place too much importance on getting every little
> >>> thing figured out whereas we could be making forward progress and
> >>> address more aspects of a color pipeline in the future. There is a
> >>> reason gamescope has made a huge difference in driving the color
> >>> management work forward.
> >>>  
> >>>> And the opposite case, if someone writes userspace without tap/scale
> >>>> colorops, and then drivers add those, and there is no pipeline without
> >>>> them, because they always exist. Would that userspace disregard all
> >>>> those pipelines because it does not understand tap/scale colorops,
> >>>> leaving no usable pipelines? Would that not be kernel regressing
> >>>> userspace?
> >>>>  
> >>>
> >>> The simple solution is to leave previously advertised pipelines
> >>> untouched and add a new version that does include scaling information.
> >>>  
> >>>> If the kernel keeps on exposing pipelines without the colorops, it
> >>>> fails the basic promise of the whole design: that all pixel value
> >>>> affecting operations are at least listed if not controllable.
> >>>>
> >>>> How will we avoid painting ourselves in a corner?
> >>>>
> >>>> Maybe we need a colorop for "here be dragons" documented as having
> >>>> unknown and unreliable effects, until driver authors are sure that
> >>>> everything has been modelled in the pipeline and there are no unknowns?
> >>>> Or a flag on the pipelines, if we can have that. Then we can at least
> >>>> tell when the pipeline does not fulfil the basic promise.
> >>>>  
> >>>
> >>> The will always be dragons at some level.  
> >>
> >> Do I understand right that the goal of fully understood color pipelines
> >> is a lost cause?
> >>
> >> That every pipeline might always have something unknown and there is no
> >> way for userspace to know if it does? Maybe because driver developers
> >> don't know either?
> >>
> >> By something unknown I refer to anything outside of basic precision
> >> issues. Doing interpolation or mixing of inputs on the wrong side of a
> >> known non-linear colorop, for example.  
> > 
> > I don't think that's the case. Hardware vendors should understand the
> > hardware and exposing everything that affects the values is the goal
> > here. There will be a transitional period where the pipelines might
> > not expose every detail yet but that's fine. It's better than what we
> > have now and should become even better with time. It would maybe be
> > helpful in the future to have a cap, or property, or whatever, to
> > indicate that the pipelines are "complete" descriptions of what
> > happens to the values but we can discuss it when it becomes relevant.
> >   
> 
> I agree, for the most part. But how do you then define "complete" if
> you exclude precision issues?

If someone can develop some kind of precision indication in the UAPI,
we might be able to answer that question then.

> >> An incremental UAPI development approach is fine by me, meaning that
> >> pipelines might not be complete at first, but I believe that requires
> >> telling userspace whether the driver developers consider the pipeline
> >> complete (no undescribed operations that would significantly change
> >> results from the expected results given the UAPI exposed pipeline).
> >>
> >> The prime example of what I would like to know is that if a FB
> >> contains PQ-encoded image and I use a color pipeline to scale that
> >> image up, will the interpolation happen before or after the non-linear
> >> colorop that decodes PQ. That is a significant difference as pointed
> >> out by Joshua.
> >>  
> 
> That's fair and I want to give that to you. My concern stems from
> the sentiment that I hear that any pipeline that doesn't explicitly
> advertise this is useless. I don't agree there. Let's not let perfect
> be the enemy of good.

It's up to the use case. The policy of what is sufficient should reside
in userspace.

What about matching compositor shader composition with KMS?

Can we use that as a rough precision threshold? If userspace implements
the exact same color pipeline as the KMS UAPI describes, then that and
the KMS composited result should be indistinguishable in side-by-side
or alternating visual inspection on any monitor in isolation.

Did this whole effort not start from wanting to off-load things to
display hardware but still maintain visual equivalence to software/GPU
composition?


Thanks,
pq

Harry Wentland Nov. 8, 2023, 4:27 p.m. UTC | #22

On 2023-11-08 11:19, Pekka Paalanen wrote:
> On Wed, 8 Nov 2023 09:31:17 -0500
> Harry Wentland <[e-mail]> wrote:
> 
>> On 2023-11-08 06:40, Sebastian Wick wrote:
>>> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:  
>>>>
>>>> On Tue, 7 Nov 2023 11:58:26 -0500
>>>> Harry Wentland <[e-mail]> wrote:
>>>>  
>>>>> On 2023-11-07 04:55, Pekka Paalanen wrote:  
>>>>>> On Mon, 6 Nov 2023 11:19:27 -0500
>>>>>> Harry Wentland <[e-mail]> wrote:
>>>>>>  
>>>>>>> On 2023-10-20 06:36, Pekka Paalanen wrote:  
>>>>>>>> On Thu, 19 Oct 2023 10:56:40 -0400
>>>>>>>> Harry Wentland <[e-mail]> wrote:
>>>>>>>>  
>>>>>>>>> On 2023-10-10 12:13, Melissa Wen wrote:  
>>>>>>>>>> O 09/08, Harry Wentland wrote:  
>>>>>>>>>>> Signed-off-by: Harry Wentland <[e-mail]>  
>>>>>>>>
>>>>>>>> ...
>>>>>>>>  
>>>>>>>>>> Also, with this new plane API in place, I understand that we will
>>>>>>>>>> already need think on how to deal with the mixing between old drm color
>>>>>>>>>> properties (color encoding and color range) and these new way of setting
>>>>>>>>>> plane color properties. IIUC, Pekka asked a related question about it
>>>>>>>>>> when talking about CRTC automatic RGB->YUV (?)
>>>>>>>>>>  
>>>>>>>>>
>>>>>>>>> We'll still need to confirm whether we'll want to deprecate these
>>>>>>>>> existing properties. If we do that we'd want a client prop. Things
>>>>>>>>> should still work without deprecating them, if drivers just pick up
>>>>>>>>> after the initial encoding and range CSC.
>>>>>>>>>
>>>>>>>>> But realistically it might be better to deprecate them and turn them
>>>>>>>>> into explicit colorops.  
>>>>>>>>
>>>>>>>> The existing properties would need to be explicitly reflected in the
>>>>>>>> new pipelines anyway, otherwise there would always be doubt at which
>>>>>>>> point of a pipeline the old properties apply, and they might even
>>>>>>>> need to change positions between pipelines.
>>>>>>>>
>>>>>>>> I think it is simply easier to just hide all old color related
>>>>>>>> properties when userspace sets the client-cap to enable pipelines. The
>>>>>>>> problem is to make sure to hide all old properties on all drivers that
>>>>>>>> support the client-cap.
>>>>>>>>
>>>>>>>> As a pipeline must be complete (describe everything that happens to
>>>>>>>> pixel values), it's going to be a flag day per driver.
>>>>>>>>
>>>>>>>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
>>>>>>>> as well. Maybe it's purely informative and non-configurable, keyed by
>>>>>>>> FB pixel format, but still.
>>>>>>>>
>>>>>>>> We also need a colorop to represent sample filtering, e.g. bilinear,
>>>>>>>> like I think Sebastian may have mentioned in the past. Everything
>>>>>>>> before the sample filter happens "per tap" as Joshua Ashton put it, and
>>>>>>>> everything after it happens on the sample that was computed as a
>>>>>>>> weighted average of the filter tap inputs (texels).
>>>>>>>>
>>>>>>>> There could be colorops other than sample filtering that operate on
>>>>>>>> more than one sample at a time, like blur or sharpness. There could
>>>>>>>> even be colorops that change the image size like adding padding that
>>>>>>>> the following colorop hardware requires, and then yet another colorop
>>>>>>>> that clips that padding away. For an example, see
>>>>>>>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
>>>>>>>>
>>>>>>>> If that padding and its color can affect the pipeline results of the
>>>>>>>> pixels near the padding (e.g. some convolution is applied with them,
>>>>>>>> which may be the reason why padding is necessary to begin with), then
>>>>>>>> it would be best to model it.
>>>>>>>>  
>>>>>>>
>>>>>>> I hear you but I'm also somewhat shying away from defining this at this point.  
>>>>>>
>>>>>> Would you define them before the new UAPI is released though?
>>>>>>
>>>>>> I agree there is no need to have them in this patch series, but I think
>>>>>> we'd hit the below problems if the UAPI is released without them.
>>>>>>  
>>>>>>> There are already too many things that need to happen and I will focus on the
>>>>>>> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
>>>>>>> (read-only) colorop type to define scaling and tap behavior at any point and
>>>>>>> a client is free to ignore a color pipeline if it doesn't find any tap/scale
>>>>>>> info in it.  
>>>>>>
>>>>>> How would userspace know to look for tap/scale info, if there is no
>>>>>> upstream definition even on paper?
>>>>>>  
>>>>>
>>>>> So far OSes did not care about this. Whether that's good or bad is
>>>>> something everyone can answer for themselves.
>>>>>
>>>>> If you write a compositor and really need this you can just ignore
>>>>> color pipelines that don't have this, i.e., you'll probably want
>>>>> to wait with implementing color pipeline support until you have what
>>>>> you need from DRM/KMS.
>>>>>
>>>>> This is not to say I don't want to have support for Weston. But I'm
>>>>> wondering if we place too much importance on getting every little
>>>>> thing figured out whereas we could be making forward progress and
>>>>> address more aspects of a color pipeline in the future. There is a
>>>>> reason gamescope has made a huge difference in driving the color
>>>>> management work forward.
>>>>>  
>>>>>> And the opposite case, if someone writes userspace without tap/scale
>>>>>> colorops, and then drivers add those, and there is no pipeline without
>>>>>> them, because they always exist. Would that userspace disregard all
>>>>>> those pipelines because it does not understand tap/scale colorops,
>>>>>> leaving no usable pipelines? Would that not be kernel regressing
>>>>>> userspace?
>>>>>>  
>>>>>
>>>>> The simple solution is to leave previously advertised pipelines
>>>>> untouched and add a new version that does include scaling information.
>>>>>  
>>>>>> If the kernel keeps on exposing pipelines without the colorops, it
>>>>>> fails the basic promise of the whole design: that all pixel value
>>>>>> affecting operations are at least listed if not controllable.
>>>>>>
>>>>>> How will we avoid painting ourselves in a corner?
>>>>>>
>>>>>> Maybe we need a colorop for "here be dragons" documented as having
>>>>>> unknown and unreliable effects, until driver authors are sure that
>>>>>> everything has been modelled in the pipeline and there are no unknowns?
>>>>>> Or a flag on the pipelines, if we can have that. Then we can at least
>>>>>> tell when the pipeline does not fulfil the basic promise.
>>>>>>  
>>>>>
>>>>> The will always be dragons at some level.  
>>>>
>>>> Do I understand right that the goal of fully understood color pipelines
>>>> is a lost cause?
>>>>
>>>> That every pipeline might always have something unknown and there is no
>>>> way for userspace to know if it does? Maybe because driver developers
>>>> don't know either?
>>>>
>>>> By something unknown I refer to anything outside of basic precision
>>>> issues. Doing interpolation or mixing of inputs on the wrong side of a
>>>> known non-linear colorop, for example.  
>>>
>>> I don't think that's the case. Hardware vendors should understand the
>>> hardware and exposing everything that affects the values is the goal
>>> here. There will be a transitional period where the pipelines might
>>> not expose every detail yet but that's fine. It's better than what we
>>> have now and should become even better with time. It would maybe be
>>> helpful in the future to have a cap, or property, or whatever, to
>>> indicate that the pipelines are "complete" descriptions of what
>>> happens to the values but we can discuss it when it becomes relevant.
>>>   
>>
>> I agree, for the most part. But how do you then define "complete" if
>> you exclude precision issues?
> 
> If someone can develop some kind of precision indication in the UAPI,
> we might be able to answer that question then.
> 
>>>> An incremental UAPI development approach is fine by me, meaning that
>>>> pipelines might not be complete at first, but I believe that requires
>>>> telling userspace whether the driver developers consider the pipeline
>>>> complete (no undescribed operations that would significantly change
>>>> results from the expected results given the UAPI exposed pipeline).
>>>>
>>>> The prime example of what I would like to know is that if a FB
>>>> contains PQ-encoded image and I use a color pipeline to scale that
>>>> image up, will the interpolation happen before or after the non-linear
>>>> colorop that decodes PQ. That is a significant difference as pointed
>>>> out by Joshua.
>>>>  
>>
>> That's fair and I want to give that to you. My concern stems from
>> the sentiment that I hear that any pipeline that doesn't explicitly
>> advertise this is useless. I don't agree there. Let's not let perfect
>> be the enemy of good.
> 
> It's up to the use case. The policy of what is sufficient should reside
> in userspace.
> 
> What about matching compositor shader composition with KMS?
> 
> Can we use that as a rough precision threshold? If userspace implements
> the exact same color pipeline as the KMS UAPI describes, then that and
> the KMS composited result should be indistinguishable in side-by-side
> or alternating visual inspection on any monitor in isolation.
> 
> Did this whole effort not start from wanting to off-load things to
> display hardware but still maintain visual equivalence to software/GPU
> composition?
> 

I agree with you and I want all that as well.

All I'm saying is that every userspace won't have the same policy of
what is sufficient. Just because Weston has a very high threshold
doesn't mean we can't move forward with a workable solution for other
userspace, as long as we don't do something that prevents us from
doing the perfect solution eventually.

And yes, I do want a solution that works for Weston and hear your
comments on what that requires.

Harry

> 
> Thanks,
> pq

Pekka Paalanen Nov. 9, 2023, 9:20 a.m. UTC | #23

On Wed, 8 Nov 2023 11:27:35 -0500
Harry Wentland <[e-mail]> wrote:

> On 2023-11-08 11:19, Pekka Paalanen wrote:
> > On Wed, 8 Nov 2023 09:31:17 -0500
> > Harry Wentland <[e-mail]> wrote:
> >   
> >> On 2023-11-08 06:40, Sebastian Wick wrote:  
> >>> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:    

...

> >>>> An incremental UAPI development approach is fine by me, meaning that
> >>>> pipelines might not be complete at first, but I believe that requires
> >>>> telling userspace whether the driver developers consider the pipeline
> >>>> complete (no undescribed operations that would significantly change
> >>>> results from the expected results given the UAPI exposed pipeline).
> >>>>
> >>>> The prime example of what I would like to know is that if a FB
> >>>> contains PQ-encoded image and I use a color pipeline to scale that
> >>>> image up, will the interpolation happen before or after the non-linear
> >>>> colorop that decodes PQ. That is a significant difference as pointed
> >>>> out by Joshua.
> >>>>    
> >>
> >> That's fair and I want to give that to you. My concern stems from
> >> the sentiment that I hear that any pipeline that doesn't explicitly
> >> advertise this is useless. I don't agree there. Let's not let perfect
> >> be the enemy of good.  
> > 
> > It's up to the use case. The policy of what is sufficient should reside
> > in userspace.
> > 
> > What about matching compositor shader composition with KMS?
> > 
> > Can we use that as a rough precision threshold? If userspace implements
> > the exact same color pipeline as the KMS UAPI describes, then that and
> > the KMS composited result should be indistinguishable in side-by-side
> > or alternating visual inspection on any monitor in isolation.
> > 
> > Did this whole effort not start from wanting to off-load things to
> > display hardware but still maintain visual equivalence to software/GPU
> > composition?
> >   
> 
> I agree with you and I want all that as well.
> 
> All I'm saying is that every userspace won't have the same policy of
> what is sufficient. Just because Weston has a very high threshold
> doesn't mean we can't move forward with a workable solution for other
> userspace, as long as we don't do something that prevents us from
> doing the perfect solution eventually.
> 
> And yes, I do want a solution that works for Weston and hear your
> comments on what that requires.

I totally agree.

How will that be reflected in the UAPI? If some pipelines are different
from others in correctness/strictness perspective, how will userspace
tell them apart?

Is the current proposal along the lines of: userspace creates a
software pipeline first, and if it cannot map all operations on it to
KMS color pipeline colorops, then the KMS pipeline is not sufficient?

The gist being, if the software pipeline is scaling the image for
example, then it must find a scaling colorop in the KMS pipeline if it
cares about the scaling correctness.

Another example is YUV pixel format on an FB that magically turns into
some kind of RGB when sampled, but there is no colorop to tell what
happens. I suppose userspace cannot assume that the lack of colorop
there means an identity YUV->RGB matrix either? How to model
that? I guess the already mentioned pixel format requirements on a
pipeline would help, making it impossible to use a pipeline without a
YUV->RGB colorop on a YUV FB unless the lack of colorop does indeed
mean an identity matrix.

The same with sub-sampled YUV which probably needs to always(?) be
expanded into fully sampled at the beginning of a pipeline? Chroma
siting.

This is in addition to the previously discussed userspace policy that
if a KMS color pipeline contains colorops the userspace does not
recognise, userspace probably should not pick that pipeline under any
conditions, because it might do something completely unexpected.

I think the above could work, but I feel it requires documenting several
examples like scaling that might not exist in the colorop definitions
at first. Otherwise particularly userspace developers might not come to
think of them, whether they care about those operations. I haven't read
the latest doc yet, so I'm not sure if it's already there.

There is still a gap though: what if the hardware does something
significant that is not modelled in the KMS pipeline with colorops? For
example, always using a hard-wired sRGB curve to decode before blending
and encode after blending. Or that cursor plane always uses the color
pipeline set on the primary plane. How to stop userspace from being
surprised?

Your comments sounded to me like you are letting go of the original
design goals. I'm happy to hear that's not the case. Even if you were,
that is a decision you can make since you are doing the work, and if I
knew you're doing that intentionally I would stop complaining.


Thanks,
pq

Harry Wentland Nov. 10, 2023, 8:27 p.m. UTC | #24

On 2023-11-09 04:20, Pekka Paalanen wrote:
> On Wed, 8 Nov 2023 11:27:35 -0500
> Harry Wentland <[e-mail]> wrote:
> 
>> On 2023-11-08 11:19, Pekka Paalanen wrote:
>>> On Wed, 8 Nov 2023 09:31:17 -0500
>>> Harry Wentland <[e-mail]> wrote:
>>>    
>>>> On 2023-11-08 06:40, Sebastian Wick wrote:
>>>>> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:
> 
> ...
> 
>>>>>> An incremental UAPI development approach is fine by me, meaning that
>>>>>> pipelines might not be complete at first, but I believe that requires
>>>>>> telling userspace whether the driver developers consider the pipeline
>>>>>> complete (no undescribed operations that would significantly change
>>>>>> results from the expected results given the UAPI exposed pipeline).
>>>>>>
>>>>>> The prime example of what I would like to know is that if a FB
>>>>>> contains PQ-encoded image and I use a color pipeline to scale that
>>>>>> image up, will the interpolation happen before or after the non-linear
>>>>>> colorop that decodes PQ. That is a significant difference as pointed
>>>>>> out by Joshua.
>>>>>>     
>>>>
>>>> That's fair and I want to give that to you. My concern stems from
>>>> the sentiment that I hear that any pipeline that doesn't explicitly
>>>> advertise this is useless. I don't agree there. Let's not let perfect
>>>> be the enemy of good.
>>>
>>> It's up to the use case. The policy of what is sufficient should reside
>>> in userspace.
>>>
>>> What about matching compositor shader composition with KMS?
>>>
>>> Can we use that as a rough precision threshold? If userspace implements
>>> the exact same color pipeline as the KMS UAPI describes, then that and
>>> the KMS composited result should be indistinguishable in side-by-side
>>> or alternating visual inspection on any monitor in isolation.
>>>
>>> Did this whole effort not start from wanting to off-load things to
>>> display hardware but still maintain visual equivalence to software/GPU
>>> composition?
>>>    
>>
>> I agree with you and I want all that as well.
>>
>> All I'm saying is that every userspace won't have the same policy of
>> what is sufficient. Just because Weston has a very high threshold
>> doesn't mean we can't move forward with a workable solution for other
>> userspace, as long as we don't do something that prevents us from
>> doing the perfect solution eventually.
>>
>> And yes, I do want a solution that works for Weston and hear your
>> comments on what that requires.
> 
> I totally agree.
> 
> How will that be reflected in the UAPI? If some pipelines are different
> from others in correctness/strictness perspective, how will userspace
> tell them apart?
> 
> Is the current proposal along the lines of: userspace creates a
> software pipeline first, and if it cannot map all operations on it to
> KMS color pipeline colorops, then the KMS pipeline is not sufficient?
> 
> The gist being, if the software pipeline is scaling the image for
> example, then it must find a scaling colorop in the KMS pipeline if it
> cares about the scaling correctness.
> 

With a simplified model of an imaginary color pipeline I expect this
to look like this:

Color Pipeline 1:
   EOTF Curve - CTM

Color Pipeline 2:
   EOTF Curve - scale - CTM

Realistically both would most likely map to the same HW blocks.

Assuming userspace A and B do the following:
   EOTF Curve - scale - CTM

Userspace A doesn't care about scaling and would only look for:
   EOTF Curve - CTM

and find a match with Color Pipeline 1.

Userspace B cares about scaling and would look for
   EOTF Curve - scale - CTM

and find a match with Color Pipeline 2.

If Color Pipeline 2 is not exposed in the first iteration of the
driver's implementation userspace A would still be able to make
use of it, but userspace B would not, since it requires a defined
scale operation. If the driver then exposes Color Pipeline 2 in a
second iteration userspace B can find a match for what it needs
and make use of it.

Realistically userspace B would not attempt to implement a DRM/KMS
color pipeline implementation unless it knows that there's a driver
that can do what it needs.

> Another example is YUV pixel format on an FB that magically turns into
> some kind of RGB when sampled, but there is no colorop to tell what
> happens. I suppose userspace cannot assume that the lack of colorop
> there means an identity YUV->RGB matrix either? How to model
> that? I guess the already mentioned pixel format requirements on a
> pipeline would help, making it impossible to use a pipeline without a
> YUV->RGB colorop on a YUV FB unless the lack of colorop does indeed
> mean an identity matrix.
> 

I agree.

> The same with sub-sampled YUV which probably needs to always(?) be
> expanded into fully sampled at the beginning of a pipeline? Chroma
> siting.
> 
> This is in addition to the previously discussed userspace policy that
> if a KMS color pipeline contains colorops the userspace does not
> recognise, userspace probably should not pick that pipeline under any
> conditions, because it might do something completely unexpected.
> 

Unless those colorops can be put into bypass.

> I think the above could work, but I feel it requires documenting several
> examples like scaling that might not exist in the colorop definitions
> at first. Otherwise particularly userspace developers might not come to
> think of them, whether they care about those operations. I haven't read
> the latest doc yet, so I'm not sure if it's already there.
> 

True.

But I'm somewhat reluctant to define things that don't have an
implementation by a driver and an associated IGT test. I've seen
too many definitions (like the drm_connector Colorspace property)
that define a bunch of things but it's unclear whether they are
actually used. Once you have those you can't change their definition
either, even if they are wrong. And you might not find out they are
wrong until you try to implement support end-to-end.

The age-old chicken-and-egg dilemma. It's really problematic to
define things that haven't been validated end-to-end.

> There is still a gap though: what if the hardware does something
> significant that is not modelled in the KMS pipeline with colorops? For
> example, always using a hard-wired sRGB curve to decode before blending
> and encode after blending. Or that cursor plane always uses the color
> pipeline set on the primary plane. How to stop userspace from being
> surprised?
> 

Yeah, it shouldn't. Anything extra that's done should be modelled with
a colorop. But I might be somewhat contradicting myself here because
this would mean that we'd need to model scaling.

Cursors are funky on AMD and I need to think about them more (though
I've been saying that for years :D ). Maybe on AMD we might want a
custom colorop for cursors that basically says "this plane will inherit
colorops from the underlying plane".

> Your comments sounded to me like you are letting go of the original
> design goals. I'm happy to hear that's not the case. Even if you were,
> that is a decision you can make since you are doing the work, and if I
> knew you're doing that intentionally I would stop complaining.
> 

Apologies for the misunderstanding. I agree with the original design
goals but I'm also trying to find a minimal workable solution that
allows us to iterate and improve on it going forward.

Harry

> 
> Thanks,
> pq

Pekka Paalanen Nov. 13, 2023, 10:01 a.m. UTC | #25

On Fri, 10 Nov 2023 15:27:03 -0500
Harry Wentland <[e-mail]> wrote:

> On 2023-11-09 04:20, Pekka Paalanen wrote:
> > On Wed, 8 Nov 2023 11:27:35 -0500
> > Harry Wentland <[e-mail]> wrote:
> >   
> >> On 2023-11-08 11:19, Pekka Paalanen wrote:  
> >>> On Wed, 8 Nov 2023 09:31:17 -0500
> >>> Harry Wentland <[e-mail]> wrote:
> >>>      
> >>>> On 2023-11-08 06:40, Sebastian Wick wrote:  
> >>>>> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <[e-mail]> wrote:  
> > 
> > ...
> >   
> >>>>>> An incremental UAPI development approach is fine by me, meaning that
> >>>>>> pipelines might not be complete at first, but I believe that requires
> >>>>>> telling userspace whether the driver developers consider the pipeline
> >>>>>> complete (no undescribed operations that would significantly change
> >>>>>> results from the expected results given the UAPI exposed pipeline).
> >>>>>>
> >>>>>> The prime example of what I would like to know is that if a FB
> >>>>>> contains PQ-encoded image and I use a color pipeline to scale that
> >>>>>> image up, will the interpolation happen before or after the non-linear
> >>>>>> colorop that decodes PQ. That is a significant difference as pointed
> >>>>>> out by Joshua.
> >>>>>>       
> >>>>
> >>>> That's fair and I want to give that to you. My concern stems from
> >>>> the sentiment that I hear that any pipeline that doesn't explicitly
> >>>> advertise this is useless. I don't agree there. Let's not let perfect
> >>>> be the enemy of good.  
> >>>
> >>> It's up to the use case. The policy of what is sufficient should reside
> >>> in userspace.
> >>>
> >>> What about matching compositor shader composition with KMS?
> >>>
> >>> Can we use that as a rough precision threshold? If userspace implements
> >>> the exact same color pipeline as the KMS UAPI describes, then that and
> >>> the KMS composited result should be indistinguishable in side-by-side
> >>> or alternating visual inspection on any monitor in isolation.
> >>>
> >>> Did this whole effort not start from wanting to off-load things to
> >>> display hardware but still maintain visual equivalence to software/GPU
> >>> composition?
> >>>      
> >>
> >> I agree with you and I want all that as well.
> >>
> >> All I'm saying is that every userspace won't have the same policy of
> >> what is sufficient. Just because Weston has a very high threshold
> >> doesn't mean we can't move forward with a workable solution for other
> >> userspace, as long as we don't do something that prevents us from
> >> doing the perfect solution eventually.
> >>
> >> And yes, I do want a solution that works for Weston and hear your
> >> comments on what that requires.  
> > 
> > I totally agree.
> > 
> > How will that be reflected in the UAPI? If some pipelines are different
> > from others in correctness/strictness perspective, how will userspace
> > tell them apart?
> > 
> > Is the current proposal along the lines of: userspace creates a
> > software pipeline first, and if it cannot map all operations on it to
> > KMS color pipeline colorops, then the KMS pipeline is not sufficient?
> > 
> > The gist being, if the software pipeline is scaling the image for
> > example, then it must find a scaling colorop in the KMS pipeline if it
> > cares about the scaling correctness.
> >   
> 
> With a simplified model of an imaginary color pipeline I expect this
> to look like this:
> 
> Color Pipeline 1:
>    EOTF Curve - CTM
> 
> Color Pipeline 2:
>    EOTF Curve - scale - CTM
> 
> Realistically both would most likely map to the same HW blocks.
> 
> Assuming userspace A and B do the following:
>    EOTF Curve - scale - CTM
> 
> Userspace A doesn't care about scaling and would only look for:
>    EOTF Curve - CTM
> 
> and find a match with Color Pipeline 1.
> 
> Userspace B cares about scaling and would look for
>    EOTF Curve - scale - CTM
> 
> and find a match with Color Pipeline 2.
> 
> If Color Pipeline 2 is not exposed in the first iteration of the
> driver's implementation userspace A would still be able to make
> use of it, but userspace B would not, since it requires a defined
> scale operation. If the driver then exposes Color Pipeline 2 in a
> second iteration userspace B can find a match for what it needs
> and make use of it.
> 
> Realistically userspace B would not attempt to implement a DRM/KMS
> color pipeline implementation unless it knows that there's a driver
> that can do what it needs.
> 
> > Another example is YUV pixel format on an FB that magically turns into
> > some kind of RGB when sampled, but there is no colorop to tell what
> > happens. I suppose userspace cannot assume that the lack of colorop
> > there means an identity YUV->RGB matrix either? How to model
> > that? I guess the already mentioned pixel format requirements on a
> > pipeline would help, making it impossible to use a pipeline without a
> > YUV->RGB colorop on a YUV FB unless the lack of colorop does indeed
> > mean an identity matrix.
> >   
> 
> I agree.
> 
> > The same with sub-sampled YUV which probably needs to always(?) be
> > expanded into fully sampled at the beginning of a pipeline? Chroma
> > siting.
> > 
> > This is in addition to the previously discussed userspace policy that
> > if a KMS color pipeline contains colorops the userspace does not
> > recognise, userspace probably should not pick that pipeline under any
> > conditions, because it might do something completely unexpected.
> >   
> 
> Unless those colorops can be put into bypass.
> 
> > I think the above could work, but I feel it requires documenting several
> > examples like scaling that might not exist in the colorop definitions
> > at first. Otherwise particularly userspace developers might not come to
> > think of them, whether they care about those operations. I haven't read
> > the latest doc yet, so I'm not sure if it's already there.
> >   
> 
> True.
> 
> But I'm somewhat reluctant to define things that don't have an
> implementation by a driver and an associated IGT test.

Hi Harry,

you do not need to define those colorops.

All I'm asking for is that the documentation includes examples, e.g. if
you do scaling and you care about how and in which color encoding space
it is done, make sure the pipeline you pick explicitly describes that.
If one cannot find even definitions for such colorops, then obviously
they are not explicit.

People might not realise that there are more than one way to do each
thing. That goes for anything that is possible to program or even
automatically programmed without being explicit in a color pipeline,
like YUV<->RGB conversions.

Or is all this so obvious it does not need pointing out in docs?

> I've seen
> too many definitions (like the drm_connector Colorspace property)
> that define a bunch of things but it's unclear whether they are
> actually used. Once you have those you can't change their definition
> either, even if they are wrong. And you might not find out they are
> wrong until you try to implement support end-to-end.
> 
> The age-old chicken-and-egg dilemma. It's really problematic to
> define things that haven't been validated end-to-end.
> 
> > There is still a gap though: what if the hardware does something
> > significant that is not modelled in the KMS pipeline with colorops? For
> > example, always using a hard-wired sRGB curve to decode before blending
> > and encode after blending. Or that cursor plane always uses the color
> > pipeline set on the primary plane. How to stop userspace from being
> > surprised?
> >   
> 
> Yeah, it shouldn't. Anything extra that's done should be modelled with
> a colorop. But I might be somewhat contradicting myself here because
> this would mean that we'd need to model scaling.

Exactly. If unknowns need to be allowed at first, there should be some
indication to userspace when all unknowns in a pipeline have been
eliminated (either explicitly modelled, or guaranteed to bypass). Most
userspace won't care, but those who would need the new color pipeline
UAPI the most will.

> Cursors are funky on AMD and I need to think about them more (though
> I've been saying that for years :D ). Maybe on AMD we might want a
> custom colorop for cursors that basically says "this plane will inherit
> colorops from the underlying plane".

Sounds good to me.

Is it really any underlying plane at that point (pixel by pixel?) and
not the primary plane?

> > Your comments sounded to me like you are letting go of the original
> > design goals. I'm happy to hear that's not the case. Even if you were,
> > that is a decision you can make since you are doing the work, and if I
> > knew you're doing that intentionally I would stop complaining.
> >   
> 
> Apologies for the misunderstanding. I agree with the original design
> goals but I'm also trying to find a minimal workable solution that
> allows us to iterate and improve on it going forward.

That's all fine.


Thanks,
pq

Patch

diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst
new file mode 100644
index 000000000000..bfa4a8f12087
--- /dev/null
+++ b/Documentation/gpu/rfc/color_pipeline.rst
@@ -0,0 +1,278 @@ 
+========================
+Linux Color Pipeline API
+========================
+
+What problem are we solving?
+============================
+
+We would like to support pre-, and post-blending complex color transformations
+in order to allow for HW-supported HDR use-cases, as well as to provide support
+to color-managed applications, such as video or image editors.
+
+While it is possible to support an HDR output on HW supporting the Colorspace
+and HDR Metadata drm_connector properties that requires the compositor or
+application to render and compose the content into one final buffer intended for
+display. Doing so is costly.
+
+Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other
+operations to support color transformations. These operations are often
+implemented in fixed-function HW and therefore much more power efficient than
+performing similar operations via shaders or CPU.
+
+We would like to make use of this HW functionality to support complex color
+transformations with no, or minimal CPU or shader load.
+
+
+How are other OSes solving this problem?
+========================================
+
+The most widely supported use-cases regard HDR content, whether video or
+gaming.
+
+Most OSes will specify the source content format (color gamut, encoding transfer
+function, and other metadata, such as max and average light levels) to a driver.
+Drivers will then program their fixed-function HW accordingly to map from a
+source content buffer's space to a display's space.
+
+When fixed-function HW is not available the compositor will assemble a shader to
+ask the GPU to perform the transformation from the source content format to the
+display's format.
+
+A compositor's mapping function and a driver's mapping function are usually
+entirely separate concepts. On OSes where a HW vendor has no insight into
+closed-source compositor code such a vendor will tune their color management
+code to visually match the compositor's. On other OSes, where both mapping
+functions are open to an implementer they will ensure both mappings match.
+
+
+Why is Linux different?
+=======================
+
+Unlike other OSes, where there is one compositor for one or more drivers, on
+Linux we have a many-to-many relationship. Many compositors; many drivers.
+In addition each compositor vendor or community has their own view of how
+color management should be done. This is what makes Linux so beautiful.
+
+This means that a HW vendor can now no longer tune their driver to one
+compositor, as tuning it to one will almost inevitably make it look very
+different from another compositor's color mapping.
+
+We need a better solution.
+
+
+Descriptive API
+===============
+
+An API that describes the source and destination colorspaces is a descriptive
+API. It describes the input and output color spaces but does not describe
+how precisely they should be mapped. Such a mapping includes many minute
+design decision that can greatly affect the look of the final result.
+
+It is not feasible to describe such mapping with enough detail to ensure the
+same result from each implementation. In fact, these mappings are a very active
+research area.
+
+
+Prescriptive API
+================
+
+A prescriptive API describes not the source and destination colorspaces. It
+instead prescribes a recipe for how to manipulate pixel values to arrive at the
+desired outcome.
+
+This recipe is generally an order straight-forward operations, with clear
+mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, or other
+operations that can be described in a precise manner.
+
+
+The Color Pipeline API
+======================
+
+HW color management pipelines can significantly differ between HW
+vendors in terms of availability, ordering, and capabilities of HW
+blocks. This makes a common definition of color management blocks and
+their ordering nigh impossible. Instead we are defining an API that
+allows user space to discover the HW capabilities.
+
+
+drm_colorop Object & IOCTLs
+===========================
+
+To support the definition of color pipelines we introduce a new DRM core
+object, a drm_colorop. Individual drm_colorop objects will be chained
+via the NEXT property of a drm_colorop to constitute a color pipeline.
+Each drm_colorop object is unique, i.e., even if multiple color
+pipelines have the same operation they won't share the same drm_colorop
+object to describe that operation.
+
+Just like other DRM objects the drm_colorop objects are discovered via
+IOCTLs:
+
+DRM_IOCTL_MODE_GETCOLOROPRESOURCES: This IOCTL is used to retrieve the
+number of all drm_colorop objects.
+
+DRM_IOCTL_MODE_GETCOLOROP: This IOCTL is used to read one drm_colorop.
+It includes the ID for the colorop object, as well as the plane_id of
+the associated plane. All other values should be registered as
+properties.
+
+Each drm_colorop has three core properties:
+
+TYPE: The type of transformation, such as
+* enumerated curve
+* custom (uniform) 1D LUT
+* 3x3 matrix
+* 3x4 matrix
+* 3D LUT
+* etc.
+
+Depending on the type of transformation other properties will describe
+more details.
+
+BYPASS: A boolean property that can be used to easily put a block into
+bypass mode. While setting other properties might fail atomic check,
+setting the BYPASS property to true should never fail. This allows DRM
+clients to fallback to other methods of color management if an atomic
+check for KMS color operations fails.
+
+NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this
+drm_colorop is the last in the chain.
+
+An example of a drm_colorop object might look like one of these::
+
+    Color operation 42
+    ├─ "type": enum {Bypass, 1D curve} = 1D curve
+    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
+    ├─ "lut_size": immutable range = 4096
+    ├─ "lut_data": blob
+    └─ "next": immutable color operation ID = 43
+
+    Color operation 42
+    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
+    ├─ "lut_size": immutable range = 33
+    ├─ "lut_data": blob
+    └─ "next": immutable color operation ID = 43
+
+    Color operation 42
+    ├─ "type": enum {Bypass, Matrix} = Matrix
+    ├─ "matrix_data": blob
+    └─ "next": immutable color operation ID = 43
+
+
+COLOR_PIPELINE Plane Property
+=============================
+
+Because we don't have existing KMS color properties in the pre-blending
+portion of display pipelines (i.e. on drm_planes) we are introducing
+color pipelines here first. Eventually we'll want to use the same
+concept for the post-blending portion, i.e. drm_crtcs.
+
+Color Pipelines are created by a driver and advertised via a new
+COLOR_PIPELINE enum property on each plane. Values of the property
+always include '0', which is the default and means all color processing
+is disabled. Additional values will be the object IDs of the first
+drm_colorop in a pipeline. A driver can create and advertise none, one,
+or more possible color pipelines. A DRM client will select a color
+pipeline by setting the COLOR PIPELINE to the respective value.
+
+In the case where drivers have custom support for pre-blending color
+processing those drivers shall reject atomic commits that are trying to
+set both the custom color properties, as well as the COLOR_PIPELINE
+property.
+
+An example of a COLOR_PIPELINE property on a plane might look like this::
+
+    Plane 10
+    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
+    ├─ …
+    └─ "color_pipeline": enum {0, 42, 52} = 0
+
+
+Color Pipeline Discovery
+========================
+
+A DRM client wanting color management on a drm_plane will:
+
+1. Read all drm_colorop objects
+2. Get the COLOR_PIPELINE property of the plane
+3. iterate all COLOR_PIPELINE enum values
+4. for each enum value walk the color pipeline (via the NEXT pointers)
+   and see if the available color operations are suitable for the
+   desired color management operations
+
+An example of chained properties to define an AMD pre-blending color
+pipeline might look like this::
+
+    Plane 10
+    ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
+    └─ "color_pipeline": enum {0, 42} = 0
+    Color operation 42 (input CSC)
+    ├─ "type": enum {Bypass, Matrix} = Matrix
+    ├─ "matrix_data": blob
+    └─ "next": immutable color operation ID = 43
+    Color operation 43
+    ├─ "type": enum {Scaling} = Scaling
+    └─ "next": immutable color operation ID = 44
+    Color operation 44 (DeGamma)
+    ├─ "type": enum {Bypass, 1D curve} = 1D curve
+    ├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
+    └─ "next": immutable color operation ID = 45
+    Color operation 45 (gamut remap)
+    ├─ "type": enum {Bypass, Matrix} = Matrix
+    ├─ "matrix_data": blob
+    └─ "next": immutable color operation ID = 46
+    Color operation 46 (shaper LUT RAM)
+    ├─ "type": enum {Bypass, 1D curve} = 1D curve
+    ├─ "1d_curve_type": enum {LUT} = LUT
+    ├─ "lut_size": immutable range = 4096
+    ├─ "lut_data": blob
+    └─ "next": immutable color operation ID = 47
+    Color operation 47 (3D LUT RAM)
+    ├─ "type": enum {Bypass, 3D LUT} = 3D LUT
+    ├─ "lut_size": immutable range = 17
+    ├─ "lut_data": blob
+    └─ "next": immutable color operation ID = 48
+    Color operation 48 (blend gamma)
+    ├─ "type": enum {Bypass, 1D curve} = 1D curve
+    ├─ "1d_curve_type": enum {LUT, sRGB, PQ, …} = LUT
+    ├─ "lut_size": immutable range = 4096
+    ├─ "lut_data": blob
+    └─ "next": immutable color operation ID = 0
+
+
+Color Pipeline Programming
+==========================
+
+Once a DRM client has found a suitable pipeline it will:
+
+1. Set the COLOR_PIPELINE enum value to the one pointing at the first
+   drm_colorop object of the desired pipeline
+2. Set the properties for all drm_colorop objects in the pipeline to the
+   desired values, setting BYPASS to true for unused drm_colorop blocks,
+   and false for enabled drm_colorop blocks
+3. Perform atomic_check/commit as desired
+
+To configure the pipeline for an HDR10 PQ plane and blending in linear
+space, a compositor might perform an atomic commit with the following
+property values::
+
+    Plane 10
+    └─ "color_pipeline" = 42
+    Color operation 42 (input CSC)
+    └─ "bypass" = true
+    Color operation 44 (DeGamma)
+    └─ "bypass" = true
+    Color operation 45 (gamut remap)
+    └─ "bypasse" = true
+    Color operation 46 (shaper LUT RAM)
+    └─ "bypass" = true
+    Color operation 47 (3D LUT RAM)
+    └─ "lut_data" = Gamut mapping + tone mapping + night mode
+    Color operation 48 (blend gamma)
+    └─ "1d_curve_type" = PQ inverse EOTF
+
+
+References
+==========
+
+1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
\ No newline at end of file