| Message ID | 20240214112457.3734871-1-nemesa.garg@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Introduce drm sharpening property | expand |
How much of this is Intel-specific? Are there any hardware vendors with a similar feature? How much is the "sharpness" knob tied to Intel hardware?
Adding a couple of compositor devs as they might be interested in this. On 2024-02-14 06:24, Nemesa Garg wrote: > Many a times images are blurred or upscaled content is also not as > crisp as original rendered image. Traditional sharpening techniques often > apply a uniform level of enhancement across entire image, which sometimes > result in over-sharpening of some areas and potential loss of natural details. > > Intel has come up with Display Engine based adaptive sharpening filter > with minimal power and performance impact. From LNL onwards, the Display > hardware can use one of the pipe scaler for adaptive sharpness filter. > This can be used for both gaming and non-gaming use cases like photos, > image viewing. It works on a region of pixels depending on the tap size. > > This RFC is an attempt to introduce an adaptive sharpness solution which > helps in improving the image quality. For this new CRTC property is added. I don't think CRTC is the right place for this. Scaling tends to be more of a plane thing. Planes can be scaled independently, or is that not the case for Intel? Or does Intel HW do this sharpening operation independent of any scaling, on the entire output? > The user can set this property with desired sharpness strength value with > 0-255. A value of 1 representing minimum sharpening strength and 255 > representing maximum sharpness strength. A strength value of 0 means no > sharpening or sharpening feature disabled. > It works on a region of pixels depending on the tap size. The coefficients > are used to generate an alpha value which is used to blend the sharpened > image to original image. > > Userspace implementation for sharpening feature and IGT implementation > is in progress. It would be very helpful to have an idea how this looks in userspace, and which compositors will implement this. Harry > > Nemesa Garg (5): > drm: Introduce sharpeness mode property > drm/i915/display/: Compute the scaler filter coefficients > drm/i915/dispaly/: Enable the second scaler > drm/i915/display/: Add registers and compute the strength > drm/i915/display: Load the lut values and enable sharpness > > drivers/gpu/drm/drm_atomic_uapi.c | 4 + > drivers/gpu/drm/drm_crtc.c | 17 ++ > drivers/gpu/drm/i915/Makefile | 1 + > drivers/gpu/drm/i915/display/intel_crtc.c | 3 + > drivers/gpu/drm/i915/display/intel_display.c | 20 +- > .../drm/i915/display/intel_display_types.h | 11 + > .../drm/i915/display/intel_modeset_verify.c | 1 + > .../drm/i915/display/intel_sharpen_filter.c | 214 ++++++++++++++++++ > .../drm/i915/display/intel_sharpen_filter.h | 31 +++ > drivers/gpu/drm/i915/display/skl_scaler.c | 86 ++++++- > drivers/gpu/drm/i915/display/skl_scaler.h | 1 + > drivers/gpu/drm/i915/i915_reg.h | 19 ++ > drivers/gpu/drm/xe/Makefile | 1 + > include/drm/drm_crtc.h | 17 ++ > 14 files changed, 416 insertions(+), 10 deletions(-) > create mode 100644 drivers/gpu/drm/i915/display/intel_sharpen_filter.c > create mode 100644 drivers/gpu/drm/i915/display/intel_sharpen_filter.h >
On Thu, Feb 15, 2024 at 11:37:54AM -0500, Harry Wentland wrote: > Adding a couple of compositor devs as they might be interested in this. > > On 2024-02-14 06:24, Nemesa Garg wrote: > > Many a times images are blurred or upscaled content is also not as > > crisp as original rendered image. Traditional sharpening techniques often > > apply a uniform level of enhancement across entire image, which sometimes > > result in over-sharpening of some areas and potential loss of natural details. > > > > Intel has come up with Display Engine based adaptive sharpening filter > > with minimal power and performance impact. From LNL onwards, the Display > > hardware can use one of the pipe scaler for adaptive sharpness filter. > > This can be used for both gaming and non-gaming use cases like photos, > > image viewing. It works on a region of pixels depending on the tap size. > > > > This RFC is an attempt to introduce an adaptive sharpness solution which > > helps in improving the image quality. For this new CRTC property is added. > > I don't think CRTC is the right place for this. Scaling tends to be more > of a plane thing. Planes can be scaled independently, or is that not the > case for Intel? Or does Intel HW do this sharpening operation independent > of any scaling, on the entire output? We can scale either individual planes, or the entire crtc. > > > The user can set this property with desired sharpness strength value with > > 0-255. A value of 1 representing minimum sharpening strength and 255 > > representing maximum sharpness strength. A strength value of 0 means no > > sharpening or sharpening feature disabled. > > It works on a region of pixels depending on the tap size. The coefficients > > are used to generate an alpha value which is used to blend the sharpened > > image to original image. > > > > Userspace implementation for sharpening feature and IGT implementation > > is in progress. > > It would be very helpful to have an idea how this looks in userspace, and > which compositors will implement this. Someone will probably need to spend some real time thinking how this interacts with the scaling filter propery (eg. if we want to extend that property with new values) and the laptop panel scaling mode property. We also want to implement the margin properties for external displays which also involves scaling. Ie. we need some kind of consistent story how all those things will work together. > > Harry > > > > > Nemesa Garg (5): > > drm: Introduce sharpeness mode property > > drm/i915/display/: Compute the scaler filter coefficients > > drm/i915/dispaly/: Enable the second scaler > > drm/i915/display/: Add registers and compute the strength > > drm/i915/display: Load the lut values and enable sharpness > > > > drivers/gpu/drm/drm_atomic_uapi.c | 4 + > > drivers/gpu/drm/drm_crtc.c | 17 ++ > > drivers/gpu/drm/i915/Makefile | 1 + > > drivers/gpu/drm/i915/display/intel_crtc.c | 3 + > > drivers/gpu/drm/i915/display/intel_display.c | 20 +- > > .../drm/i915/display/intel_display_types.h | 11 + > > .../drm/i915/display/intel_modeset_verify.c | 1 + > > .../drm/i915/display/intel_sharpen_filter.c | 214 ++++++++++++++++++ > > .../drm/i915/display/intel_sharpen_filter.h | 31 +++ > > drivers/gpu/drm/i915/display/skl_scaler.c | 86 ++++++- > > drivers/gpu/drm/i915/display/skl_scaler.h | 1 + > > drivers/gpu/drm/i915/i915_reg.h | 19 ++ > > drivers/gpu/drm/xe/Makefile | 1 + > > include/drm/drm_crtc.h | 17 ++ > > 14 files changed, 416 insertions(+), 10 deletions(-) > > create mode 100644 drivers/gpu/drm/i915/display/intel_sharpen_filter.c > > create mode 100644 drivers/gpu/drm/i915/display/intel_sharpen_filter.h > >
It is not intel specific and the goal is to have a generic API for configuring Sharpness, accessible to various vendors. Intel currently offers sharpness support through the Display Engine, while other vendors seem to support Sharpness through the GPU using GL shaders (Vulcan/Open GL based). In terms of sharpness intensity adjustment, the plan is to provide users with the ability to customize and regulate sharpness levels. We suggest setting a minimum and maximum strength range from 1 to 255, where a value of 0 signifies that the sharpness feature is disabled, indicated by a u8 data type. For now we have mapped the strength value 0.0 to 14.9375 to 0-239 but as the datatype is u8 user can give value upto 255 which is gets clamped to 239. We are also open to have alternative scales, such as 1-100 or 1-10, as long as a general consensus is reached within the community comprising OEMs and vendors. > -----Original Message----- > From: Simon Ser <contact@emersion.fr> > Sent: Thursday, February 15, 2024 2:03 PM > To: Garg, Nemesa <nemesa.garg@intel.com> > Cc: intel-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > How much of this is Intel-specific? Are there any hardware vendors with a similar > feature? How much is the "sharpness" knob tied to Intel hardware?
On Fri, 16 Feb 2024 04:28:41 +0000 "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > It is not intel specific and the goal is to have a generic API for > configuring Sharpness, accessible to various vendors. Intel currently > offers sharpness support through the Display Engine, while other > vendors seem to support Sharpness through the GPU using GL shaders > (Vulcan/Open GL based). Do you mean that all vendors use the exact same mathematical algorithm (with only differences in operation precision at most)? If yes, good. If not, then we need to know where exactly in the virtual KMS color pipeline the operation happens, whether this can be generic or not. Does this also work the same regardless of pixel formats, dynamic range, color gamut, transfer functions etc. on both plane input and connector output configurations? > In terms of sharpness intensity adjustment, the plan is to provide > users with the ability to customize and regulate sharpness levels. We > suggest setting a minimum and maximum strength range from 1 to 255, > where a value of 0 signifies that the sharpness feature is disabled, > indicated by a u8 data type. For now we have mapped the strength > value 0.0 to 14.9375 to 0-239 but as the datatype is u8 user can give > value upto 255 which is gets clamped to 239. Naturally you will need to document all that, so that all drivers and vendors do the exact same thing. I did not see any actual documentation in the patch series yet, e.g. a reference to a specific algorithm. As Ville pointed out, there was also no specification at which point of the virtual color pipeline this operation will apply. Before/after DEGAMMA/CTM/GAMMA/scaling in plane/blending/CRTC? Is the property being added to the list in https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#standard-crtc-properties or where-ever it belongs? Thanks, pq > We are also open to have alternative scales, such as 1-100 or 1-10, > as long as a general consensus is reached within the community > comprising OEMs and vendors. > > > -----Original Message----- > > From: Simon Ser <contact@emersion.fr> > > Sent: Thursday, February 15, 2024 2:03 PM > > To: Garg, Nemesa <nemesa.garg@intel.com> > > Cc: intel-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org > > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > > > How much of this is Intel-specific? Are there any hardware vendors > > with a similar feature? How much is the "sharpness" knob tied to > > Intel hardware?
This is generic as sharpness effect is applied post blending. Depending on the color gamut, pixel format and other inputs the image gets blended and once we get blended output it can be sharpened based on strength value provided by the user. On intel platform it is implemented through pipe scaler. Pipe scaler can be used as either scaler or sharpness filter. As mentioned earlier the client can provide any strength value between 0-255 or any other scale based on discussions. Perhaps userspace can have provide options like low-med-high sharpness or in percentage form or steps which is mapped to 0-255. I will add the documentation regarding property. > -----Original Message----- > From: dri-devel <dri-devel-bounces@lists.freedesktop.org> On Behalf Of Pekka > Paalanen > Sent: Friday, February 16, 2024 2:06 PM > To: Garg, Nemesa <nemesa.garg@intel.com> > Cc: Simon Ser <contact@emersion.fr>; intel-gfx@lists.freedesktop.org; dri- > devel@lists.freedesktop.org; G M, Adarsh <adarsh.g.m@intel.com> > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > On Fri, 16 Feb 2024 04:28:41 +0000 > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > It is not intel specific and the goal is to have a generic API for > > configuring Sharpness, accessible to various vendors. Intel currently > > offers sharpness support through the Display Engine, while other > > vendors seem to support Sharpness through the GPU using GL shaders > > (Vulcan/Open GL based). > > Do you mean that all vendors use the exact same mathematical algorithm (with > only differences in operation precision at most)? > > If yes, good. > > If not, then we need to know where exactly in the virtual KMS color pipeline the > operation happens, whether this can be generic or not. > > Does this also work the same regardless of pixel formats, dynamic range, color > gamut, transfer functions etc. on both plane input and connector output > configurations? > > > In terms of sharpness intensity adjustment, the plan is to provide > > users with the ability to customize and regulate sharpness levels. We > > suggest setting a minimum and maximum strength range from 1 to 255, > > where a value of 0 signifies that the sharpness feature is disabled, > > indicated by a u8 data type. For now we have mapped the strength value > > 0.0 to 14.9375 to 0-239 but as the datatype is u8 user can give value > > upto 255 which is gets clamped to 239. > > Naturally you will need to document all that, so that all drivers and vendors do the > exact same thing. > > I did not see any actual documentation in the patch series yet, e.g. a reference to > a specific algorithm. > > As Ville pointed out, there was also no specification at which point of the virtual > color pipeline this operation will apply. Before/after > DEGAMMA/CTM/GAMMA/scaling in plane/blending/CRTC? > > Is the property being added to the list in > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#standard-crtc- > properties > or where-ever it belongs? > > > Thanks, > pq > > > We are also open to have alternative scales, such as 1-100 or 1-10, as > > long as a general consensus is reached within the community comprising > > OEMs and vendors. > > > > > -----Original Message----- > > > From: Simon Ser <contact@emersion.fr> > > > Sent: Thursday, February 15, 2024 2:03 PM > > > To: Garg, Nemesa <nemesa.garg@intel.com> > > > Cc: intel-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org > > > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > > > > > How much of this is Intel-specific? Are there any hardware vendors > > > with a similar feature? How much is the "sharpness" knob tied to > > > Intel hardware?
On Monday, March 4th, 2024 at 15:04, Garg, Nemesa <nemesa.garg@intel.com> wrote: > This is generic as sharpness effect is applied post blending. Depending > on the color gamut, pixel format and other inputs the image gets > blended and once we get blended output it can be sharpened based on > strength value provided by the user. It would really help if you could provide the exact mathematical formula applied by this KMS property.
This KMS property is not implementing any formula and the values that are being used are based on empirical analysis and certain experiments done on the hardware. These values are fixed and is not expected to change and this can change from vendor to vendor. The client can choose any sharpness value on the scale and on the basis of it the sharpness will be set. The sharpness effect can be changed from content to content and from display to display so user needs to adjust the optimum intensity value so as to get good experience on the screen. > -----Original Message----- > From: dri-devel <dri-devel-bounces@lists.freedesktop.org> On Behalf Of Simon > Ser > Sent: Monday, March 4, 2024 7:46 PM > To: Garg, Nemesa <nemesa.garg@intel.com> > Cc: Pekka Paalanen <pekka.paalanen@haloniitty.fi>; intel- > gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org; G M, Adarsh > <adarsh.g.m@intel.com> > Subject: RE: [RFC 0/5] Introduce drm sharpening property > > On Monday, March 4th, 2024 at 15:04, Garg, Nemesa <nemesa.garg@intel.com> > wrote: > > > This is generic as sharpness effect is applied post blending. > > Depending on the color gamut, pixel format and other inputs the image > > gets blended and once we get blended output it can be sharpened based > > on strength value provided by the user. > > It would really help if you could provide the exact mathematical formula applied > by this KMS property.
On Tue, 12 Mar 2024 08:30:34 +0000 "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > This KMS property is not implementing any formula Sure it is. Maybe Intel just does not want to tell what the algorithm is, or maybe it's even patented. > and the values > that are being used are based on empirical analysis and certain > experiments done on the hardware. These values are fixed and is not > expected to change and this can change from vendor to vendor. The > client can choose any sharpness value on the scale and on the basis > of it the sharpness will be set. The sharpness effect can be changed > from content to content and from display to display so user needs to > adjust the optimum intensity value so as to get good experience on > the screen. > IOW, it's an opaque box operation, and there is no way to reproduce its results without the specific Intel hardware. Definitely no way to reproduce its results in free open source software alone. Such opaque box operations can only occur after KMS blending, at the CRTC or later stage. They cannot appear before blending, not in the new KMS color pipeline design at least. The reason is that the modern way to use KMS planes is opportunistic composition off-loading. Opportunistic means that userspace decides from time to time whether it composes the final picture using KMS or some other rendering method (usually GPU and shaders). Since userspace will arbitrarily switch between KMS and render composition, both must result in the exact same image, or end users will observe unwanted flicker. Such opaque box operations are fine after blending, because there they can be configured once and remain on forever. No switching, no flicker. Where does "sharpeness" operation occur in the Intel color processing chain? Is it before or after blending? What kind of transfer characteristics does it expect from the image, and can those be realized with KMS CRTC properties if KMS is configured such that the blending happens using some other characteristics (e.g. blending in optical space)? What about SDR vs. HDR imagery? Thanks, pq > > -----Original Message----- > > From: dri-devel <dri-devel-bounces@lists.freedesktop.org> On Behalf Of Simon > > Ser > > Sent: Monday, March 4, 2024 7:46 PM > > To: Garg, Nemesa <nemesa.garg@intel.com> > > Cc: Pekka Paalanen <pekka.paalanen@haloniitty.fi>; intel- > > gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org; G M, Adarsh > > <adarsh.g.m@intel.com> > > Subject: RE: [RFC 0/5] Introduce drm sharpening property > > > > On Monday, March 4th, 2024 at 15:04, Garg, Nemesa <nemesa.garg@intel.com> > > wrote: > > > > > This is generic as sharpness effect is applied post blending. > > > Depending on the color gamut, pixel format and other inputs the image > > > gets blended and once we get blended output it can be sharpened based > > > on strength value provided by the user. > > > > It would really help if you could provide the exact mathematical formula applied > > by this KMS property.
On Tue, 12 Mar 2024 16:26:00 +0200 Pekka Paalanen <pekka.paalanen@haloniitty.fi> wrote: > On Tue, 12 Mar 2024 08:30:34 +0000 > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > This KMS property is not implementing any formula > > Sure it is. Maybe Intel just does not want to tell what the algorithm > is, or maybe it's even patented. > > > and the values > > that are being used are based on empirical analysis and certain > > experiments done on the hardware. These values are fixed and is not > > expected to change and this can change from vendor to vendor. The > > client can choose any sharpness value on the scale and on the basis > > of it the sharpness will be set. The sharpness effect can be changed > > from content to content and from display to display so user needs to > > adjust the optimum intensity value so as to get good experience on > > the screen. > > > > IOW, it's an opaque box operation, and there is no way to reproduce its > results without the specific Intel hardware. Definitely no way to > reproduce its results in free open source software alone. > > Such opaque box operations can only occur after KMS blending, at the > CRTC or later stage. They cannot appear before blending, not in the new > KMS color pipeline design at least. The reason is that the modern way > to use KMS planes is opportunistic composition off-loading. > Opportunistic means that userspace decides from time to time whether it > composes the final picture using KMS or some other rendering method > (usually GPU and shaders). Since userspace will arbitrarily switch > between KMS and render composition, both must result in the exact same > image, or end users will observe unwanted flicker. > > Such opaque box operations are fine after blending, because there they > can be configured once and remain on forever. No switching, no flicker. If you want to see how sharpness property would apply in Wayland design, it would be in step 5 "Adjust (settings UI)" of https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/color-management-model.md#compositor-color-management-model To relate that diagram to KMS color processing, you can identify step 3 "Compose" as the KMS blending step. Everything before step 3 happens in KMS plane color processing, and steps 4-5 happen in KMS CRTC color processing. Sharpening would essentially be a "compositor color effect", it just happens to be implementable only by specific Intel hardware. If a color effect is dynamic or content-dependant, it will preclude colorimetric monitor calibration. Thanks, pq > Where does "sharpeness" operation occur in the Intel color processing > chain? Is it before or after blending? > > What kind of transfer characteristics does it expect from the image, > and can those be realized with KMS CRTC properties if KMS is configured > such that the blending happens using some other characteristics (e.g. > blending in optical space)? > > What about SDR vs. HDR imagery? > > > Thanks, > pq > > > > -----Original Message----- > > > From: dri-devel <dri-devel-bounces@lists.freedesktop.org> On Behalf Of Simon > > > Ser > > > Sent: Monday, March 4, 2024 7:46 PM > > > To: Garg, Nemesa <nemesa.garg@intel.com> > > > Cc: Pekka Paalanen <pekka.paalanen@haloniitty.fi>; intel- > > > gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org; G M, Adarsh > > > <adarsh.g.m@intel.com> > > > Subject: RE: [RFC 0/5] Introduce drm sharpening property > > > > > > On Monday, March 4th, 2024 at 15:04, Garg, Nemesa <nemesa.garg@intel.com> > > > wrote: > > > > > > > This is generic as sharpness effect is applied post blending. > > > > Depending on the color gamut, pixel format and other inputs the image > > > > gets blended and once we get blended output it can be sharpened based > > > > on strength value provided by the user. > > > > > > It would really help if you could provide the exact mathematical formula applied > > > by this KMS property. >
> -----Original Message----- > From: Pekka Paalanen <pekka.paalanen@haloniitty.fi> > Sent: Wednesday, March 13, 2024 3:07 PM > To: Garg, Nemesa <nemesa.garg@intel.com> > Cc: Simon Ser <contact@emersion.fr>; intel-gfx@lists.freedesktop.org; dri- > devel@lists.freedesktop.org; G M, Adarsh <adarsh.g.m@intel.com> > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > On Tue, 12 Mar 2024 16:26:00 +0200 > Pekka Paalanen <pekka.paalanen@haloniitty.fi> wrote: > > > On Tue, 12 Mar 2024 08:30:34 +0000 > > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > > > This KMS property is not implementing any formula > > > > Sure it is. Maybe Intel just does not want to tell what the algorithm > > is, or maybe it's even patented. > > > > > and the values > > > that are being used are based on empirical analysis and certain > > > experiments done on the hardware. These values are fixed and is not > > > expected to change and this can change from vendor to vendor. The > > > client can choose any sharpness value on the scale and on the basis > > > of it the sharpness will be set. The sharpness effect can be changed > > > from content to content and from display to display so user needs to > > > adjust the optimum intensity value so as to get good experience on > > > the screen. > > > > > > > IOW, it's an opaque box operation, and there is no way to reproduce > > its results without the specific Intel hardware. Definitely no way to > > reproduce its results in free open source software alone. > > > > Such opaque box operations can only occur after KMS blending, at the > > CRTC or later stage. They cannot appear before blending, not in the > > new KMS color pipeline design at least. The reason is that the modern > > way to use KMS planes is opportunistic composition off-loading. > > Opportunistic means that userspace decides from time to time whether > > it composes the final picture using KMS or some other rendering method > > (usually GPU and shaders). Since userspace will arbitrarily switch > > between KMS and render composition, both must result in the exact same > > image, or end users will observe unwanted flicker. > > > > Such opaque box operations are fine after blending, because there they > > can be configured once and remain on forever. No switching, no flicker. > > If you want to see how sharpness property would apply in Wayland design, it > would be in step 5 "Adjust (settings UI)" of > https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/color- > management-model.md#compositor-color-management-model > > To relate that diagram to KMS color processing, you can identify step 3 "Compose" > as the KMS blending step. Everything before step 3 happens in KMS plane color > processing, and steps 4-5 happen in KMS CRTC color processing. > > Sharpening would essentially be a "compositor color effect", it just happens to be > implementable only by specific Intel hardware. > > If a color effect is dynamic or content-dependant, it will preclude colorimetric > monitor calibration. > > > Thanks, > pq > > > > Where does "sharpeness" operation occur in the Intel color processing > > chain? Is it before or after blending? > > Thank you for detail explanation and link. Sharpness operation occur post blending in CRTC ie on the final composed output after blending . Yes Pekka you are right as per the diagram it is done at step 5 "Adjust (settings UI)"). I will also document this thing along with documentation change. > > What kind of transfer characteristics does it expect from the image, > > and can those be realized with KMS CRTC properties if KMS is > > configured such that the blending happens using some other characteristics > (e.g. > > blending in optical space)? > > The filter values are not dependent/calculated on the inputs of image but depending on the blending space and other inputs the blended output gets changed and the sharpness is applied post blending so according to the content user needs to adjust the strength value to get the better visual effect. So tuning of sharpness strength may be needed by user based on the input contents and blending policy to get the desired experience. > > What about SDR vs. HDR imagery? > > The interface can be used for both HDR and SDR. The effect is more prominent for SDR use cases. For HDR filter values and tap value may change. Thanks, Nemesa > > > > Thanks, > > pq > > > > > > -----Original Message----- > > > > From: dri-devel <dri-devel-bounces@lists.freedesktop.org> On > > > > Behalf Of Simon Ser > > > > Sent: Monday, March 4, 2024 7:46 PM > > > > To: Garg, Nemesa <nemesa.garg@intel.com> > > > > Cc: Pekka Paalanen <pekka.paalanen@haloniitty.fi>; intel- > > > > gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org; G M, > > > > Adarsh <adarsh.g.m@intel.com> > > > > Subject: RE: [RFC 0/5] Introduce drm sharpening property > > > > > > > > On Monday, March 4th, 2024 at 15:04, Garg, Nemesa > > > > <nemesa.garg@intel.com> > > > > wrote: > > > > > > > > > This is generic as sharpness effect is applied post blending. > > > > > Depending on the color gamut, pixel format and other inputs the > > > > > image gets blended and once we get blended output it can be sharpened > based > > > > > on strength value provided by the user. > > > > > > > > It would really help if you could provide the exact mathematical formula > applied > > > > by this KMS property. > >
On Wed, 27 Mar 2024 07:11:48 +0000 "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > -----Original Message----- > > From: Pekka Paalanen <pekka.paalanen@haloniitty.fi> > > Sent: Wednesday, March 13, 2024 3:07 PM > > To: Garg, Nemesa <nemesa.garg@intel.com> > > Cc: Simon Ser <contact@emersion.fr>; intel-gfx@lists.freedesktop.org; dri- > > devel@lists.freedesktop.org; G M, Adarsh <adarsh.g.m@intel.com> > > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > > > On Tue, 12 Mar 2024 16:26:00 +0200 > > Pekka Paalanen <pekka.paalanen@haloniitty.fi> wrote: > > > > > On Tue, 12 Mar 2024 08:30:34 +0000 > > > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > > > > > This KMS property is not implementing any formula > > > > > > Sure it is. Maybe Intel just does not want to tell what the algorithm > > > is, or maybe it's even patented. > > > > > > > and the values > > > > that are being used are based on empirical analysis and certain > > > > experiments done on the hardware. These values are fixed and is not > > > > expected to change and this can change from vendor to vendor. The > > > > client can choose any sharpness value on the scale and on the basis > > > > of it the sharpness will be set. The sharpness effect can be changed > > > > from content to content and from display to display so user needs to > > > > adjust the optimum intensity value so as to get good experience on > > > > the screen. > > > > > > > > > > IOW, it's an opaque box operation, and there is no way to reproduce > > > its results without the specific Intel hardware. Definitely no way to > > > reproduce its results in free open source software alone. > > > > > > Such opaque box operations can only occur after KMS blending, at the > > > CRTC or later stage. They cannot appear before blending, not in the > > > new KMS color pipeline design at least. The reason is that the modern > > > way to use KMS planes is opportunistic composition off-loading. > > > Opportunistic means that userspace decides from time to time whether > > > it composes the final picture using KMS or some other rendering method > > > (usually GPU and shaders). Since userspace will arbitrarily switch > > > between KMS and render composition, both must result in the exact same > > > image, or end users will observe unwanted flicker. > > > > > > Such opaque box operations are fine after blending, because there they > > > can be configured once and remain on forever. No switching, no flicker. > > > > If you want to see how sharpness property would apply in Wayland design, it > > would be in step 5 "Adjust (settings UI)" of > > https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/color- > > management-model.md#compositor-color-management-model > > > > To relate that diagram to KMS color processing, you can identify step 3 "Compose" > > as the KMS blending step. Everything before step 3 happens in KMS plane color > > processing, and steps 4-5 happen in KMS CRTC color processing. > > > > Sharpening would essentially be a "compositor color effect", it just happens to be > > implementable only by specific Intel hardware. > > > > If a color effect is dynamic or content-dependant, it will preclude colorimetric > > monitor calibration. > > > > > > Thanks, > > pq > > > > > > > Where does "sharpeness" operation occur in the Intel color processing > > > chain? Is it before or after blending? > > > > Thank you for detail explanation and link. > Sharpness operation occur post blending in CRTC ie on the final > composed output after blending . Yes Pekka you are right as per the > diagram it is done at step 5 "Adjust (settings UI)"). I will also document this thing > along with documentation change. > > > > What kind of transfer characteristics does it expect from the image, > > > and can those be realized with KMS CRTC properties if KMS is > > > configured such that the blending happens using some other characteristics > > (e.g. > > > blending in optical space)? > > > > The filter values are not dependent/calculated on the inputs of > image but depending on the blending space and other inputs the > blended output gets changed and the sharpness is applied post > blending so according to the content user needs to adjust the > strength value to get the better visual effect. So tuning of sharpness strength > may be needed by user based on the input contents and blending policy > to get the desired experience. > > > > What about SDR vs. HDR imagery? > > > > The interface can be used for both HDR and SDR. The effect is more prominent for SDR use cases. > For HDR filter values and tap value may change. Who will be providing these values? The kernel driver cannot know if it is dealing with SDR or HDR or which transfer function is in effect at that point of the post-blending color pipeline. If the UAPI is one "strength" value, then how can it work? Maybe the UAPI needs more controls, if not providing all "filter and tap" values directly. Maybe all the filter and tap values should be provided by userspace? Thanks, pq
On Wed, 27 Mar 2024 13:29:16 +0200 Pekka Paalanen <pekka.paalanen@haloniitty.fi> wrote: > On Wed, 27 Mar 2024 07:11:48 +0000 > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > > -----Original Message----- > > > From: Pekka Paalanen <pekka.paalanen@haloniitty.fi> > > > Sent: Wednesday, March 13, 2024 3:07 PM > > > To: Garg, Nemesa <nemesa.garg@intel.com> > > > Cc: Simon Ser <contact@emersion.fr>; intel-gfx@lists.freedesktop.org; dri- > > > devel@lists.freedesktop.org; G M, Adarsh <adarsh.g.m@intel.com> > > > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > > > > > On Tue, 12 Mar 2024 16:26:00 +0200 > > > Pekka Paalanen <pekka.paalanen@haloniitty.fi> wrote: > > > > > > > On Tue, 12 Mar 2024 08:30:34 +0000 > > > > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > > > > > > > This KMS property is not implementing any formula > > > > > > > > Sure it is. Maybe Intel just does not want to tell what the algorithm > > > > is, or maybe it's even patented. > > > > > > > > > and the values > > > > > that are being used are based on empirical analysis and certain > > > > > experiments done on the hardware. These values are fixed and is not > > > > > expected to change and this can change from vendor to vendor. The > > > > > client can choose any sharpness value on the scale and on the basis > > > > > of it the sharpness will be set. The sharpness effect can be changed > > > > > from content to content and from display to display so user needs to > > > > > adjust the optimum intensity value so as to get good experience on > > > > > the screen. > > > > > > > > > > > > > IOW, it's an opaque box operation, and there is no way to reproduce > > > > its results without the specific Intel hardware. Definitely no way to > > > > reproduce its results in free open source software alone. > > > > > > > > Such opaque box operations can only occur after KMS blending, at the > > > > CRTC or later stage. They cannot appear before blending, not in the > > > > new KMS color pipeline design at least. The reason is that the modern > > > > way to use KMS planes is opportunistic composition off-loading. > > > > Opportunistic means that userspace decides from time to time whether > > > > it composes the final picture using KMS or some other rendering method > > > > (usually GPU and shaders). Since userspace will arbitrarily switch > > > > between KMS and render composition, both must result in the exact same > > > > image, or end users will observe unwanted flicker. > > > > > > > > Such opaque box operations are fine after blending, because there they > > > > can be configured once and remain on forever. No switching, no flicker. > > > > > > If you want to see how sharpness property would apply in Wayland design, it > > > would be in step 5 "Adjust (settings UI)" of > > > https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/color- > > > management-model.md#compositor-color-management-model > > > > > > To relate that diagram to KMS color processing, you can identify step 3 "Compose" > > > as the KMS blending step. Everything before step 3 happens in KMS plane color > > > processing, and steps 4-5 happen in KMS CRTC color processing. > > > > > > Sharpening would essentially be a "compositor color effect", it just happens to be > > > implementable only by specific Intel hardware. > > > > > > If a color effect is dynamic or content-dependant, it will preclude colorimetric > > > monitor calibration. > > > > > > > > > Thanks, > > > pq > > > > > > > > > > Where does "sharpeness" operation occur in the Intel color processing > > > > chain? Is it before or after blending? > > > > > > Thank you for detail explanation and link. > > Sharpness operation occur post blending in CRTC ie on the final > > composed output after blending . Yes Pekka you are right as per the > > diagram it is done at step 5 "Adjust (settings UI)"). I will also document this thing > > along with documentation change. > > > > > > What kind of transfer characteristics does it expect from the image, > > > > and can those be realized with KMS CRTC properties if KMS is > > > > configured such that the blending happens using some other characteristics > > > (e.g. > > > > blending in optical space)? > > > > > > The filter values are not dependent/calculated on the inputs of > > image but depending on the blending space and other inputs the > > blended output gets changed and the sharpness is applied post > > blending so according to the content user needs to adjust the > > strength value to get the better visual effect. So tuning of sharpness strength > > may be needed by user based on the input contents and blending policy > > to get the desired experience. > > > > > > What about SDR vs. HDR imagery? > > > > > > The interface can be used for both HDR and SDR. The effect is more prominent for SDR use cases. > > For HDR filter values and tap value may change. > > Who will be providing these values? > > The kernel driver cannot know if it is dealing with SDR or HDR or which > transfer function is in effect at that point of the post-blending color > pipeline. > > If the UAPI is one "strength" value, then how can it work? > > Maybe the UAPI needs more controls, if not providing all "filter and > tap" values directly. Maybe all the filter and tap values should be > provided by userspace? Actually, is the hardware just doing a convolution with a filter defined by the driver? Convolution algorithm (it is a formula!) is pretty standard stuff I believe. If the hardware is actually doing convolution, then the driver really should be exposing the convolution operation. Then people can choose to use it for sharpening with the Intel developed kernels, or for custom effects with custom kernels. Everyone would win. Convolution is also something that other hardware vendors could implement. A convolution filter would fit very well in the new KMS color pipeline design for post-compositing operations, too. Is the sharpening element doing something similar to the unsharp masking? I suppose users might want different strength based on what kind of content is the majority on the screen. That makes it something that a Wayland compositor would adjust automatically based on Wayland content type (similar to HDMI content type), for example. Thanks, pq
> -----Original Message----- > From: Pekka Paalanen <pekka.paalanen@haloniitty.fi> > Sent: Thursday, March 28, 2024 3:35 PM > To: Garg, Nemesa <nemesa.garg@intel.com> > Cc: Simon Ser <contact@emersion.fr>; intel-gfx@lists.freedesktop.org; dri- > devel@lists.freedesktop.org; G M, Adarsh <adarsh.g.m@intel.com> > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > On Wed, 27 Mar 2024 13:29:16 +0200 > Pekka Paalanen <pekka.paalanen@haloniitty.fi> wrote: > > > On Wed, 27 Mar 2024 07:11:48 +0000 > > "Garg, Nemesa" <nemesa.garg@intel.com> wrote: > > > > > > -----Original Message----- > > > > From: Pekka Paalanen <pekka.paalanen@haloniitty.fi> > > > > Sent: Wednesday, March 13, 2024 3:07 PM > > > > To: Garg, Nemesa <nemesa.garg@intel.com> > > > > Cc: Simon Ser <contact@emersion.fr>; > > > > intel-gfx@lists.freedesktop.org; dri- devel@lists.freedesktop.org; > > > > G M, Adarsh <adarsh.g.m@intel.com> > > > > Subject: Re: [RFC 0/5] Introduce drm sharpening property > > > > > > > > On Tue, 12 Mar 2024 16:26:00 +0200 Pekka Paalanen > > > > <pekka.paalanen@haloniitty.fi> wrote: > > > > > > > > > On Tue, 12 Mar 2024 08:30:34 +0000 "Garg, Nemesa" > > > > > <nemesa.garg@intel.com> wrote: > > > > > > > > > > > This KMS property is not implementing any formula > > > > > > > > > > Sure it is. Maybe Intel just does not want to tell what the > > > > > algorithm is, or maybe it's even patented. > > > > > > > > > > > and the values > > > > > > that are being used are based on empirical analysis and > > > > > > certain experiments done on the hardware. These values are > > > > > > fixed and is not expected to change and this can change from > > > > > > vendor to vendor. The client can choose any sharpness value on > > > > > > the scale and on the basis of it the sharpness will be set. > > > > > > The sharpness effect can be changed from content to content > > > > > > and from display to display so user needs to adjust the > > > > > > optimum intensity value so as to get good experience on the screen. > > > > > > > > > > > > > > > > IOW, it's an opaque box operation, and there is no way to > > > > > reproduce its results without the specific Intel hardware. > > > > > Definitely no way to reproduce its results in free open source software > alone. > > > > > > > > > > Such opaque box operations can only occur after KMS blending, at > > > > > the CRTC or later stage. They cannot appear before blending, not > > > > > in the new KMS color pipeline design at least. The reason is > > > > > that the modern way to use KMS planes is opportunistic composition off- > loading. > > > > > Opportunistic means that userspace decides from time to time > > > > > whether it composes the final picture using KMS or some other > > > > > rendering method (usually GPU and shaders). Since userspace will > > > > > arbitrarily switch between KMS and render composition, both must > > > > > result in the exact same image, or end users will observe unwanted flicker. > > > > > > > > > > Such opaque box operations are fine after blending, because there they > > > > > can be configured once and remain on forever. No switching, no flicker. > > > > > > > > If you want to see how sharpness property would apply in Wayland > > > > design, it would be in step 5 "Adjust (settings UI)" of > > > > https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/co > > > > lor- management-model.md#compositor-color-management-model > > > > > > > > To relate that diagram to KMS color processing, you can identify step 3 > "Compose" > > > > as the KMS blending step. Everything before step 3 happens in KMS > > > > plane color processing, and steps 4-5 happen in KMS CRTC color processing. > > > > > > > > Sharpening would essentially be a "compositor color effect", it > > > > just happens to be implementable only by specific Intel hardware. > > > > > > > > If a color effect is dynamic or content-dependant, it will > > > > preclude colorimetric monitor calibration. > > > > > > > > > > > > Thanks, > > > > pq > > > > > > > > > > > > > Where does "sharpeness" operation occur in the Intel color > > > > > processing chain? Is it before or after blending? > > > > > > > > Thank you for detail explanation and link. > > > Sharpness operation occur post blending in CRTC ie on the final > > > composed output after blending . Yes Pekka you are right as per the > > > diagram it is done at step 5 "Adjust (settings UI)"). I will also > > > document this thing along with documentation change. > > > > > > > > What kind of transfer characteristics does it expect from the > > > > > image, and can those be realized with KMS CRTC properties if KMS is > > > > > configured such that the blending happens using some other > characteristics > > > > (e.g. > > > > > blending in optical space)? > > > > > > > > The filter values are not dependent/calculated on the inputs of > > > image but depending on the blending space and other inputs the > > > blended output gets changed and the sharpness is applied post > > > blending so according to the content user needs to adjust the > > > strength value to get the better visual effect. So tuning of > > > sharpness strength may be needed by user based on the input > > > contents and blending policy to get the desired experience. > > > > > > > > What about SDR vs. HDR imagery? > > > > > > > > The interface can be used for both HDR and SDR. The effect is more > prominent for SDR use cases. > > > For HDR filter values and tap value may change. > > > > Who will be providing these values? > > > > The kernel driver cannot know if it is dealing with SDR or HDR or > > which transfer function is in effect at that point of the > > post-blending color pipeline. > > > > If the UAPI is one "strength" value, then how can it work? > > > > Maybe the UAPI needs more controls, if not providing all "filter and > > tap" values directly. Maybe all the filter and tap values should be > > provided by userspace? > > Actually, is the hardware just doing a convolution with a filter defined by the > driver? > > Convolution algorithm (it is a formula!) is pretty standard stuff I believe. If the > hardware is actually doing convolution, then the driver really should be exposing > the convolution operation. Then people can choose to use it for sharpening with > the Intel developed kernels, or for custom effects with custom kernels. Everyone > would win. Convolution is also something that other hardware vendors could > implement. > > A convolution filter would fit very well in the new KMS color pipeline design for > post-compositing operations, too. > > Is the sharpening element doing something similar to the unsharp masking? > > I suppose users might want different strength based on what kind of content is > the majority on the screen. That makes it something that a Wayland compositor > would adjust automatically based on Wayland content type (similar to HDMI > content type), for example. > > > Thanks, > pq Hi Pekka, Thank you for the feedback and suggestions. Based on the discussions in Hackfest(2024 Linux Display Next) and community feedback we were exploring various solutions from the user-space side and come up with a looking glass solution. Below is the link for mutter MR: https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/3665 There is a sort of convolution along with other optimizations which are within the hardware and the user-space should just control the extent of sharpness. This value will be received in the driver and appropriate programming should be done based on respective hardware design. Also as agreed it should be fine if sharpness is applied post blending or if we drive a single plane use-case. In cases when hardware doesn't support sharpness due to various reasons we can implement the generic shader using some open source algorithm. This may not match pixel by pixel with hardware but should be ok if hardware applies post blending. Thanks and Regards, Nemesa