[2/2] x86/pat: add functions to query specific cache mode availability

Commit Message

Jürgen Groß May 3, 2022, 1:22 p.m. UTC

Some drivers are using pat_enabled() in order to test availability of
special caching modes (WC and UC-). This will lead to false negatives
in case the system was booted e.g. with the "nopat" variant and the
BIOS did setup the PAT MSR supporting the queried mode, or if the
system is running as a Xen PV guest.

Add test functions for those caching modes instead and use them at the
appropriate places.

For symmetry reasons export the already existing x86_has_pat_wp() for
modules, too.

Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with pat_enabled()")
Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
Signed-off-by: Juergen Gross <[e-mail]>
---
 arch/x86/include/asm/memtype.h           |  2 ++
 arch/x86/include/asm/pci.h               |  2 +-
 arch/x86/mm/init.c                       | 25 +++++++++++++++++++++---
 drivers/gpu/drm/i915/gem/i915_gem_mman.c |  8 ++++----
 4 files changed, 29 insertions(+), 8 deletions(-)

Comments

Jan Beulich May 4, 2022, 8:31 a.m. UTC | #1

On 03.05.2022 15:22, Juergen Gross wrote:
> Some drivers are using pat_enabled() in order to test availability of
> special caching modes (WC and UC-). This will lead to false negatives
> in case the system was booted e.g. with the "nopat" variant and the
> BIOS did setup the PAT MSR supporting the queried mode, or if the
> system is running as a Xen PV guest.

While, as per my earlier patch, I agree with the Xen PV case, I'm not
convinced "nopat" is supposed to honor firmware-provided settings. In
fact in my patch I did arrange for "nopat" to also take effect under
Xen PV.

> Add test functions for those caching modes instead and use them at the
> appropriate places.
> 
> For symmetry reasons export the already existing x86_has_pat_wp() for
> modules, too.
> 
> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with pat_enabled()")
> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
> Signed-off-by: Juergen Gross <[e-mail]>

I think this wants a Reported-by as well.

> --- a/arch/x86/include/asm/pci.h
> +++ b/arch/x86/include/asm/pci.h
> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev, int pin, int irq);
>  
>  
>  #define HAVE_PCI_MMAP
> -#define arch_can_pci_mmap_wc()	pat_enabled()
> +#define arch_can_pci_mmap_wc()	x86_has_pat_wc()

Besides this and ...

> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
>  	if (args->flags & ~(I915_MMAP_WC))
>  		return -EINVAL;
>  
> -	if (args->flags & I915_MMAP_WC && !pat_enabled())
> +	if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>  		return -ENODEV;
>  
>  	obj = i915_gem_object_lookup(file, args->handle);
> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>  
>  	if (HAS_LMEM(to_i915(dev)))
>  		mmap_type = I915_MMAP_TYPE_FIXED;
> -	else if (pat_enabled())
> +	else if (x86_has_pat_wc())
>  		mmap_type = I915_MMAP_TYPE_WC;
>  	else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>  		return -ENODEV;
> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>  		break;
>  
>  	case I915_MMAP_OFFSET_WC:
> -		if (!pat_enabled())
> +		if (!x86_has_pat_wc())
>  			return -ENODEV;
>  		type = I915_MMAP_TYPE_WC;
>  		break;
> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>  		break;
>  
>  	case I915_MMAP_OFFSET_UC:
> -		if (!pat_enabled())
> +		if (!x86_has_pat_uc_minus())
>  			return -ENODEV;
>  		type = I915_MMAP_TYPE_UC;
>  		break;

... these uses there are several more. You say nothing on why those want
leaving unaltered. When preparing my earlier patch I did inspect them
and came to the conclusion that these all would also better observe the
adjusted behavior (or else I couldn't have left pat_enabled() as the only
predicate). In fact, as said in the description of my earlier patch, in
my debugging I did find the use in i915_gem_object_pin_map() to be the
problematic one, which you leave alone.

Jan

Jürgen Groß May 4, 2022, 9:14 a.m. UTC | #2

On 04.05.22 10:31, Jan Beulich wrote:
> On 03.05.2022 15:22, Juergen Gross wrote:
>> Some drivers are using pat_enabled() in order to test availability of
>> special caching modes (WC and UC-). This will lead to false negatives
>> in case the system was booted e.g. with the "nopat" variant and the
>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>> system is running as a Xen PV guest.
> 
> While, as per my earlier patch, I agree with the Xen PV case, I'm not
> convinced "nopat" is supposed to honor firmware-provided settings. In
> fact in my patch I did arrange for "nopat" to also take effect under
> Xen PV.

Depends on what the wanted semantics for "nopat" are.

Right now "nopat" will result in the PAT MSR left unchanged and the
cache mode translation tables be initialized accordingly.

So does "nopat" mean that the PAT MSR shouldn't be changed, or that
PAGE_BIT_PAT will never be set?

>> Add test functions for those caching modes instead and use them at the
>> appropriate places.
>>
>> For symmetry reasons export the already existing x86_has_pat_wp() for
>> modules, too.
>>
>> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with pat_enabled()")
>> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
>> Signed-off-by: Juergen Gross <[e-mail]>
> 
> I think this wants a Reported-by as well.

Okay.

> 
>> --- a/arch/x86/include/asm/pci.h
>> +++ b/arch/x86/include/asm/pci.h
>> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev, int pin, int irq);
>>   
>>   
>>   #define HAVE_PCI_MMAP
>> -#define arch_can_pci_mmap_wc()	pat_enabled()
>> +#define arch_can_pci_mmap_wc()	x86_has_pat_wc()
> 
> Besides this and ...
> 
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
>>   	if (args->flags & ~(I915_MMAP_WC))
>>   		return -EINVAL;
>>   
>> -	if (args->flags & I915_MMAP_WC && !pat_enabled())
>> +	if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>>   		return -ENODEV;
>>   
>>   	obj = i915_gem_object_lookup(file, args->handle);
>> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>>   
>>   	if (HAS_LMEM(to_i915(dev)))
>>   		mmap_type = I915_MMAP_TYPE_FIXED;
>> -	else if (pat_enabled())
>> +	else if (x86_has_pat_wc())
>>   		mmap_type = I915_MMAP_TYPE_WC;
>>   	else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>>   		return -ENODEV;
>> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>>   		break;
>>   
>>   	case I915_MMAP_OFFSET_WC:
>> -		if (!pat_enabled())
>> +		if (!x86_has_pat_wc())
>>   			return -ENODEV;
>>   		type = I915_MMAP_TYPE_WC;
>>   		break;
>> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>>   		break;
>>   
>>   	case I915_MMAP_OFFSET_UC:
>> -		if (!pat_enabled())
>> +		if (!x86_has_pat_uc_minus())
>>   			return -ENODEV;
>>   		type = I915_MMAP_TYPE_UC;
>>   		break;
> 
> ... these uses there are several more. You say nothing on why those want
> leaving unaltered. When preparing my earlier patch I did inspect them
> and came to the conclusion that these all would also better observe the
> adjusted behavior (or else I couldn't have left pat_enabled() as the only
> predicate). In fact, as said in the description of my earlier patch, in
> my debugging I did find the use in i915_gem_object_pin_map() to be the
> problematic one, which you leave alone.

Oh, I missed that one, sorry.

I wanted to be rather defensive in my changes, but I agree at least the
case in arch_phys_wc_add() might want to be changed, too.

kvm_is_mmio_pfn() should not really matter at least for the Xen case.

With the other use cases in memtype.c I'm rather on the edge.

In case the x86 maintainers think those should be changed, too, I agree
that your approach might be the better one.


Juergen

Jan Beulich May 4, 2022, 9:51 a.m. UTC | #3

On 04.05.2022 11:14, Juergen Gross wrote:
> On 04.05.22 10:31, Jan Beulich wrote:
>> On 03.05.2022 15:22, Juergen Gross wrote:
>>> Some drivers are using pat_enabled() in order to test availability of
>>> special caching modes (WC and UC-). This will lead to false negatives
>>> in case the system was booted e.g. with the "nopat" variant and the
>>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>>> system is running as a Xen PV guest.
>>
>> While, as per my earlier patch, I agree with the Xen PV case, I'm not
>> convinced "nopat" is supposed to honor firmware-provided settings. In
>> fact in my patch I did arrange for "nopat" to also take effect under
>> Xen PV.
> 
> Depends on what the wanted semantics for "nopat" are.
> 
> Right now "nopat" will result in the PAT MSR left unchanged and the
> cache mode translation tables be initialized accordingly.
> 
> So does "nopat" mean that the PAT MSR shouldn't be changed, or that
> PAGE_BIT_PAT will never be set?

According to the documentation for the option ("Disable PAT (page
attribute table extension of pagetables) support") I'd say the latter.

Jan

Chuck Zmudzinski May 20, 2022, 2:15 a.m. UTC | #4

On 5/3/22 9:22 AM, Juergen Gross wrote:
> Some drivers are using pat_enabled() in order to test availability of
> special caching modes (WC and UC-). This will lead to false negatives
> in case the system was booted e.g. with the "nopat" variant and the
> BIOS did setup the PAT MSR supporting the queried mode, or if the
> system is running as a Xen PV guest.
Hello,

I am also getting a false positive in a Xen Dom0 from
pat_enabled() where bdd8b6c98239 patched the file

drivers/gpu/drm/i915/gem/i915_gem_pages.c

I think this patch also needs to touch that file to
fix the issue I am seeing.

Ever since bdd8b6c98239 was committed, I get the
following in the logs when running as a Dom0 on my
Haswell processor, including with the untainted
official Debian build of 5.17.6:

May 15 06:31:59 debian kernel: [    3.721146] i915 0000:00:02.0: 
[drm:add_taint_for_CI [i915]] CI tainted:0x9 by intel_gt_init+0xb6/0x2e0 
[i915]

This causes the system to hang with the backlight on.
The only recovery is by hitting the reset button and
rebooting Linux Dom0 on Xen with Linux version 5.16
or earlier, or by rebooting Linux version 5.17 without
Xen.

I was able to fix it with a kernel that fixes the
false negative I was getting in
drivers/gpu/drm/i915/gem/i915_gem_pages.c

Can the patch also touch that file, replacing
pat_enabled() with x86_has_pat_wc() in the
place where bdd8b6c98239 patched that file?

Thanks,

Chuck Zmudzinski

>
> Add test functions for those caching modes instead and use them at the
> appropriate places.
>
> For symmetry reasons export the already existing x86_has_pat_wp() for
> modules, too.
>
> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with pat_enabled()")
> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
> Signed-off-by: Juergen Gross <[e-mail]>
> ---
>   arch/x86/include/asm/memtype.h           |  2 ++
>   arch/x86/include/asm/pci.h               |  2 +-
>   arch/x86/mm/init.c                       | 25 +++++++++++++++++++++---
>   drivers/gpu/drm/i915/gem/i915_gem_mman.c |  8 ++++----
>   4 files changed, 29 insertions(+), 8 deletions(-)
>
> diff --git a/arch/x86/include/asm/memtype.h b/arch/x86/include/asm/memtype.h
> index 9ca760e430b9..d00e0be854d4 100644
> --- a/arch/x86/include/asm/memtype.h
> +++ b/arch/x86/include/asm/memtype.h
> @@ -25,6 +25,8 @@ extern void memtype_free_io(resource_size_t start, resource_size_t end);
>   extern bool pat_pfn_immune_to_uc_mtrr(unsigned long pfn);
>   
>   bool x86_has_pat_wp(void);
> +bool x86_has_pat_wc(void);
> +bool x86_has_pat_uc_minus(void);
>   enum page_cache_mode pgprot2cachemode(pgprot_t pgprot);
>   
>   #endif /* _ASM_X86_MEMTYPE_H */
> diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
> index f3fd5928bcbb..a5742268dec1 100644
> --- a/arch/x86/include/asm/pci.h
> +++ b/arch/x86/include/asm/pci.h
> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev, int pin, int irq);
>   
>   
>   #define HAVE_PCI_MMAP
> -#define arch_can_pci_mmap_wc()	pat_enabled()
> +#define arch_can_pci_mmap_wc()	x86_has_pat_wc()
>   #define ARCH_GENERIC_PCI_MMAP_RESOURCE
>   
>   #ifdef CONFIG_PCI
> diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
> index 71e182ebced3..b6431f714dc2 100644
> --- a/arch/x86/mm/init.c
> +++ b/arch/x86/mm/init.c
> @@ -77,12 +77,31 @@ static uint8_t __pte2cachemode_tbl[8] = {
>   	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
>   };
>   
> -/* Check that the write-protect PAT entry is set for write-protect */
> +static bool x86_has_pat_mode(unsigned int mode)
> +{
> +	return __pte2cachemode_tbl[__cachemode2pte_tbl[mode]] == mode;
> +}
> +
> +/* Check that PAT supports write-protect */
>   bool x86_has_pat_wp(void)
>   {
> -	return __pte2cachemode_tbl[__cachemode2pte_tbl[_PAGE_CACHE_MODE_WP]] ==
> -	       _PAGE_CACHE_MODE_WP;
> +	return x86_has_pat_mode(_PAGE_CACHE_MODE_WP);
> +}
> +EXPORT_SYMBOL_GPL(x86_has_pat_wp);
> +
> +/* Check that PAT supports WC */
> +bool x86_has_pat_wc(void)
> +{
> +	return x86_has_pat_mode(_PAGE_CACHE_MODE_WC);
> +}
> +EXPORT_SYMBOL_GPL(x86_has_pat_wc);
> +
> +/* Check that PAT supports UC- */
> +bool x86_has_pat_uc_minus(void)
> +{
> +	return x86_has_pat_mode(_PAGE_CACHE_MODE_UC_MINUS);
>   }
> +EXPORT_SYMBOL_GPL(x86_has_pat_uc_minus);
>   
>   enum page_cache_mode pgprot2cachemode(pgprot_t pgprot)
>   {
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> index 0c5c43852e24..f43ecf3f63eb 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
>   	if (args->flags & ~(I915_MMAP_WC))
>   		return -EINVAL;
>   
> -	if (args->flags & I915_MMAP_WC && !pat_enabled())
> +	if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>   		return -ENODEV;
>   
>   	obj = i915_gem_object_lookup(file, args->handle);
> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>   
>   	if (HAS_LMEM(to_i915(dev)))
>   		mmap_type = I915_MMAP_TYPE_FIXED;
> -	else if (pat_enabled())
> +	else if (x86_has_pat_wc())
>   		mmap_type = I915_MMAP_TYPE_WC;
>   	else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>   		return -ENODEV;
> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>   		break;
>   
>   	case I915_MMAP_OFFSET_WC:
> -		if (!pat_enabled())
> +		if (!x86_has_pat_wc())
>   			return -ENODEV;
>   		type = I915_MMAP_TYPE_WC;
>   		break;
> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>   		break;
>   
>   	case I915_MMAP_OFFSET_UC:
> -		if (!pat_enabled())
> +		if (!x86_has_pat_uc_minus())
>   			return -ENODEV;
>   		type = I915_MMAP_TYPE_UC;
>   		break;

Chuck Zmudzinski May 20, 2022, 2:21 a.m. UTC | #5

On 5/19/22 10:15 PM, Chuck Zmudzinski wrote:
> On 5/3/22 9:22 AM, Juergen Gross wrote:
>> Some drivers are using pat_enabled() in order to test availability of
>> special caching modes (WC and UC-). This will lead to false negatives
>> in case the system was booted e.g. with the "nopat" variant and the
>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>> system is running as a Xen PV guest.
> Hello,
>
> I am also getting a false positive

Sorry, I meant false negative here, not false
positive.

Chuck

> in a Xen Dom0 from
> pat_enabled() where bdd8b6c98239 patched the file
>
> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>
> I think this patch also needs to touch that file to
> fix the issue I am seeing.
...

Chuck Zmudzinski May 20, 2022, 4:43 a.m. UTC | #6

On 5/4/22 5:14 AM, Juergen Gross wrote:
> On 04.05.22 10:31, Jan Beulich wrote:
>> On 03.05.2022 15:22, Juergen Gross wrote:
>>> Some drivers are using pat_enabled() in order to test availability of
>>> special caching modes (WC and UC-). This will lead to false negatives
>>> in case the system was booted e.g. with the "nopat" variant and the
>>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>>> system is running as a Xen PV guest.
>> ...
>>> Add test functions for those caching modes instead and use them at the
>>> appropriate places.
>>>
>>> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with 
>>> pat_enabled()")
>>> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
>>> Signed-off-by: Juergen Gross <[e-mail]>
>> ...
>>
>>> --- a/arch/x86/include/asm/pci.h
>>> +++ b/arch/x86/include/asm/pci.h
>>> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev, 
>>> int pin, int irq);
>>>       #define HAVE_PCI_MMAP
>>> -#define arch_can_pci_mmap_wc()    pat_enabled()
>>> +#define arch_can_pci_mmap_wc()    x86_has_pat_wc()
>>
>> Besides this and ...
>>
>>> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>>> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void 
>>> *data,
>>>       if (args->flags & ~(I915_MMAP_WC))
>>>           return -EINVAL;
>>>   -    if (args->flags & I915_MMAP_WC && !pat_enabled())
>>> +    if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>>>           return -ENODEV;
>>>         obj = i915_gem_object_lookup(file, args->handle);
>>> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>>>         if (HAS_LMEM(to_i915(dev)))
>>>           mmap_type = I915_MMAP_TYPE_FIXED;
>>> -    else if (pat_enabled())
>>> +    else if (x86_has_pat_wc())
>>>           mmap_type = I915_MMAP_TYPE_WC;
>>>       else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>>>           return -ENODEV;
>>> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device 
>>> *dev, void *data,
>>>           break;
>>>         case I915_MMAP_OFFSET_WC:
>>> -        if (!pat_enabled())
>>> +        if (!x86_has_pat_wc())
>>>               return -ENODEV;
>>>           type = I915_MMAP_TYPE_WC;
>>>           break;
>>> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device 
>>> *dev, void *data,
>>>           break;
>>>         case I915_MMAP_OFFSET_UC:
>>> -        if (!pat_enabled())
>>> +        if (!x86_has_pat_uc_minus())
>>>               return -ENODEV;
>>>           type = I915_MMAP_TYPE_UC;
>>>           break;
>>
>> ... these uses there are several more. You say nothing on why those want
>> leaving unaltered. When preparing my earlier patch I did inspect them
>> and came to the conclusion that these all would also better observe the
>> adjusted behavior (or else I couldn't have left pat_enabled() as the 
>> only
>> predicate). In fact, as said in the description of my earlier patch, in
>> my debugging I did find the use in i915_gem_object_pin_map() to be the
>> problematic one, which you leave alone.
>
> Oh, I missed that one, sorry.

That is why your patch would not fix my Haswell unless
it also touches i915_gem_object_pin_map() in
drivers/gpu/drm/i915/gem/i915_gem_pages.c

>
> I wanted to be rather defensive in my changes, but I agree at least the
> case in arch_phys_wc_add() might want to be changed, too.

I think your approach needs to be more aggressive so it will fix
all the known false negatives introduced by bdd8b6c98239
such as the one in i915_gem_object_pin_map().

I looked at Jan's approach and I think it would fix the issue
with my Haswell as long as I don't use the nopat option. I
really don't have a strong opinion on that question, but I
think the nopat option as a Linux kernel option, as opposed
to a hypervisor option, should only affect the kernel, and
if the hypervisor provides the pat feature, then the kernel
should not override that, but because of the confusion, maybe
a warning could be printed with the nopat option when a
hypervisor provides the feature so the user can at least have a
knob to tweak if if does not behave the way the user intends.
But I must admit, I don't know if the Xen hypervisor has an
option also to disable pat. If not, then maybe Jan's more
aggressive approach with nopat might be needed if for
some reason pat really needs to be disabled in the Linux
when Linux is running on Xen or another hypervisor, but I don't
know of any cases when that would be needed.

Chuck

Chuck Zmudzinski May 20, 2022, 5:56 a.m. UTC | #7

On 5/20/22 12:43 AM, Chuck Zmudzinski wrote:
> On 5/4/22 5:14 AM, Juergen Gross wrote:
>> On 04.05.22 10:31, Jan Beulich wrote:
>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>> Some drivers are using pat_enabled() in order to test availability of
>>>> special caching modes (WC and UC-). This will lead to false negatives
>>>> in case the system was booted e.g. with the "nopat" variant and the
>>>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>>>> system is running as a Xen PV guest.
>>> ...
>>>> Add test functions for those caching modes instead and use them at the
>>>> appropriate places.
>>>>
>>>> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with 
>>>> pat_enabled()")
>>>> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
>>>> Signed-off-by: Juergen Gross <[e-mail]>
>>> ...
>>>
>>>
>>> ... these uses there are several more. You say nothing on why those 
>>> want
>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>> and came to the conclusion that these all would also better observe the
>>> adjusted behavior (or else I couldn't have left pat_enabled() as the 
>>> only
>>> predicate). In fact, as said in the description of my earlier patch, in
>>> my debugging I did find the use in i915_gem_object_pin_map() to be the
>>> problematic one, which you leave alone.
>>
>> Oh, I missed that one, sorry.
>
> That is why your patch would not fix my Haswell unless
> it also touches i915_gem_object_pin_map() in
> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>
>>
>> I wanted to be rather defensive in my changes, but I agree at least the
>> case in arch_phys_wc_add() might want to be changed, too.
>
> I think your approach needs to be more aggressive so it will fix
> all the known false negatives introduced by bdd8b6c98239
> such as the one in i915_gem_object_pin_map().
>
> I looked at Jan's approach and I think it would fix the issue
> with my Haswell as long as I don't use the nopat option. I
> really don't have a strong opinion on that question, but I
> think the nopat option as a Linux kernel option, as opposed
> to a hypervisor option, should only affect the kernel, and
> if the hypervisor provides the pat feature, then the kernel
> should not override that, but because of the confusion,

The confusion is: does "nopat" only mean the kernel does not
provide pat to device drivers, or does it mean kernel drivers
are not to use pat even if the hypervisor provides it?
I think the original purpose of bdd8b6c98239 was to
enable "nopat" to disable the use or pat in the i915 driver
even if the feature is present from either the kernel or the
hypervisor. This interpretation of the meaning of "nopat"
would favor Jan's approach, I think.

Chuck

Jan Beulich May 20, 2022, 6:05 a.m. UTC | #8

On 20.05.2022 06:43, Chuck Zmudzinski wrote:
> On 5/4/22 5:14 AM, Juergen Gross wrote:
>> On 04.05.22 10:31, Jan Beulich wrote:
>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>> Some drivers are using pat_enabled() in order to test availability of
>>>> special caching modes (WC and UC-). This will lead to false negatives
>>>> in case the system was booted e.g. with the "nopat" variant and the
>>>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>>>> system is running as a Xen PV guest.
>>> ...
>>>> Add test functions for those caching modes instead and use them at the
>>>> appropriate places.
>>>>
>>>> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with 
>>>> pat_enabled()")
>>>> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
>>>> Signed-off-by: Juergen Gross <[e-mail]>
>>> ...
>>>
>>>> --- a/arch/x86/include/asm/pci.h
>>>> +++ b/arch/x86/include/asm/pci.h
>>>> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev, 
>>>> int pin, int irq);
>>>>       #define HAVE_PCI_MMAP
>>>> -#define arch_can_pci_mmap_wc()    pat_enabled()
>>>> +#define arch_can_pci_mmap_wc()    x86_has_pat_wc()
>>>
>>> Besides this and ...
>>>
>>>> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>>>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>>>> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void 
>>>> *data,
>>>>       if (args->flags & ~(I915_MMAP_WC))
>>>>           return -EINVAL;
>>>>   -    if (args->flags & I915_MMAP_WC && !pat_enabled())
>>>> +    if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>>>>           return -ENODEV;
>>>>         obj = i915_gem_object_lookup(file, args->handle);
>>>> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>>>>         if (HAS_LMEM(to_i915(dev)))
>>>>           mmap_type = I915_MMAP_TYPE_FIXED;
>>>> -    else if (pat_enabled())
>>>> +    else if (x86_has_pat_wc())
>>>>           mmap_type = I915_MMAP_TYPE_WC;
>>>>       else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>>>>           return -ENODEV;
>>>> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device 
>>>> *dev, void *data,
>>>>           break;
>>>>         case I915_MMAP_OFFSET_WC:
>>>> -        if (!pat_enabled())
>>>> +        if (!x86_has_pat_wc())
>>>>               return -ENODEV;
>>>>           type = I915_MMAP_TYPE_WC;
>>>>           break;
>>>> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device 
>>>> *dev, void *data,
>>>>           break;
>>>>         case I915_MMAP_OFFSET_UC:
>>>> -        if (!pat_enabled())
>>>> +        if (!x86_has_pat_uc_minus())
>>>>               return -ENODEV;
>>>>           type = I915_MMAP_TYPE_UC;
>>>>           break;
>>>
>>> ... these uses there are several more. You say nothing on why those want
>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>> and came to the conclusion that these all would also better observe the
>>> adjusted behavior (or else I couldn't have left pat_enabled() as the 
>>> only
>>> predicate). In fact, as said in the description of my earlier patch, in
>>> my debugging I did find the use in i915_gem_object_pin_map() to be the
>>> problematic one, which you leave alone.
>>
>> Oh, I missed that one, sorry.
> 
> That is why your patch would not fix my Haswell unless
> it also touches i915_gem_object_pin_map() in
> drivers/gpu/drm/i915/gem/i915_gem_pages.c
> 
>>
>> I wanted to be rather defensive in my changes, but I agree at least the
>> case in arch_phys_wc_add() might want to be changed, too.
> 
> I think your approach needs to be more aggressive so it will fix
> all the known false negatives introduced by bdd8b6c98239
> such as the one in i915_gem_object_pin_map().
> 
> I looked at Jan's approach and I think it would fix the issue
> with my Haswell as long as I don't use the nopat option. I
> really don't have a strong opinion on that question, but I
> think the nopat option as a Linux kernel option, as opposed
> to a hypervisor option, should only affect the kernel, and
> if the hypervisor provides the pat feature, then the kernel
> should not override that,

Hmm, why would the kernel not be allowed to override that? Such
an override would affect only the single domain where the
kernel runs; other domains could take their own decisions.

Also, for the sake of completeness: "nopat" used when running on
bare metal has the same bad effect on system boot, so there
pretty clearly is an error cleanup issue in the i915 driver. But
that's orthogonal, and I expect the maintainers may not even care
(but tell us "don't do that then").

Jan

> but because of the confusion, maybe
> a warning could be printed with the nopat option when a
> hypervisor provides the feature so the user can at least have a
> knob to tweak if if does not behave the way the user intends.
> But I must admit, I don't know if the Xen hypervisor has an
> option also to disable pat. If not, then maybe Jan's more
> aggressive approach with nopat might be needed if for
> some reason pat really needs to be disabled in the Linux
> when Linux is running on Xen or another hypervisor, but I don't
> know of any cases when that would be needed.
> 
> Chuck
>

Chuck Zmudzinski May 20, 2022, 6:59 a.m. UTC | #9

On 5/20/2022 2:05 AM, Jan Beulich wrote:
> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>> Some drivers are using pat_enabled() in order to test availability of
>>>>> special caching modes (WC and UC-). This will lead to false negatives
>>>>> in case the system was booted e.g. with the "nopat" variant and the
>>>>> BIOS did setup the PAT MSR supporting the queried mode, or if the
>>>>> system is running as a Xen PV guest.
>>>> ...
>>>>> Add test functions for those caching modes instead and use them at the
>>>>> appropriate places.
>>>>>
>>>>> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with
>>>>> pat_enabled()")
>>>>> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
>>>>> Signed-off-by: Juergen Gross <[e-mail]>
>>>> ...
>>>>
>>>>> --- a/arch/x86/include/asm/pci.h
>>>>> +++ b/arch/x86/include/asm/pci.h
>>>>> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev,
>>>>> int pin, int irq);
>>>>>        #define HAVE_PCI_MMAP
>>>>> -#define arch_can_pci_mmap_wc()    pat_enabled()
>>>>> +#define arch_can_pci_mmap_wc()    x86_has_pat_wc()
>>>> Besides this and ...
>>>>
>>>>> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>>>>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
>>>>> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void
>>>>> *data,
>>>>>        if (args->flags & ~(I915_MMAP_WC))
>>>>>            return -EINVAL;
>>>>>    -    if (args->flags & I915_MMAP_WC && !pat_enabled())
>>>>> +    if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>>>>>            return -ENODEV;
>>>>>          obj = i915_gem_object_lookup(file, args->handle);
>>>>> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>>>>>          if (HAS_LMEM(to_i915(dev)))
>>>>>            mmap_type = I915_MMAP_TYPE_FIXED;
>>>>> -    else if (pat_enabled())
>>>>> +    else if (x86_has_pat_wc())
>>>>>            mmap_type = I915_MMAP_TYPE_WC;
>>>>>        else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>>>>>            return -ENODEV;
>>>>> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device
>>>>> *dev, void *data,
>>>>>            break;
>>>>>          case I915_MMAP_OFFSET_WC:
>>>>> -        if (!pat_enabled())
>>>>> +        if (!x86_has_pat_wc())
>>>>>                return -ENODEV;
>>>>>            type = I915_MMAP_TYPE_WC;
>>>>>            break;
>>>>> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device
>>>>> *dev, void *data,
>>>>>            break;
>>>>>          case I915_MMAP_OFFSET_UC:
>>>>> -        if (!pat_enabled())
>>>>> +        if (!x86_has_pat_uc_minus())
>>>>>                return -ENODEV;
>>>>>            type = I915_MMAP_TYPE_UC;
>>>>>            break;
>>>> ... these uses there are several more. You say nothing on why those want
>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>> and came to the conclusion that these all would also better observe the
>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>> only
>>>> predicate). In fact, as said in the description of my earlier patch, in
>>>> my debugging I did find the use in i915_gem_object_pin_map() to be the
>>>> problematic one, which you leave alone.
>>> Oh, I missed that one, sorry.
>> That is why your patch would not fix my Haswell unless
>> it also touches i915_gem_object_pin_map() in
>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>
>>> I wanted to be rather defensive in my changes, but I agree at least the
>>> case in arch_phys_wc_add() might want to be changed, too.
>> I think your approach needs to be more aggressive so it will fix
>> all the known false negatives introduced by bdd8b6c98239
>> such as the one in i915_gem_object_pin_map().
>>
>> I looked at Jan's approach and I think it would fix the issue
>> with my Haswell as long as I don't use the nopat option. I
>> really don't have a strong opinion on that question, but I
>> think the nopat option as a Linux kernel option, as opposed
>> to a hypervisor option, should only affect the kernel, and
>> if the hypervisor provides the pat feature, then the kernel
>> should not override that,
> Hmm, why would the kernel not be allowed to override that? Such
> an override would affect only the single domain where the
> kernel runs; other domains could take their own decisions.
>
> Also, for the sake of completeness: "nopat" used when running on
> bare metal has the same bad effect on system boot, so there
> pretty clearly is an error cleanup issue in the i915 driver. But
> that's orthogonal, and I expect the maintainers may not even care
> (but tell us "don't do that then").
>
> Jan
>
>> but because of the confusion,

As I just wrote earlier, the confusion is whether or not "nopat"
means the kernel drivers will not use pat even if the firmware
and hypervisor provides it. I think you are correct to
point out that is the way the i915 driver behaved with the nopat
option before bdd8b6c98239 was applied, with the same
bad effects on bare metal as with the hypervisor. I think perhaps
dealing with the nopat option to fix bdd8b6c98239 is a solution in
search of a problem, at least as regards the i915 driver.

The only problem we have, as I see it, is with a false negative
when the nopat option is *not* enabled. But the forced disabling
of pat in Jan's patch when the nopat option is enabled is probably
needed if the goal of the patch is to preserve the same
behavior of the i915 driver that it had before bdd8b6c98239
was applied.

In any case, especially if we do include Jan's aggressive approach
of disabling pat with the nopat option and preserving the same bad
behavior we had with nopat before bdd8b6c98239 was applied, the
i915 driver should log a warning when pat is disabled. Right now,
the driver returns -ENODEV with the problem in
i915_gem_object_pin_map(), but it does not log an error. The only
log message I get now is the add_taint_for_CI in intel_gt_init
which was not very helpful information for debugging
this problem. It was only the starting point of a longer debugging
process because of a lack of error log messages in the i915 driver.

Chuck

Chuck Zmudzinski May 20, 2022, 8:30 a.m. UTC | #10

On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>
>>>>> ... these uses there are several more. You say nothing on why 
>>>>> those want
>>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>>> and came to the conclusion that these all would also better 
>>>>> observe the
>>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>>> only predicate). In fact, as said in the description of my earlier 
>>>>> patch, in
>>>>> my debugging I did find the use in i915_gem_object_pin_map() to be 
>>>>> the
>>>>> problematic one, which you leave alone.
>>>> Oh, I missed that one, sorry.
>>> That is why your patch would not fix my Haswell unless
>>> it also touches i915_gem_object_pin_map() in
>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>
>>>> I wanted to be rather defensive in my changes, but I agree at least 
>>>> the
>>>> case in arch_phys_wc_add() might want to be changed, too.
>>> I think your approach needs to be more aggressive so it will fix
>>> all the known false negatives introduced by bdd8b6c98239
>>> such as the one in i915_gem_object_pin_map().
>>>
>>> I looked at Jan's approach and I think it would fix the issue
>>> with my Haswell as long as I don't use the nopat option. I
>>> really don't have a strong opinion on that question, but I
>>> think the nopat option as a Linux kernel option, as opposed
>>> to a hypervisor option, should only affect the kernel, and
>>> if the hypervisor provides the pat feature, then the kernel
>>> should not override that,
>> Hmm, why would the kernel not be allowed to override that? Such
>> an override would affect only the single domain where the
>> kernel runs; other domains could take their own decisions.
>>
>> Also, for the sake of completeness: "nopat" used when running on
>> bare metal has the same bad effect on system boot, so there
>> pretty clearly is an error cleanup issue in the i915 driver. But
>> that's orthogonal, and I expect the maintainers may not even care
>> (but tell us "don't do that then").

Actually I just did a test with the last official Debian kernel
build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
applied. In fact, the nopat option does *not* break the i915 driver
in 5.16. That is, with the nopat option, the i915 driver loads
normally on both the bare metal and on the Xen hypervisor.
That means your presumption (and the presumption of
the author of bdd8b6c98239) that the "nopat" option was
being observed by the i915 driver is incorrect. Setting "nopat"
had no effect on my system with Linux 5.16. So after doing these
tests, I am against the aggressive approach of breaking the i915
driver with the "nopat" option because prior to bdd8b6c98239,
nopat did not break the i915 driver. Why break it now?

Prior to bdd8b6c98239, the i915 driver used
static_cpu_has(X86_FEATURE_PAT) to test for the PAT
feature, and apparently this returns true even if nopat
is set, but the new test, pat_enabled(), returns false on
the Xen hypervisor even if nopat is not set. That is
the only problem I see. The question of nopat should
be irrelevant to the i915 driver.

It was unfortunate that the author of bdd8b6c98239
mentioned nopat in the commit message when in fact
nopat was never intended to be used to break the
i915 driver. The i915 driver should ignore the nopat
option and decide what to do based solely on the
capability of the cpu, firmware, and the compiled
options of the Linux kernel. That is how it behaved
before bdd8b6c98239, and that behavior is what needs
to be restored with a patch.

Chuck

Jan Beulich May 20, 2022, 9:41 a.m. UTC | #11

On 20.05.2022 10:30, Chuck Zmudzinski wrote:
> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>
>>>>>> ... these uses there are several more. You say nothing on why 
>>>>>> those want
>>>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>>>> and came to the conclusion that these all would also better 
>>>>>> observe the
>>>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>>>> only predicate). In fact, as said in the description of my earlier 
>>>>>> patch, in
>>>>>> my debugging I did find the use in i915_gem_object_pin_map() to be 
>>>>>> the
>>>>>> problematic one, which you leave alone.
>>>>> Oh, I missed that one, sorry.
>>>> That is why your patch would not fix my Haswell unless
>>>> it also touches i915_gem_object_pin_map() in
>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>
>>>>> I wanted to be rather defensive in my changes, but I agree at least 
>>>>> the
>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>> I think your approach needs to be more aggressive so it will fix
>>>> all the known false negatives introduced by bdd8b6c98239
>>>> such as the one in i915_gem_object_pin_map().
>>>>
>>>> I looked at Jan's approach and I think it would fix the issue
>>>> with my Haswell as long as I don't use the nopat option. I
>>>> really don't have a strong opinion on that question, but I
>>>> think the nopat option as a Linux kernel option, as opposed
>>>> to a hypervisor option, should only affect the kernel, and
>>>> if the hypervisor provides the pat feature, then the kernel
>>>> should not override that,
>>> Hmm, why would the kernel not be allowed to override that? Such
>>> an override would affect only the single domain where the
>>> kernel runs; other domains could take their own decisions.
>>>
>>> Also, for the sake of completeness: "nopat" used when running on
>>> bare metal has the same bad effect on system boot, so there
>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>> that's orthogonal, and I expect the maintainers may not even care
>>> (but tell us "don't do that then").
> 
> Actually I just did a test with the last official Debian kernel
> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
> applied. In fact, the nopat option does *not* break the i915 driver
> in 5.16. That is, with the nopat option, the i915 driver loads
> normally on both the bare metal and on the Xen hypervisor.
> That means your presumption (and the presumption of
> the author of bdd8b6c98239) that the "nopat" option was
> being observed by the i915 driver is incorrect. Setting "nopat"
> had no effect on my system with Linux 5.16. So after doing these
> tests, I am against the aggressive approach of breaking the i915
> driver with the "nopat" option because prior to bdd8b6c98239,
> nopat did not break the i915 driver. Why break it now?

Because that's, in my understanding, is the purpose of "nopat"
(not breaking the driver of course - that's a driver bug -, but
having an effect on the driver).

Jan

Chuck Zmudzinski May 20, 2022, 1:33 p.m. UTC | #12

On 5/20/2022 5:41 AM, Jan Beulich wrote:
> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>
>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>> those want
>>>>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>>>>> and came to the conclusion that these all would also better
>>>>>>> observe the
>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>>>>> only predicate). In fact, as said in the description of my earlier
>>>>>>> patch, in
>>>>>>> my debugging I did find the use in i915_gem_object_pin_map() to be
>>>>>>> the
>>>>>>> problematic one, which you leave alone.
>>>>>> Oh, I missed that one, sorry.
>>>>> That is why your patch would not fix my Haswell unless
>>>>> it also touches i915_gem_object_pin_map() in
>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>
>>>>>> I wanted to be rather defensive in my changes, but I agree at least
>>>>>> the
>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>> I think your approach needs to be more aggressive so it will fix
>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>> such as the one in i915_gem_object_pin_map().
>>>>>
>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>> really don't have a strong opinion on that question, but I
>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>> to a hypervisor option, should only affect the kernel, and
>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>> should not override that,
>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>> an override would affect only the single domain where the
>>>> kernel runs; other domains could take their own decisions.
>>>>
>>>> Also, for the sake of completeness: "nopat" used when running on
>>>> bare metal has the same bad effect on system boot, so there
>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>> that's orthogonal, and I expect the maintainers may not even care
>>>> (but tell us "don't do that then").
>> Actually I just did a test with the last official Debian kernel
>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>> applied. In fact, the nopat option does *not* break the i915 driver
>> in 5.16. That is, with the nopat option, the i915 driver loads
>> normally on both the bare metal and on the Xen hypervisor.
>> That means your presumption (and the presumption of
>> the author of bdd8b6c98239) that the "nopat" option was
>> being observed by the i915 driver is incorrect. Setting "nopat"
>> had no effect on my system with Linux 5.16. So after doing these
>> tests, I am against the aggressive approach of breaking the i915
>> driver with the "nopat" option because prior to bdd8b6c98239,
>> nopat did not break the i915 driver. Why break it now?
> Because that's, in my understanding, is the purpose of "nopat"
> (not breaking the driver of course - that's a driver bug -, but
> having an effect on the driver).

I wouldn't call it a driver bug, but an incorrect configuration of the
kernel by the user.  I presume X86_FEATURE_PAT is required by the
i915 driver and therefore the driver should refuse to disable
it if the user requests to disable it and instead warn the user that
the driver did not disable the feature, contrary to what the user
requested with the nopat option.

In any case, my test did not verify that when nopat is set in Linux 5.16,
the thread takes the same code path as when nopat is not set,
so I am not totally sure that the reason nopat does not break the
i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
returns true even when nopat is set. I could test it with a custom
log message in 5.16 if that is necessary.

Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
to return true in 5.16 when the user requests nopat? I think that is
just permitting a bad configuration to break the driver that a
well-written operating system should not allow. The i915 driver
was, in my opinion, correctly ignoring the nopat option in 5.16
because that option is not compatible with the hardware the
i915 driver is trying to initialize and setup at boot time. At least
that is my understanding now, but I will need to test it on 5.16
to be sure I understand it correctly.

Also, AFAICT, your patch would break the driver when the nopat
option is set and only fix the regression introduced by bdd8b6c98239
when nopat is not set on my box, so your patch would
introduce a regression relative to Linux 5.16 and earlier for the
case when nopat is set on my box. I think your point would
be that it is not a regression if it is an incorrect user configuration.
I respond by saying a well-written driver should refuse to honor
the incorrect configuration requested by the user and instead
warn the user that it did not honor the incorrect kernel option.

I am only presuming what your patch would do on my box based
on what I learned about this problem from my debugging. I can
also test your patch on my box to verify that my understanding of
it is correct.

I also have not yet verified Juergen's patch will not fix it, but
I am almost certain it will not unless it is expanded so it also
touches i915_gem_object_pin_map() with the fix. I plan to test
his patch, but expanded so it touches that function also.

I also plan to test your patch with and without nopat and report the
results in the thread where you posted your patch. Hopefully
by tomorrow I will have the results.

Chuck

Jan Beulich May 20, 2022, 2:06 p.m. UTC | #13

On 20.05.2022 15:33, Chuck Zmudzinski wrote:
> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>
>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>> those want
>>>>>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>> observe the
>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>>>>>> only predicate). In fact, as said in the description of my earlier
>>>>>>>> patch, in
>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map() to be
>>>>>>>> the
>>>>>>>> problematic one, which you leave alone.
>>>>>>> Oh, I missed that one, sorry.
>>>>>> That is why your patch would not fix my Haswell unless
>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>
>>>>>>> I wanted to be rather defensive in my changes, but I agree at least
>>>>>>> the
>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>
>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>> really don't have a strong opinion on that question, but I
>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>> should not override that,
>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>> an override would affect only the single domain where the
>>>>> kernel runs; other domains could take their own decisions.
>>>>>
>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>> bare metal has the same bad effect on system boot, so there
>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>> (but tell us "don't do that then").
>>> Actually I just did a test with the last official Debian kernel
>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>> applied. In fact, the nopat option does *not* break the i915 driver
>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>> normally on both the bare metal and on the Xen hypervisor.
>>> That means your presumption (and the presumption of
>>> the author of bdd8b6c98239) that the "nopat" option was
>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>> had no effect on my system with Linux 5.16. So after doing these
>>> tests, I am against the aggressive approach of breaking the i915
>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>> nopat did not break the i915 driver. Why break it now?
>> Because that's, in my understanding, is the purpose of "nopat"
>> (not breaking the driver of course - that's a driver bug -, but
>> having an effect on the driver).
> 
> I wouldn't call it a driver bug, but an incorrect configuration of the
> kernel by the user.  I presume X86_FEATURE_PAT is required by the
> i915 driver

The driver ought to work fine without PAT (and hence without being
able to make WC mappings). It would use UC instead and be slow, but
it ought to work.

> and therefore the driver should refuse to disable
> it if the user requests to disable it and instead warn the user that
> the driver did not disable the feature, contrary to what the user
> requested with the nopat option.
> 
> In any case, my test did not verify that when nopat is set in Linux 5.16,
> the thread takes the same code path as when nopat is not set,
> so I am not totally sure that the reason nopat does not break the
> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
> returns true even when nopat is set. I could test it with a custom
> log message in 5.16 if that is necessary.
> 
> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
> to return true in 5.16 when the user requests nopat?

No, I'm not saying that. It was wrong for this construct to be used
in the driver, which was fixed for 5.17 (and which had caused the
regression I did observe, leading to the patch as a hopefully least
bad option).

> I think that is
> just permitting a bad configuration to break the driver that a
> well-written operating system should not allow. The i915 driver
> was, in my opinion, correctly ignoring the nopat option in 5.16
> because that option is not compatible with the hardware the
> i915 driver is trying to initialize and setup at boot time. At least
> that is my understanding now, but I will need to test it on 5.16
> to be sure I understand it correctly.
> 
> Also, AFAICT, your patch would break the driver when the nopat
> option is set and only fix the regression introduced by bdd8b6c98239
> when nopat is not set on my box, so your patch would
> introduce a regression relative to Linux 5.16 and earlier for the
> case when nopat is set on my box. I think your point would
> be that it is not a regression if it is an incorrect user configuration.

Again no - my view is that there's a separate, pre-existing issue
in the driver which was uncovered by the change. This may be a
perceived regression, but is imo different from a real one.

Jan

> I respond by saying a well-written driver should refuse to honor
> the incorrect configuration requested by the user and instead
> warn the user that it did not honor the incorrect kernel option.
> 
> I am only presuming what your patch would do on my box based
> on what I learned about this problem from my debugging. I can
> also test your patch on my box to verify that my understanding of
> it is correct.
> 
> I also have not yet verified Juergen's patch will not fix it, but
> I am almost certain it will not unless it is expanded so it also
> touches i915_gem_object_pin_map() with the fix. I plan to test
> his patch, but expanded so it touches that function also.
> 
> I also plan to test your patch with and without nopat and report the
> results in the thread where you posted your patch. Hopefully
> by tomorrow I will have the results.
> 
> Chuck
>

Chuck Zmudzinski May 20, 2022, 2:48 p.m. UTC | #14

On 5/20/2022 10:06 AM, Jan Beulich wrote:
> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>
>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>> those want
>>>>>>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>> observe the
>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>>>>>>> only predicate). In fact, as said in the description of my earlier
>>>>>>>>> patch, in
>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map() to be
>>>>>>>>> the
>>>>>>>>> problematic one, which you leave alone.
>>>>>>>> Oh, I missed that one, sorry.
>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>
>>>>>>>> I wanted to be rather defensive in my changes, but I agree at least
>>>>>>>> the
>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>
>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>> should not override that,
>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>> an override would affect only the single domain where the
>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>
>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>> (but tell us "don't do that then").
>>>> Actually I just did a test with the last official Debian kernel
>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>> normally on both the bare metal and on the Xen hypervisor.
>>>> That means your presumption (and the presumption of
>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>> had no effect on my system with Linux 5.16. So after doing these
>>>> tests, I am against the aggressive approach of breaking the i915
>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>> nopat did not break the i915 driver. Why break it now?
>>> Because that's, in my understanding, is the purpose of "nopat"
>>> (not breaking the driver of course - that's a driver bug -, but
>>> having an effect on the driver).
>> I wouldn't call it a driver bug, but an incorrect configuration of the
>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>> i915 driver
> The driver ought to work fine without PAT (and hence without being
> able to make WC mappings). It would use UC instead and be slow, but
> it ought to work.
>
>> and therefore the driver should refuse to disable
>> it if the user requests to disable it and instead warn the user that
>> the driver did not disable the feature, contrary to what the user
>> requested with the nopat option.
>>
>> In any case, my test did not verify that when nopat is set in Linux 5.16,
>> the thread takes the same code path as when nopat is not set,
>> so I am not totally sure that the reason nopat does not break the
>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>> returns true even when nopat is set. I could test it with a custom
>> log message in 5.16 if that is necessary.
>>
>> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
>> to return true in 5.16 when the user requests nopat?
> No, I'm not saying that. It was wrong for this construct to be used
> in the driver, which was fixed for 5.17 (and which had caused the
> regression I did observe, leading to the patch as a hopefully least
> bad option).
>
>> I think that is
>> just permitting a bad configuration to break the driver that a
>> well-written operating system should not allow. The i915 driver
>> was, in my opinion, correctly ignoring the nopat option in 5.16
>> because that option is not compatible with the hardware the
>> i915 driver is trying to initialize and setup at boot time. At least
>> that is my understanding now, but I will need to test it on 5.16
>> to be sure I understand it correctly.
>>
>> Also, AFAICT, your patch would break the driver when the nopat
>> option is set and only fix the regression introduced by bdd8b6c98239
>> when nopat is not set on my box, so your patch would
>> introduce a regression relative to Linux 5.16 and earlier for the
>> case when nopat is set on my box. I think your point would
>> be that it is not a regression if it is an incorrect user configuration.
> Again no - my view is that there's a separate, pre-existing issue
> in the driver which was uncovered by the change. This may be a
> perceived regression, but is imo different from a real one.
>
> Jan

Since it is a regression, I think for now bdd8b6c98239 should
be reverted and the fix backported to Linux 5.17 stable until
the underlying memory subsystem can provide the i915 driver
with an updated test for the PAT feature that also meets the
requirements of the author of bdd8b6c98239 without breaking
the i915 driver. The i915 driver relies on the memory subsytem
to provide it with an accurate test for the existence of
X86_FEATURE_PAT. I think your patch provides that more accurate
test so that bdd8b6c98239 could be re-applied when your patch is
committed. Juergen's patch would have to touch bdd8b6c98239
with new functions that probably have unknown and unintended
consequences, so I think your approach is also better in that regard.
As regards your patch, there is just a disagreement about how the
i915 driver should behave if nopat is set. I agree the i915 driver
could do a better job handling that case, at least with better error
logs.

Chuck

>
>> I respond by saying a well-written driver should refuse to honor
>> the incorrect configuration requested by the user and instead
>> warn the user that it did not honor the incorrect kernel option.
>>
>> I am only presuming what your patch would do on my box based
>> on what I learned about this problem from my debugging. I can
>> also test your patch on my box to verify that my understanding of
>> it is correct.
>>
>> I also have not yet verified Juergen's patch will not fix it, but
>> I am almost certain it will not unless it is expanded so it also
>> touches i915_gem_object_pin_map() with the fix. I plan to test
>> his patch, but expanded so it touches that function also.
>>
>> I also plan to test your patch with and without nopat and report the
>> results in the thread where you posted your patch. Hopefully
>> by tomorrow I will have the results.
>>
>> Chuck
>>

Chuck Zmudzinski May 20, 2022, 3:46 p.m. UTC | #15

On 5/20/2022 10:06 AM, Jan Beulich wrote:
> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>
>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>> those want
>>>>>>>>> leaving unaltered. When preparing my earlier patch I did inspect them
>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>> observe the
>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled() as the
>>>>>>>>> only predicate). In fact, as said in the description of my earlier
>>>>>>>>> patch, in
>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map() to be
>>>>>>>>> the
>>>>>>>>> problematic one, which you leave alone.
>>>>>>>> Oh, I missed that one, sorry.
>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>
>>>>>>>> I wanted to be rather defensive in my changes, but I agree at least
>>>>>>>> the
>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>
>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>> should not override that,
>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>> an override would affect only the single domain where the
>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>
>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>> (but tell us "don't do that then").
>>>> Actually I just did a test with the last official Debian kernel
>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>> normally on both the bare metal and on the Xen hypervisor.
>>>> That means your presumption (and the presumption of
>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>> had no effect on my system with Linux 5.16. So after doing these
>>>> tests, I am against the aggressive approach of breaking the i915
>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>> nopat did not break the i915 driver. Why break it now?
>>> Because that's, in my understanding, is the purpose of "nopat"
>>> (not breaking the driver of course - that's a driver bug -, but
>>> having an effect on the driver).
>> I wouldn't call it a driver bug, but an incorrect configuration of the
>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>> i915 driver
> The driver ought to work fine without PAT (and hence without being
> able to make WC mappings). It would use UC instead and be slow, but
> it ought to work.

I am not an expert, but I think the reason it failed on my box was
because of the requirements of CI. Maybe the driver would fall back
to UC if the add_taint_for_CI function did not halt the entire system
in response to the failed test for PAT when trying to use WC mappings.

>> and therefore the driver should refuse to disable
>> it if the user requests to disable it and instead warn the user that
>> the driver did not disable the feature, contrary to what the user
>> requested with the nopat option.
>>
>> In any case, my test did not verify that when nopat is set in Linux 5.16,
>> the thread takes the same code path as when nopat is not set,
>> so I am not totally sure that the reason nopat does not break the
>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>> returns true even when nopat is set. I could test it with a custom
>> log message in 5.16 if that is necessary.
>>
>> Are you saying it was wrong for
>> to return true in 5.16 when the user requests nopat?
> No, I'm not saying that. It was wrong for this construct to be used
> in the driver, which was fixed for 5.17 (and which had caused the
> regression I did observe, leading to the patch as a hopefully least
> bad option).

Hmm, the patch I used to fix my box with 5.17.6 used
static_cpu_has(X86_FEATURE_PAT) so the driver could
continue to configure the hardware using WC. This is the
relevant part of the patch I used to fix my box, which includes
extra error logs, (against Debian's official build of 5.17.6):

--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c    2022-05-09 
03:16:33.000000000 -0400
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c    2022-05-19 
15:55:40.339778818 -0400
...
@@ -430,17 +434,23 @@
          err = i915_gem_object_wait_moving_fence(obj, true);
          if (err) {
              ptr = ERR_PTR(err);
+            DRM_ERROR("i915_gem_object_wait_moving_fence error, err = 
%d\n", err);
              goto err_unpin;
          }

-        if (GEM_WARN_ON(type == I915_MAP_WC && !pat_enabled()))
+        if (GEM_WARN_ON(type == I915_MAP_WC &&
+                !pat_enabled() && !static_cpu_has(X86_FEATURE_PAT))) {
+            DRM_ERROR("type == I915_MAP_WC && !pat_enabled(), err = 
%d\n", -ENODEV);
              ptr = ERR_PTR(-ENODEV);
+        }
          else if (i915_gem_object_has_struct_page(obj))
              ptr = i915_gem_object_map_page(obj, type);
          else
              ptr = i915_gem_object_map_pfn(obj, type);
-        if (IS_ERR(ptr))
+        if (IS_ERR(ptr)) {
+            DRM_ERROR("IS_ERR(PTR) is true, returning a (ptr) error\n");
              goto err_unpin;
+        }

          obj->mm.mapping = page_pack_bits(ptr, type);
      }

As you can see, adding the static_cpu_has(X86_FEATURE_PAT)
function to the test for PAT restored the behavior of 5.16 on the
Xen hypervisor to 5.17, and that is how I discovered the solution
to this problem on 5.17 on my box.

>> I think that is
>> just permitting a bad configuration to break the driver that a
>> well-written operating system should not allow. The i915 driver
>> was, in my opinion, correctly ignoring the nopat option in 5.16
>> because that option is not compatible with the hardware the
>> i915 driver is trying to initialize and setup at boot time. At least
>> that is my understanding now, but I will need to test it on 5.16
>> to be sure I understand it correctly.
>>
>> Also, AFAICT, your patch would break the driver when the nopat
>> option is set and only fix the regression introduced by bdd8b6c98239
>> when nopat is not set on my box, so your patch would
>> introduce a regression relative to Linux 5.16 and earlier for the
>> case when nopat is set on my box. I think your point would
>> be that it is not a regression if it is an incorrect user configuration.
> Again no - my view is that there's a separate, pre-existing issue
> in the driver which was uncovered by the change. This may be a
> perceived regression, but is imo different from a real one.

Maybe it is only a perceived regression if nopat is set, but
imo bdd8b6c98239 introduced a real regression in 5.17
relative to 5.16 for the correctly and identically configured
case when the nopat option is not set. That is why I still think
it should be reverted and the fix backported to 5.17 until the
regression for the case when nopat is not set is fixed. As I
said before, the i915 driver relies on the memory subsyste
to provide it with an accurate test for the x86 pat feature.
The test the driver used in bdd8b6c98239 gives the i915 driver
a false negative, and that caused a real regression when nopat
is not set. bdd8b6c98239 can be re-applied if we apply your
patch which corrects the false negative that pat_enabled() is
currently providing the i915 driver with. That false negative
from pat_enabled() is not an i915 bug, it is a bug in x86/pat.

Chuck

Chuck Zmudzinski May 20, 2022, 5:13 p.m. UTC | #16

I think this summary of the regression is appropriate for a top-post. 
Details follow below.

commit bdd8b6c98239: introduced what I call a real regression which 
persists in 5.17.x

Jan's proposed patch: 
https://lore.kernel.org/lkml/[e-mail]/

Jan's patch would fix the real regression introduced by bdd8b6c98239 when
the nopat option is not enabled, but when the nopat option is enabled, this
patch would introduce what Jan calls a "perceived regression" that is really
caused by the failure of the i915 driver to handle the case of the nopat 
option
being provided on the command line properly.

What I request: commit Jan's proposed patch, and backport it to 5.17. That
would fix the real regression and only cause a perceived regression for the
case when nopat is enabled. In that case, patches to the i915 driver
would be helpful but necessary to fix a regression.

Regard,

Chuck Zmudzinski

On 5/20/2022 11:46 AM, Chuck Zmudzinski wrote:
> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>
>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>> those want
>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did 
>>>>>>>>>> inspect them
>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>> observe the
>>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled() 
>>>>>>>>>> as the
>>>>>>>>>> only predicate). In fact, as said in the description of my 
>>>>>>>>>> earlier
>>>>>>>>>> patch, in
>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map() 
>>>>>>>>>> to be
>>>>>>>>>> the
>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>
>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at 
>>>>>>>>> least
>>>>>>>>> the
>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>
>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>> should not override that,
>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>> an override would affect only the single domain where the
>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>
>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>> (but tell us "don't do that then").
>>>>> Actually I just did a test with the last official Debian kernel
>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>> That means your presumption (and the presumption of
>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>> nopat did not break the i915 driver. Why break it now?
>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>> (not breaking the driver of course - that's a driver bug -, but
>>>> having an effect on the driver).
>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>> i915 driver
>> The driver ought to work fine without PAT (and hence without being
>> able to make WC mappings). It would use UC instead and be slow, but
>> it ought to work.
>
> I am not an expert, but I think the reason it failed on my box was
> because of the requirements of CI. Maybe the driver would fall back
> to UC if the add_taint_for_CI function did not halt the entire system
> in response to the failed test for PAT when trying to use WC mappings.
>
>>> and therefore the driver should refuse to disable
>>> it if the user requests to disable it and instead warn the user that
>>> the driver did not disable the feature, contrary to what the user
>>> requested with the nopat option.
>>>
>>> In any case, my test did not verify that when nopat is set in Linux 
>>> 5.16,
>>> the thread takes the same code path as when nopat is not set,
>>> so I am not totally sure that the reason nopat does not break the
>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>> returns true even when nopat is set. I could test it with a custom
>>> log message in 5.16 if that is necessary.
>>>
>>> Are you saying it was wrong for
>>> to return true in 5.16 when the user requests nopat?
>> No, I'm not saying that. It was wrong for this construct to be used
>> in the driver, which was fixed for 5.17 (and which had caused the
>> regression I did observe, leading to the patch as a hopefully least
>> bad option).
>
> Hmm, the patch I used to fix my box with 5.17.6 used
> static_cpu_has(X86_FEATURE_PAT) so the driver could
> continue to configure the hardware using WC. This is the
> relevant part of the patch I used to fix my box, which includes
> extra error logs, (against Debian's official build of 5.17.6):
>
> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c    2022-05-09 
> 03:16:33.000000000 -0400
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c    2022-05-19 
> 15:55:40.339778818 -0400
> ...
> @@ -430,17 +434,23 @@
>          err = i915_gem_object_wait_moving_fence(obj, true);
>          if (err) {
>              ptr = ERR_PTR(err);
> +            DRM_ERROR("i915_gem_object_wait_moving_fence error, err = 
> %d\n", err);
>              goto err_unpin;
>          }
>
> -        if (GEM_WARN_ON(type == I915_MAP_WC && !pat_enabled()))
> +        if (GEM_WARN_ON(type == I915_MAP_WC &&
> +                !pat_enabled() && !static_cpu_has(X86_FEATURE_PAT))) {
> +            DRM_ERROR("type == I915_MAP_WC && !pat_enabled(), err = 
> %d\n", -ENODEV);
>              ptr = ERR_PTR(-ENODEV);
> +        }
>          else if (i915_gem_object_has_struct_page(obj))
>              ptr = i915_gem_object_map_page(obj, type);
>          else
>              ptr = i915_gem_object_map_pfn(obj, type);
> -        if (IS_ERR(ptr))
> +        if (IS_ERR(ptr)) {
> +            DRM_ERROR("IS_ERR(PTR) is true, returning a (ptr) error\n");
>              goto err_unpin;
> +        }
>
>          obj->mm.mapping = page_pack_bits(ptr, type);
>      }
>
> As you can see, adding the static_cpu_has(X86_FEATURE_PAT)
> function to the test for PAT restored the behavior of 5.16 on the
> Xen hypervisor to 5.17, and that is how I discovered the solution
> to this problem on 5.17 on my box.
>
>>> I think that is
>>> just permitting a bad configuration to break the driver that a
>>> well-written operating system should not allow. The i915 driver
>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>> because that option is not compatible with the hardware the
>>> i915 driver is trying to initialize and setup at boot time. At least
>>> that is my understanding now, but I will need to test it on 5.16
>>> to be sure I understand it correctly.
>>>
>>> Also, AFAICT, your patch would break the driver when the nopat
>>> option is set and only fix the regression introduced by bdd8b6c98239
>>> when nopat is not set on my box, so your patch would
>>> introduce a regression relative to Linux 5.16 and earlier for the
>>> case when nopat is set on my box. I think your point would
>>> be that it is not a regression if it is an incorrect user 
>>> configuration.
>> Again no - my view is that there's a separate, pre-existing issue
>> in the driver which was uncovered by the change. This may be a
>> perceived regression, but is imo different from a real one.
>
> Maybe it is only a perceived regression if nopat is set, but
> imo bdd8b6c98239 introduced a real regression in 5.17
> relative to 5.16 for the correctly and identically configured
> case when the nopat option is not set. That is why I still think
> it should be reverted and the fix backported to 5.17 until the
> regression for the case when nopat is not set is fixed. As I
> said before, the i915 driver relies on the memory subsyste
> to provide it with an accurate test for the x86 pat feature.
> The test the driver used in bdd8b6c98239 gives the i915 driver
> a false negative, and that caused a real regression when nopat
> is not set. bdd8b6c98239 can be re-applied if we apply your
> patch which corrects the false negative that pat_enabled() is
> currently providing the i915 driver with. That false negative
> from pat_enabled() is not an i915 bug, it is a bug in x86/pat.
>
> Chuck

Chuck Zmudzinski May 20, 2022, 5:17 p.m. UTC | #17

On 5/20/2022 1:13 PM, Chuck Zmudzinski wrote:
> I think this summary of the regression is appropriate for a top-post. 
> Details follow below.
>
> commit bdd8b6c98239: introduced what I call a real regression which 
> persists in 5.17.x
>
> Jan's proposed patch: 
> https://lore.kernel.org/lkml/[e-mail]/
>
> Jan's patch would fix the real regression introduced by bdd8b6c98239 when
> the nopat option is not enabled, but when the nopat option is enabled, 
> this
> patch would introduce what Jan calls a "perceived regression" that is 
> really
> caused by the failure of the i915 driver to handle the case of the 
> nopat option
> being provided on the command line properly.
>
> What I request: commit Jan's proposed patch, and backport it to 5.17. 
> That
> would fix the real regression and only cause a perceived regression 
> for the
> case when nopat is enabled. In that case, patches to the i915 driver
> would be helpful but necessary to fix a regression.

Sorry again, I mean patches to i915 would be helpful but *not* necessary
to fix a regression.

Regards,

Chuck Zmudzinski

>
> On 5/20/2022 11:46 AM, Chuck Zmudzinski wrote:
>> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>>
>>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>>> those want
>>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did 
>>>>>>>>>>> inspect them
>>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>>> observe the
>>>>>>>>>>> adjusted behavior (or else I couldn't have left 
>>>>>>>>>>> pat_enabled() as the
>>>>>>>>>>> only predicate). In fact, as said in the description of my 
>>>>>>>>>>> earlier
>>>>>>>>>>> patch, in
>>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map() 
>>>>>>>>>>> to be
>>>>>>>>>>> the
>>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>>
>>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at 
>>>>>>>>>> least
>>>>>>>>>> the
>>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>>
>>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>>> should not override that,
>>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>>> an override would affect only the single domain where the
>>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>>
>>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>>> (but tell us "don't do that then").
>>>>>> Actually I just did a test with the last official Debian kernel
>>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>>> That means your presumption (and the presumption of
>>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>>> nopat did not break the i915 driver. Why break it now?
>>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>>> (not breaking the driver of course - that's a driver bug -, but
>>>>> having an effect on the driver).
>>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>>> i915 driver
>>> The driver ought to work fine without PAT (and hence without being
>>> able to make WC mappings). It would use UC instead and be slow, but
>>> it ought to work.
>>
>> I am not an expert, but I think the reason it failed on my box was
>> because of the requirements of CI. Maybe the driver would fall back
>> to UC if the add_taint_for_CI function did not halt the entire system
>> in response to the failed test for PAT when trying to use WC mappings.
>>
>>>> and therefore the driver should refuse to disable
>>>> it if the user requests to disable it and instead warn the user that
>>>> the driver did not disable the feature, contrary to what the user
>>>> requested with the nopat option.
>>>>
>>>> In any case, my test did not verify that when nopat is set in Linux 
>>>> 5.16,
>>>> the thread takes the same code path as when nopat is not set,
>>>> so I am not totally sure that the reason nopat does not break the
>>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>>> returns true even when nopat is set. I could test it with a custom
>>>> log message in 5.16 if that is necessary.
>>>>
>>>> Are you saying it was wrong for
>>>> to return true in 5.16 when the user requests nopat?
>>> No, I'm not saying that. It was wrong for this construct to be used
>>> in the driver, which was fixed for 5.17 (and which had caused the
>>> regression I did observe, leading to the patch as a hopefully least
>>> bad option).
>>
>> Hmm, the patch I used to fix my box with 5.17.6 used
>> static_cpu_has(X86_FEATURE_PAT) so the driver could
>> continue to configure the hardware using WC. This is the
>> relevant part of the patch I used to fix my box, which includes
>> extra error logs, (against Debian's official build of 5.17.6):
>>
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c    2022-05-09 
>> 03:16:33.000000000 -0400
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c    2022-05-19 
>> 15:55:40.339778818 -0400
>> ...
>> @@ -430,17 +434,23 @@
>>          err = i915_gem_object_wait_moving_fence(obj, true);
>>          if (err) {
>>              ptr = ERR_PTR(err);
>> +            DRM_ERROR("i915_gem_object_wait_moving_fence error, err 
>> = %d\n", err);
>>              goto err_unpin;
>>          }
>>
>> -        if (GEM_WARN_ON(type == I915_MAP_WC && !pat_enabled()))
>> +        if (GEM_WARN_ON(type == I915_MAP_WC &&
>> +                !pat_enabled() && !static_cpu_has(X86_FEATURE_PAT))) {
>> +            DRM_ERROR("type == I915_MAP_WC && !pat_enabled(), err = 
>> %d\n", -ENODEV);
>>              ptr = ERR_PTR(-ENODEV);
>> +        }
>>          else if (i915_gem_object_has_struct_page(obj))
>>              ptr = i915_gem_object_map_page(obj, type);
>>          else
>>              ptr = i915_gem_object_map_pfn(obj, type);
>> -        if (IS_ERR(ptr))
>> +        if (IS_ERR(ptr)) {
>> +            DRM_ERROR("IS_ERR(PTR) is true, returning a (ptr) 
>> error\n");
>>              goto err_unpin;
>> +        }
>>
>>          obj->mm.mapping = page_pack_bits(ptr, type);
>>      }
>>
>> As you can see, adding the static_cpu_has(X86_FEATURE_PAT)
>> function to the test for PAT restored the behavior of 5.16 on the
>> Xen hypervisor to 5.17, and that is how I discovered the solution
>> to this problem on 5.17 on my box.
>>
>>>> I think that is
>>>> just permitting a bad configuration to break the driver that a
>>>> well-written operating system should not allow. The i915 driver
>>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>>> because that option is not compatible with the hardware the
>>>> i915 driver is trying to initialize and setup at boot time. At least
>>>> that is my understanding now, but I will need to test it on 5.16
>>>> to be sure I understand it correctly.
>>>>
>>>> Also, AFAICT, your patch would break the driver when the nopat
>>>> option is set and only fix the regression introduced by bdd8b6c98239
>>>> when nopat is not set on my box, so your patch would
>>>> introduce a regression relative to Linux 5.16 and earlier for the
>>>> case when nopat is set on my box. I think your point would
>>>> be that it is not a regression if it is an incorrect user 
>>>> configuration.
>>> Again no - my view is that there's a separate, pre-existing issue
>>> in the driver which was uncovered by the change. This may be a
>>> perceived regression, but is imo different from a real one.
>>
>> Maybe it is only a perceived regression if nopat is set, but
>> imo bdd8b6c98239 introduced a real regression in 5.17
>> relative to 5.16 for the correctly and identically configured
>> case when the nopat option is not set. That is why I still think
>> it should be reverted and the fix backported to 5.17 until the
>> regression for the case when nopat is not set is fixed. As I
>> said before, the i915 driver relies on the memory subsyste
>> to provide it with an accurate test for the x86 pat feature.
>> The test the driver used in bdd8b6c98239 gives the i915 driver
>> a false negative, and that caused a real regression when nopat
>> is not set. bdd8b6c98239 can be re-applied if we apply your
>> patch which corrects the false negative that pat_enabled() is
>> currently providing the i915 driver with. That false negative
>> from pat_enabled() is not an i915 bug, it is a bug in x86/pat.
>>
>> Chuck
>

Thorsten Leemhuis May 21, 2022, 10:47 a.m. UTC | #18

On 20.05.22 16:48, Chuck Zmudzinski wrote:
> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>
>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>> those want
>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did
>>>>>>>>>> inspect them
>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>> observe the
>>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled()
>>>>>>>>>> as the
>>>>>>>>>> only predicate). In fact, as said in the description of my
>>>>>>>>>> earlier
>>>>>>>>>> patch, in
>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map()
>>>>>>>>>> to be
>>>>>>>>>> the
>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>
>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at
>>>>>>>>> least
>>>>>>>>> the
>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>
>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>> should not override that,
>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>> an override would affect only the single domain where the
>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>
>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>> (but tell us "don't do that then").
>>>>> Actually I just did a test with the last official Debian kernel
>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>> That means your presumption (and the presumption of
>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>> nopat did not break the i915 driver. Why break it now?
>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>> (not breaking the driver of course - that's a driver bug -, but
>>>> having an effect on the driver).
>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>> i915 driver
>> The driver ought to work fine without PAT (and hence without being
>> able to make WC mappings). It would use UC instead and be slow, but
>> it ought to work.
>>
>>> and therefore the driver should refuse to disable
>>> it if the user requests to disable it and instead warn the user that
>>> the driver did not disable the feature, contrary to what the user
>>> requested with the nopat option.
>>>
>>> In any case, my test did not verify that when nopat is set in Linux
>>> 5.16,
>>> the thread takes the same code path as when nopat is not set,
>>> so I am not totally sure that the reason nopat does not break the
>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>> returns true even when nopat is set. I could test it with a custom
>>> log message in 5.16 if that is necessary.
>>>
>>> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
>>> to return true in 5.16 when the user requests nopat?
>> No, I'm not saying that. It was wrong for this construct to be used
>> in the driver, which was fixed for 5.17 (and which had caused the
>> regression I did observe, leading to the patch as a hopefully least
>> bad option).
>>
>>> I think that is
>>> just permitting a bad configuration to break the driver that a
>>> well-written operating system should not allow. The i915 driver
>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>> because that option is not compatible with the hardware the
>>> i915 driver is trying to initialize and setup at boot time. At least
>>> that is my understanding now, but I will need to test it on 5.16
>>> to be sure I understand it correctly.
>>>
>>> Also, AFAICT, your patch would break the driver when the nopat
>>> option is set and only fix the regression introduced by bdd8b6c98239
>>> when nopat is not set on my box, so your patch would
>>> introduce a regression relative to Linux 5.16 and earlier for the
>>> case when nopat is set on my box. I think your point would
>>> be that it is not a regression if it is an incorrect user configuration.
>> Again no - my view is that there's a separate, pre-existing issue
>> in the driver which was uncovered by the change. This may be a
>> perceived regression, but is imo different from a real one.

Sorry, for you maybe, but I'm pretty sure for Linus it's not when it
comes to the "no regressions rule". Just took a quick look at quotes
from Linus
https://www.kernel.org/doc/html/latest/process/handling-regressions.html
and found this statement from Linus to back this up:

```
One _particularly_ last-minute revert is the top-most commit (ignoring
the version change itself) done just before the release, and while
it's very annoying, it's perhaps also instructive.

What's instructive about it is that I reverted a commit that wasn't
actually buggy. In fact, it was doing exactly what it set out to do,
and did it very well. In fact it did it _so_ well that the much
improved IO patterns it caused then ended up revealing a user-visible
regression due to a real bug in a completely unrelated area.
```

He said that here:
https://www.kernel.org/doc/html/latest/process/handling-regressions.html

The situation is of course different here, but similar enough.

> Since it is a regression, I think for now bdd8b6c98239 should
> be reverted and the fix backported to Linux 5.17 stable until
> the underlying memory subsystem can provide the i915 driver
> with an updated test for the PAT feature that also meets the
> requirements of the author of bdd8b6c98239 without breaking
> the i915 driver.

I'm not a developer and I'm don't known the details of this thread and
the backstory of the regression, but it sounds like that's the approach
that is needed here until someone comes up with a fix for the regression
exposed by bdd8b6c98239.

But if I'm wrong, please tell me.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

> The i915 driver relies on the memory subsytem
> to provide it with an accurate test for the existence of
> X86_FEATURE_PAT. I think your patch provides that more accurate
> test so that bdd8b6c98239 could be re-applied when your patch is
> committed. Juergen's patch would have to touch bdd8b6c98239
> with new functions that probably have unknown and unintended
> consequences, so I think your approach is also better in that regard.
> As regards your patch, there is just a disagreement about how the
> i915 driver should behave if nopat is set. I agree the i915 driver
> could do a better job handling that case, at least with better error
> logs.
> 
> Chuck
> 
>>
>>> I respond by saying a well-written driver should refuse to honor
>>> the incorrect configuration requested by the user and instead
>>> warn the user that it did not honor the incorrect kernel option.
>>>
>>> I am only presuming what your patch would do on my box based
>>> on what I learned about this problem from my debugging. I can
>>> also test your patch on my box to verify that my understanding of
>>> it is correct.
>>>
>>> I also have not yet verified Juergen's patch will not fix it, but
>>> I am almost certain it will not unless it is expanded so it also
>>> touches i915_gem_object_pin_map() with the fix. I plan to test
>>> his patch, but expanded so it touches that function also.
>>>
>>> I also plan to test your patch with and without nopat and report the
>>> results in the thread where you posted your patch. Hopefully
>>> by tomorrow I will have the results.
>>>
>>> Chuck

Chuck Zmudzinski May 21, 2022, 1:24 p.m. UTC | #19

On 5/3/22 9:22 AM, Juergen Gross wrote:
> Some drivers are using pat_enabled() in order to test availability of
> special caching modes (WC and UC-). This will lead to false negatives
> in case the system was booted e.g. with the "nopat" variant and the
> BIOS did setup the PAT MSR supporting the queried mode, or if the
> system is running as a Xen PV guest.
>
> Add test functions for those caching modes instead and use them at the
> appropriate places.
>
> For symmetry reasons export the already existing x86_has_pat_wp() for
> modules, too.
>
> Fixes: bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with pat_enabled()")
> Fixes: ae749c7ab475 ("PCI: Add arch_can_pci_mmap_wc() macro")
> Signed-off-by: Juergen Gross<[e-mail]>
> ---
>   arch/x86/include/asm/memtype.h           |  2 ++
>   arch/x86/include/asm/pci.h               |  2 +-
>   arch/x86/mm/init.c                       | 25 +++++++++++++++++++++---
>   drivers/gpu/drm/i915/gem/i915_gem_mman.c |  8 ++++----
>   4 files changed, 29 insertions(+), 8 deletions(-)
>
> diff --git a/arch/x86/include/asm/memtype.h b/arch/x86/include/asm/memtype.h
> index 9ca760e430b9..d00e0be854d4 100644
> --- a/arch/x86/include/asm/memtype.h
> +++ b/arch/x86/include/asm/memtype.h
> @@ -25,6 +25,8 @@ extern void memtype_free_io(resource_size_t start, resource_size_t end);
>   extern bool pat_pfn_immune_to_uc_mtrr(unsigned long pfn);
>   
>   bool x86_has_pat_wp(void);
> +bool x86_has_pat_wc(void);
> +bool x86_has_pat_uc_minus(void);
>   enum page_cache_mode pgprot2cachemode(pgprot_t pgprot);
>   
>   #endif /* _ASM_X86_MEMTYPE_H */
> diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
> index f3fd5928bcbb..a5742268dec1 100644
> --- a/arch/x86/include/asm/pci.h
> +++ b/arch/x86/include/asm/pci.h
> @@ -94,7 +94,7 @@ int pcibios_set_irq_routing(struct pci_dev *dev, int pin, int irq);
>   
>   
>   #define HAVE_PCI_MMAP
> -#define arch_can_pci_mmap_wc()	pat_enabled()
> +#define arch_can_pci_mmap_wc()	x86_has_pat_wc()
>   #define ARCH_GENERIC_PCI_MMAP_RESOURCE
>   
>   #ifdef CONFIG_PCI
> diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
> index 71e182ebced3..b6431f714dc2 100644
> --- a/arch/x86/mm/init.c
> +++ b/arch/x86/mm/init.c
> @@ -77,12 +77,31 @@ static uint8_t __pte2cachemode_tbl[8] = {
>   	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
>   };
>   
> -/* Check that the write-protect PAT entry is set for write-protect */
> +static bool x86_has_pat_mode(unsigned int mode)
> +{
> +	return __pte2cachemode_tbl[__cachemode2pte_tbl[mode]] == mode;
> +}
> +
> +/* Check that PAT supports write-protect */
>   bool x86_has_pat_wp(void)
>   {
> -	return __pte2cachemode_tbl[__cachemode2pte_tbl[_PAGE_CACHE_MODE_WP]] ==
> -	       _PAGE_CACHE_MODE_WP;
> +	return x86_has_pat_mode(_PAGE_CACHE_MODE_WP);
> +}
> +EXPORT_SYMBOL_GPL(x86_has_pat_wp);
> +
> +/* Check that PAT supports WC */
> +bool x86_has_pat_wc(void)
> +{
> +	return x86_has_pat_mode(_PAGE_CACHE_MODE_WC);
> +}
> +EXPORT_SYMBOL_GPL(x86_has_pat_wc);
> +
> +/* Check that PAT supports UC- */
> +bool x86_has_pat_uc_minus(void)
> +{
> +	return x86_has_pat_mode(_PAGE_CACHE_MODE_UC_MINUS);
>   }
> +EXPORT_SYMBOL_GPL(x86_has_pat_uc_minus);
>   
>   enum page_cache_mode pgprot2cachemode(pgprot_t pgprot)
>   {
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> index 0c5c43852e24..f43ecf3f63eb 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
> @@ -76,7 +76,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
>   	if (args->flags & ~(I915_MMAP_WC))
>   		return -EINVAL;
>   
> -	if (args->flags & I915_MMAP_WC && !pat_enabled())
> +	if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
>   		return -ENODEV;
>   
>   	obj = i915_gem_object_lookup(file, args->handle);
> @@ -757,7 +757,7 @@ i915_gem_dumb_mmap_offset(struct drm_file *file,
>   
>   	if (HAS_LMEM(to_i915(dev)))
>   		mmap_type = I915_MMAP_TYPE_FIXED;
> -	else if (pat_enabled())
> +	else if (x86_has_pat_wc())
>   		mmap_type = I915_MMAP_TYPE_WC;
>   	else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
>   		return -ENODEV;
> @@ -813,7 +813,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>   		break;
>   
>   	case I915_MMAP_OFFSET_WC:
> -		if (!pat_enabled())
> +		if (!x86_has_pat_wc())
>   			return -ENODEV;
>   		type = I915_MMAP_TYPE_WC;
>   		break;
> @@ -823,7 +823,7 @@ i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
>   		break;
>   
>   	case I915_MMAP_OFFSET_UC:
> -		if (!pat_enabled())
> +		if (!x86_has_pat_uc_minus())
>   			return -ENODEV;
>   		type = I915_MMAP_TYPE_UC;
>   		break;

This patch is advertised as a fix for
bdd8b6c98239 ("drm/i915: replace X86_FEATURE_PAT with pat_enabled()")

bdd8b6c98239 causes a serious regression on my system when
running Linux as a Dom0 on Xen.

The regression is that on my system, the error caused by this issue
causes the i915driver to call its add_taint_for_CI function, which
in turn totally halts the system during early boot. So this makes
it impossible for either 5.17.y or the 5.18-rc versions to run
as a Dom0 on my system. I cannot upgrade my system to the 5.17.y
or to 5.18-rc versions without a proper fix for bdd8b6c98239.

I did some testing with this patch on my system (my tests included
the first patch of this 2-patch series), and here are the results:

This patch does *not* fix it. I expected this patch, as is, to not
fix it but allow me to add a simple patch that uses the new
x86_has_pat_wc() function provided by this patch to the
i915_gem_object_pin_map() function in i915_gem_pages.c
that would fix it.

However, even by adding the following simple patch to the
i915_gem_object_pin_map() function to the patch, the
patch series still does *not* fix the regression on my system:

--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -428,7 +428,7 @@
              goto err_unpin;
          }

-        if (GEM_WARN_ON(type == I915_MAP_WC && !pat_enabled()))
+        if (GEM_WARN_ON(type == I915_MAP_WC && !x86_has_pat_wc()))
              ptr = ERR_PTR(-ENODEV);
          else if (i915_gem_object_has_struct_page(obj))
              ptr = i915_gem_object_map_page(obj, type);

I verified that this is the function where pat_enabled() is returning
a false negative on my system.

This means x86_has_pat_wc() is still giving me a false negative, even
when running as a Xen Dom0. I am not sure you understand what is
really causing the problem Jan is trying to fix here with false
negatives from pat_enabled(). I also tested Jan's patch that
you are trying to replace with this patch, and his patch *does* fix
the problem on my system. Jan's patch is very simple and solves the
problem by editing pat_enabled() so that it returns true if
boot_cpu_has(X86_FEATURE_HYPERVISOR)) is true after the
other checks for the x86 pat feature failed.

I expect you do not have a system that actually has the problem
that Jan and I are trying to fix because the problem only exists on
systems with specific hardware, and in my case it is an Intel Haswell
CPU with integrated GPU. You might be able to test your patch,
though, if you boot the patched kernel with the nopat option and
check if your new functions return false when running on the bare
metal and true when running in a Dom0 on the Xen hypervisor. That is
what the new functions should do. I think you were expecting your
new x86_has_pat_wc() function to return true when Linux is running
as a Dom0 on the Xen hypervisor even when pat_enabled() returns
false. But it does not seem to be working.

In any case, after testing this patch, I cannot confirm that it
fixes bdd8b6c98239.

Best regards,

Chuck

Chuck Zmudzinski May 24, 2022, 6:32 p.m. UTC | #20

On 5/21/22 6:47 AM, Thorsten Leemhuis wrote:
> On 20.05.22 16:48, Chuck Zmudzinski wrote:
>> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>>
>>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>>> those want
>>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did
>>>>>>>>>>> inspect them
>>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>>> observe the
>>>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled()
>>>>>>>>>>> as the
>>>>>>>>>>> only predicate). In fact, as said in the description of my
>>>>>>>>>>> earlier
>>>>>>>>>>> patch, in
>>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map()
>>>>>>>>>>> to be
>>>>>>>>>>> the
>>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>>
>>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at
>>>>>>>>>> least
>>>>>>>>>> the
>>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>>
>>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>>> should not override that,
>>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>>> an override would affect only the single domain where the
>>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>>
>>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>>> (but tell us "don't do that then").
>>>>>> Actually I just did a test with the last official Debian kernel
>>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>>> That means your presumption (and the presumption of
>>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>>> nopat did not break the i915 driver. Why break it now?
>>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>>> (not breaking the driver of course - that's a driver bug -, but
>>>>> having an effect on the driver).
>>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>>> i915 driver
>>> The driver ought to work fine without PAT (and hence without being
>>> able to make WC mappings). It would use UC instead and be slow, but
>>> it ought to work.
>>>
>>>> and therefore the driver should refuse to disable
>>>> it if the user requests to disable it and instead warn the user that
>>>> the driver did not disable the feature, contrary to what the user
>>>> requested with the nopat option.
>>>>
>>>> In any case, my test did not verify that when nopat is set in Linux
>>>> 5.16,
>>>> the thread takes the same code path as when nopat is not set,
>>>> so I am not totally sure that the reason nopat does not break the
>>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>>> returns true even when nopat is set. I could test it with a custom
>>>> log message in 5.16 if that is necessary.
>>>>
>>>> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
>>>> to return true in 5.16 when the user requests nopat?
>>> No, I'm not saying that. It was wrong for this construct to be used
>>> in the driver, which was fixed for 5.17 (and which had caused the
>>> regression I did observe, leading to the patch as a hopefully least
>>> bad option).
>>>
>>>> I think that is
>>>> just permitting a bad configuration to break the driver that a
>>>> well-written operating system should not allow. The i915 driver
>>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>>> because that option is not compatible with the hardware the
>>>> i915 driver is trying to initialize and setup at boot time. At least
>>>> that is my understanding now, but I will need to test it on 5.16
>>>> to be sure I understand it correctly.
>>>>
>>>> Also, AFAICT, your patch would break the driver when the nopat
>>>> option is set and only fix the regression introduced by bdd8b6c98239
>>>> when nopat is not set on my box, so your patch would
>>>> introduce a regression relative to Linux 5.16 and earlier for the
>>>> case when nopat is set on my box. I think your point would
>>>> be that it is not a regression if it is an incorrect user configuration.
>>> Again no - my view is that there's a separate, pre-existing issue
>>> in the driver which was uncovered by the change. This may be a
>>> perceived regression, but is imo different from a real one.
> Sorry, for you maybe, but I'm pretty sure for Linus it's not when it
> comes to the "no regressions rule". Just took a quick look at quotes
> from Linus
> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
> and found this statement from Linus to back this up:
>
> ```
> One _particularly_ last-minute revert is the top-most commit (ignoring
> the version change itself) done just before the release, and while
> it's very annoying, it's perhaps also instructive.
>
> What's instructive about it is that I reverted a commit that wasn't
> actually buggy. In fact, it was doing exactly what it set out to do,
> and did it very well. In fact it did it _so_ well that the much
> improved IO patterns it caused then ended up revealing a user-visible
> regression due to a real bug in a completely unrelated area.
> ```
>
> He said that here:
> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>
> The situation is of course different here, but similar enough.
>
>> Since it is a regression, I think for now bdd8b6c98239 should
>> be reverted and the fix backported to Linux 5.17 stable until
>> the underlying memory subsystem can provide the i915 driver
>> with an updated test for the PAT feature that also meets the
>> requirements of the author of bdd8b6c98239 without breaking
>> the i915 driver.
> I'm not a developer and I'm don't known the details of this thread and
> the backstory of the regression, but it sounds like that's the approach
> that is needed here until someone comes up with a fix for the regression
> exposed by bdd8b6c98239.
>
> But if I'm wrong, please tell me.

You are mostly right, I think. Reverting bdd8b6c98239 fixes
it. There is another way to fix it, though. The patch proposed
by Jan Beulich also fixes the regression on my system, so as
the person reporting this is a regression, I would also be satisfied
with Jan's patch instead of reverting bdd8b6c98239 as a fix. Jan
posted his proposed patch here:

https://lore.kernel.org/lkml/[e-mail]/

The only reservation I have about Jan's patch is that the commit
message does not clearly explain how the patch changes what
the nopat kernel boot option does. It doesn't affect me because
I don't use nopat, but it should probably be mentioned in the
commit message, as pointed out here:

https://lore.kernel.org/lkml/[e-mail]/

Whatever fix for the regression exposed by bdd8b6c98239 also
needs to be backported to the stable versions 5.17 and 5.18.

Regards,

Chuck Zmudzinski
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>
> P.S.: As the Linux kernel's regression tracker I deal with a lot of
> reports and sometimes miss something important when writing mails like
> this. If that's the case here, don't hesitate to tell me in a public
> reply, it's in everyone's interest to set the public record straight.
>
>> The i915 driver relies on the memory subsytem
>> to provide it with an accurate test for the existence of
>> X86_FEATURE_PAT. I think your patch provides that more accurate
>> test so that bdd8b6c98239 could be re-applied when your patch is
>> committed. Juergen's patch would have to touch bdd8b6c98239
>> with new functions that probably have unknown and unintended
>> consequences, so I think your approach is also better in that regard.
>> As regards your patch, there is just a disagreement about how the
>> i915 driver should behave if nopat is set. I agree the i915 driver
>> could do a better job handling that case, at least with better error
>> logs.
>>
>> Chuck
>>
>>>> I respond by saying a well-written driver should refuse to honor
>>>> the incorrect configuration requested by the user and instead
>>>> warn the user that it did not honor the incorrect kernel option.
>>>>
>>>> I am only presuming what your patch would do on my box based
>>>> on what I learned about this problem from my debugging. I can
>>>> also test your patch on my box to verify that my understanding of
>>>> it is correct.
>>>>
>>>> I also have not yet verified Juergen's patch will not fix it, but
>>>> I am almost certain it will not unless it is expanded so it also
>>>> touches i915_gem_object_pin_map() with the fix. I plan to test
>>>> his patch, but expanded so it touches that function also.
>>>>
>>>> I also plan to test your patch with and without nopat and report the
>>>> results in the thread where you posted your patch. Hopefully
>>>> by tomorrow I will have the results.
>>>>
>>>> Chuck

Thorsten Leemhuis May 25, 2022, 7:45 a.m. UTC | #21

On 24.05.22 20:32, Chuck Zmudzinski wrote:
> On 5/21/22 6:47 AM, Thorsten Leemhuis wrote:
>> On 20.05.22 16:48, Chuck Zmudzinski wrote:
>>> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>>>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>>>> those want
>>>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did
>>>>>>>>>>>> inspect them
>>>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>>>> observe the
>>>>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled()
>>>>>>>>>>>> as the
>>>>>>>>>>>> only predicate). In fact, as said in the description of my
>>>>>>>>>>>> earlier
>>>>>>>>>>>> patch, in
>>>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map()
>>>>>>>>>>>> to be
>>>>>>>>>>>> the
>>>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>>>
>>>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at
>>>>>>>>>>> least
>>>>>>>>>>> the
>>>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>>>
>>>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>>>> should not override that,
>>>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>>>> an override would affect only the single domain where the
>>>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>>>
>>>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>>>> (but tell us "don't do that then").
>>>>>>> Actually I just did a test with the last official Debian kernel
>>>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>>>> That means your presumption (and the presumption of
>>>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>>>> nopat did not break the i915 driver. Why break it now?
>>>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>>>> (not breaking the driver of course - that's a driver bug -, but
>>>>>> having an effect on the driver).
>>>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>>>> i915 driver
>>>> The driver ought to work fine without PAT (and hence without being
>>>> able to make WC mappings). It would use UC instead and be slow, but
>>>> it ought to work.
>>>>
>>>>> and therefore the driver should refuse to disable
>>>>> it if the user requests to disable it and instead warn the user that
>>>>> the driver did not disable the feature, contrary to what the user
>>>>> requested with the nopat option.
>>>>>
>>>>> In any case, my test did not verify that when nopat is set in Linux
>>>>> 5.16,
>>>>> the thread takes the same code path as when nopat is not set,
>>>>> so I am not totally sure that the reason nopat does not break the
>>>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>>>> returns true even when nopat is set. I could test it with a custom
>>>>> log message in 5.16 if that is necessary.
>>>>>
>>>>> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
>>>>> to return true in 5.16 when the user requests nopat?
>>>> No, I'm not saying that. It was wrong for this construct to be used
>>>> in the driver, which was fixed for 5.17 (and which had caused the
>>>> regression I did observe, leading to the patch as a hopefully least
>>>> bad option).
>>>>
>>>>> I think that is
>>>>> just permitting a bad configuration to break the driver that a
>>>>> well-written operating system should not allow. The i915 driver
>>>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>>>> because that option is not compatible with the hardware the
>>>>> i915 driver is trying to initialize and setup at boot time. At least
>>>>> that is my understanding now, but I will need to test it on 5.16
>>>>> to be sure I understand it correctly.
>>>>>
>>>>> Also, AFAICT, your patch would break the driver when the nopat
>>>>> option is set and only fix the regression introduced by bdd8b6c98239
>>>>> when nopat is not set on my box, so your patch would
>>>>> introduce a regression relative to Linux 5.16 and earlier for the
>>>>> case when nopat is set on my box. I think your point would
>>>>> be that it is not a regression if it is an incorrect user
>>>>> configuration.
>>>> Again no - my view is that there's a separate, pre-existing issue
>>>> in the driver which was uncovered by the change. This may be a
>>>> perceived regression, but is imo different from a real one.
>> Sorry, for you maybe, but I'm pretty sure for Linus it's not when it
>> comes to the "no regressions rule". Just took a quick look at quotes
>> from Linus
>> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>> and found this statement from Linus to back this up:
>>
>> ```
>> One _particularly_ last-minute revert is the top-most commit (ignoring
>> the version change itself) done just before the release, and while
>> it's very annoying, it's perhaps also instructive.
>>
>> What's instructive about it is that I reverted a commit that wasn't
>> actually buggy. In fact, it was doing exactly what it set out to do,
>> and did it very well. In fact it did it _so_ well that the much
>> improved IO patterns it caused then ended up revealing a user-visible
>> regression due to a real bug in a completely unrelated area.
>> ```
>>
>> He said that here:
>> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>>
>> The situation is of course different here, but similar enough.
>>
>>> Since it is a regression, I think for now bdd8b6c98239 should
>>> be reverted and the fix backported to Linux 5.17 stable until
>>> the underlying memory subsystem can provide the i915 driver
>>> with an updated test for the PAT feature that also meets the
>>> requirements of the author of bdd8b6c98239 without breaking
>>> the i915 driver.
>> I'm not a developer and I'm don't known the details of this thread and
>> the backstory of the regression, but it sounds like that's the approach
>> that is needed here until someone comes up with a fix for the regression
>> exposed by bdd8b6c98239.
>>
>> But if I'm wrong, please tell me.
> 
> You are mostly right, I think. Reverting bdd8b6c98239 fixes
> it. There is another way to fix it, though.

Yeah, I'm aware of it. But it seems...

> The patch proposed
> by Jan Beulich also fixes the regression on my system, so as
> the person reporting this is a regression, I would also be satisfied
> with Jan's patch instead of reverting bdd8b6c98239 as a fix. Jan
> posted his proposed patch here:
> 
> https://lore.kernel.org/lkml/[e-mail]/

...that approach is not making any progress either?

Jan, can could provide a short status update here? I'd really like to
get this regression fixed one way or another rather sooner than later,
as this is taken way to long already IMHO.

> The only reservation I have about Jan's patch is that the commit
> message does not clearly explain how the patch changes what
> the nopat kernel boot option does. It doesn't affect me because
> I don't use nopat, but it should probably be mentioned in the
> commit message, as pointed out here:
> 
> https://lore.kernel.org/lkml/[e-mail]/
> 
> 
> Whatever fix for the regression exposed by bdd8b6c98239 also
> needs to be backported to the stable versions 5.17 and 5.18.

Sure.

BTW, as you seem to be familiar with the issue: there is another report
about a regression WRT to Xen and i915 (that is also not making really
progress):
https://lore.kernel.org/lkml/Yn%2FTgj1Ehs%2FBdpHp@itl-email/

It's just a wild guess, but bould this somehow be related?

Ciao, Thorsten

>>> The i915 driver relies on the memory subsytem
>>> to provide it with an accurate test for the existence of
>>> X86_FEATURE_PAT. I think your patch provides that more accurate
>>> test so that bdd8b6c98239 could be re-applied when your patch is
>>> committed. Juergen's patch would have to touch bdd8b6c98239
>>> with new functions that probably have unknown and unintended
>>> consequences, so I think your approach is also better in that regard.
>>> As regards your patch, there is just a disagreement about how the
>>> i915 driver should behave if nopat is set. I agree the i915 driver
>>> could do a better job handling that case, at least with better error
>>> logs.
>>>
>>> Chuck
>>>
>>>>> I respond by saying a well-written driver should refuse to honor
>>>>> the incorrect configuration requested by the user and instead
>>>>> warn the user that it did not honor the incorrect kernel option.
>>>>>
>>>>> I am only presuming what your patch would do on my box based
>>>>> on what I learned about this problem from my debugging. I can
>>>>> also test your patch on my box to verify that my understanding of
>>>>> it is correct.
>>>>>
>>>>> I also have not yet verified Juergen's patch will not fix it, but
>>>>> I am almost certain it will not unless it is expanded so it also
>>>>> touches i915_gem_object_pin_map() with the fix. I plan to test
>>>>> his patch, but expanded so it touches that function also.
>>>>>
>>>>> I also plan to test your patch with and without nopat and report the
>>>>> results in the thread where you posted your patch. Hopefully
>>>>> by tomorrow I will have the results.
>>>>>
>>>>> Chuck
> 
>

Jürgen Groß May 25, 2022, 8:04 a.m. UTC | #22

On 25.05.22 09:45, Thorsten Leemhuis wrote:
> 
> 
> On 24.05.22 20:32, Chuck Zmudzinski wrote:
>> On 5/21/22 6:47 AM, Thorsten Leemhuis wrote:
>>> On 20.05.22 16:48, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>>>>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>>>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>>>>> those want
>>>>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did
>>>>>>>>>>>>> inspect them
>>>>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>>>>> observe the
>>>>>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled()
>>>>>>>>>>>>> as the
>>>>>>>>>>>>> only predicate). In fact, as said in the description of my
>>>>>>>>>>>>> earlier
>>>>>>>>>>>>> patch, in
>>>>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map()
>>>>>>>>>>>>> to be
>>>>>>>>>>>>> the
>>>>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>>>>
>>>>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at
>>>>>>>>>>>> least
>>>>>>>>>>>> the
>>>>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>>>>
>>>>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>>>>> should not override that,
>>>>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>>>>> an override would affect only the single domain where the
>>>>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>>>>
>>>>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>>>>> (but tell us "don't do that then").
>>>>>>>> Actually I just did a test with the last official Debian kernel
>>>>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>>>>> That means your presumption (and the presumption of
>>>>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>>>>> nopat did not break the i915 driver. Why break it now?
>>>>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>>>>> (not breaking the driver of course - that's a driver bug -, but
>>>>>>> having an effect on the driver).
>>>>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>>>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>>>>> i915 driver
>>>>> The driver ought to work fine without PAT (and hence without being
>>>>> able to make WC mappings). It would use UC instead and be slow, but
>>>>> it ought to work.
>>>>>
>>>>>> and therefore the driver should refuse to disable
>>>>>> it if the user requests to disable it and instead warn the user that
>>>>>> the driver did not disable the feature, contrary to what the user
>>>>>> requested with the nopat option.
>>>>>>
>>>>>> In any case, my test did not verify that when nopat is set in Linux
>>>>>> 5.16,
>>>>>> the thread takes the same code path as when nopat is not set,
>>>>>> so I am not totally sure that the reason nopat does not break the
>>>>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>>>>> returns true even when nopat is set. I could test it with a custom
>>>>>> log message in 5.16 if that is necessary.
>>>>>>
>>>>>> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
>>>>>> to return true in 5.16 when the user requests nopat?
>>>>> No, I'm not saying that. It was wrong for this construct to be used
>>>>> in the driver, which was fixed for 5.17 (and which had caused the
>>>>> regression I did observe, leading to the patch as a hopefully least
>>>>> bad option).
>>>>>
>>>>>> I think that is
>>>>>> just permitting a bad configuration to break the driver that a
>>>>>> well-written operating system should not allow. The i915 driver
>>>>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>>>>> because that option is not compatible with the hardware the
>>>>>> i915 driver is trying to initialize and setup at boot time. At least
>>>>>> that is my understanding now, but I will need to test it on 5.16
>>>>>> to be sure I understand it correctly.
>>>>>>
>>>>>> Also, AFAICT, your patch would break the driver when the nopat
>>>>>> option is set and only fix the regression introduced by bdd8b6c98239
>>>>>> when nopat is not set on my box, so your patch would
>>>>>> introduce a regression relative to Linux 5.16 and earlier for the
>>>>>> case when nopat is set on my box. I think your point would
>>>>>> be that it is not a regression if it is an incorrect user
>>>>>> configuration.
>>>>> Again no - my view is that there's a separate, pre-existing issue
>>>>> in the driver which was uncovered by the change. This may be a
>>>>> perceived regression, but is imo different from a real one.
>>> Sorry, for you maybe, but I'm pretty sure for Linus it's not when it
>>> comes to the "no regressions rule". Just took a quick look at quotes
>>> from Linus
>>> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>>> and found this statement from Linus to back this up:
>>>
>>> ```
>>> One _particularly_ last-minute revert is the top-most commit (ignoring
>>> the version change itself) done just before the release, and while
>>> it's very annoying, it's perhaps also instructive.
>>>
>>> What's instructive about it is that I reverted a commit that wasn't
>>> actually buggy. In fact, it was doing exactly what it set out to do,
>>> and did it very well. In fact it did it _so_ well that the much
>>> improved IO patterns it caused then ended up revealing a user-visible
>>> regression due to a real bug in a completely unrelated area.
>>> ```
>>>
>>> He said that here:
>>> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>>>
>>> The situation is of course different here, but similar enough.
>>>
>>>> Since it is a regression, I think for now bdd8b6c98239 should
>>>> be reverted and the fix backported to Linux 5.17 stable until
>>>> the underlying memory subsystem can provide the i915 driver
>>>> with an updated test for the PAT feature that also meets the
>>>> requirements of the author of bdd8b6c98239 without breaking
>>>> the i915 driver.
>>> I'm not a developer and I'm don't known the details of this thread and
>>> the backstory of the regression, but it sounds like that's the approach
>>> that is needed here until someone comes up with a fix for the regression
>>> exposed by bdd8b6c98239.
>>>
>>> But if I'm wrong, please tell me.
>>
>> You are mostly right, I think. Reverting bdd8b6c98239 fixes
>> it. There is another way to fix it, though.
> 
> Yeah, I'm aware of it. But it seems...
> 
>> The patch proposed
>> by Jan Beulich also fixes the regression on my system, so as
>> the person reporting this is a regression, I would also be satisfied
>> with Jan's patch instead of reverting bdd8b6c98239 as a fix. Jan
>> posted his proposed patch here:
>>
>> https://lore.kernel.org/lkml/[e-mail]/
> 
> ...that approach is not making any progress either?
> 
> Jan, can could provide a short status update here? I'd really like to
> get this regression fixed one way or another rather sooner than later,
> as this is taken way to long already IMHO.
> 
>> The only reservation I have about Jan's patch is that the commit
>> message does not clearly explain how the patch changes what
>> the nopat kernel boot option does. It doesn't affect me because
>> I don't use nopat, but it should probably be mentioned in the
>> commit message, as pointed out here:
>>
>> https://lore.kernel.org/lkml/[e-mail]/
>>
>>
>> Whatever fix for the regression exposed by bdd8b6c98239 also
>> needs to be backported to the stable versions 5.17 and 5.18.
> 
> Sure.
> 
> BTW, as you seem to be familiar with the issue: there is another report
> about a regression WRT to Xen and i915 (that is also not making really
> progress):
> https://lore.kernel.org/lkml/Yn%2FTgj1Ehs%2FBdpHp@itl-email/
> 
> It's just a wild guess, but bould this somehow be related?

No, doesn't seem so.

I'm just reviewing the suggested fix:

https://lore.kernel.org/lkml/Yo0LwmVUDSBZb44K@itl-email/


Juergen

Jan Beulich May 25, 2022, 8:37 a.m. UTC | #23

On 25.05.2022 09:45, Thorsten Leemhuis wrote:
> On 24.05.22 20:32, Chuck Zmudzinski wrote:
>> On 5/21/22 6:47 AM, Thorsten Leemhuis wrote:
>>> I'm not a developer and I'm don't known the details of this thread and
>>> the backstory of the regression, but it sounds like that's the approach
>>> that is needed here until someone comes up with a fix for the regression
>>> exposed by bdd8b6c98239.
>>>
>>> But if I'm wrong, please tell me.
>>
>> You are mostly right, I think. Reverting bdd8b6c98239 fixes
>> it. There is another way to fix it, though.
> 
> Yeah, I'm aware of it. But it seems...
> 
>> The patch proposed
>> by Jan Beulich also fixes the regression on my system, so as
>> the person reporting this is a regression, I would also be satisfied
>> with Jan's patch instead of reverting bdd8b6c98239 as a fix. Jan
>> posted his proposed patch here:
>>
>> https://lore.kernel.org/lkml/[e-mail]/
> 
> ...that approach is not making any progress either?
> 
> Jan, can could provide a short status update here? I'd really like to
> get this regression fixed one way or another rather sooner than later,
> as this is taken way to long already IMHO.

What kind of status update could I provide? I've not heard back from
anyone of the maintainers, so I have no way to know what (if anything)
I need to do.

Jan

Thorsten Leemhuis May 25, 2022, 8:51 a.m. UTC | #24

On 25.05.22 10:37, Jan Beulich wrote:
> On 25.05.2022 09:45, Thorsten Leemhuis wrote:
>> On 24.05.22 20:32, Chuck Zmudzinski wrote:
>>> On 5/21/22 6:47 AM, Thorsten Leemhuis wrote:
>>>> I'm not a developer and I'm don't known the details of this thread and
>>>> the backstory of the regression, but it sounds like that's the approach
>>>> that is needed here until someone comes up with a fix for the regression
>>>> exposed by bdd8b6c98239.
>>>>
>>>> But if I'm wrong, please tell me.
>>>
>>> You are mostly right, I think. Reverting bdd8b6c98239 fixes
>>> it. There is another way to fix it, though.
>>
>> Yeah, I'm aware of it. But it seems...
>>
>>> The patch proposed
>>> by Jan Beulich also fixes the regression on my system, so as
>>> the person reporting this is a regression, I would also be satisfied
>>> with Jan's patch instead of reverting bdd8b6c98239 as a fix. Jan
>>> posted his proposed patch here:
>>>
>>> https://lore.kernel.org/lkml/[e-mail]/
>>
>> ...that approach is not making any progress either?
>>
>> Jan, can could provide a short status update here? I'd really like to
>> get this regression fixed one way or another rather sooner than later,
>> as this is taken way to long already IMHO.
> 
> What kind of status update could I provide? I've not heard back from
> anyone of the maintainers, so I have no way to know what (if anything)
> I need to do.

That is perfectly fine as a status update for me (I track a lot of
regression and it's easy to miss updated patches, discussion in other
places, and things like that).

Could you maybe send a reminder to the maintainer that this is a fix for
regression that is bothering people and needs to be handled with high
priority? Feel free to tell them the Linux kernel regression tracker is
pestering you because things are taken so long. :-D

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Chuck Zmudzinski May 25, 2022, 7:25 p.m. UTC | #25

On 5/25/2022 3:45 AM, Thorsten Leemhuis wrote:
> On 24.05.22 20:32, Chuck Zmudzinski wrote:
>> On 5/21/22 6:47 AM, Thorsten Leemhuis wrote:
>>> On 20.05.22 16:48, Chuck Zmudzinski wrote:
>>>> On 5/20/2022 10:06 AM, Jan Beulich wrote:
>>>>> On 20.05.2022 15:33, Chuck Zmudzinski wrote:
>>>>>> On 5/20/2022 5:41 AM, Jan Beulich wrote:
>>>>>>> On 20.05.2022 10:30, Chuck Zmudzinski wrote:
>>>>>>>> On 5/20/2022 2:59 AM, Chuck Zmudzinski wrote:
>>>>>>>>> On 5/20/2022 2:05 AM, Jan Beulich wrote:
>>>>>>>>>> On 20.05.2022 06:43, Chuck Zmudzinski wrote:
>>>>>>>>>>> On 5/4/22 5:14 AM, Juergen Gross wrote:
>>>>>>>>>>>> On 04.05.22 10:31, Jan Beulich wrote:
>>>>>>>>>>>>> On 03.05.2022 15:22, Juergen Gross wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ... these uses there are several more. You say nothing on why
>>>>>>>>>>>>> those want
>>>>>>>>>>>>> leaving unaltered. When preparing my earlier patch I did
>>>>>>>>>>>>> inspect them
>>>>>>>>>>>>> and came to the conclusion that these all would also better
>>>>>>>>>>>>> observe the
>>>>>>>>>>>>> adjusted behavior (or else I couldn't have left pat_enabled()
>>>>>>>>>>>>> as the
>>>>>>>>>>>>> only predicate). In fact, as said in the description of my
>>>>>>>>>>>>> earlier
>>>>>>>>>>>>> patch, in
>>>>>>>>>>>>> my debugging I did find the use in i915_gem_object_pin_map()
>>>>>>>>>>>>> to be
>>>>>>>>>>>>> the
>>>>>>>>>>>>> problematic one, which you leave alone.
>>>>>>>>>>>> Oh, I missed that one, sorry.
>>>>>>>>>>> That is why your patch would not fix my Haswell unless
>>>>>>>>>>> it also touches i915_gem_object_pin_map() in
>>>>>>>>>>> drivers/gpu/drm/i915/gem/i915_gem_pages.c
>>>>>>>>>>>
>>>>>>>>>>>> I wanted to be rather defensive in my changes, but I agree at
>>>>>>>>>>>> least
>>>>>>>>>>>> the
>>>>>>>>>>>> case in arch_phys_wc_add() might want to be changed, too.
>>>>>>>>>>> I think your approach needs to be more aggressive so it will fix
>>>>>>>>>>> all the known false negatives introduced by bdd8b6c98239
>>>>>>>>>>> such as the one in i915_gem_object_pin_map().
>>>>>>>>>>>
>>>>>>>>>>> I looked at Jan's approach and I think it would fix the issue
>>>>>>>>>>> with my Haswell as long as I don't use the nopat option. I
>>>>>>>>>>> really don't have a strong opinion on that question, but I
>>>>>>>>>>> think the nopat option as a Linux kernel option, as opposed
>>>>>>>>>>> to a hypervisor option, should only affect the kernel, and
>>>>>>>>>>> if the hypervisor provides the pat feature, then the kernel
>>>>>>>>>>> should not override that,
>>>>>>>>>> Hmm, why would the kernel not be allowed to override that? Such
>>>>>>>>>> an override would affect only the single domain where the
>>>>>>>>>> kernel runs; other domains could take their own decisions.
>>>>>>>>>>
>>>>>>>>>> Also, for the sake of completeness: "nopat" used when running on
>>>>>>>>>> bare metal has the same bad effect on system boot, so there
>>>>>>>>>> pretty clearly is an error cleanup issue in the i915 driver. But
>>>>>>>>>> that's orthogonal, and I expect the maintainers may not even care
>>>>>>>>>> (but tell us "don't do that then").
>>>>>>>> Actually I just did a test with the last official Debian kernel
>>>>>>>> build of Linux 5.16, that is, a kernel before bdd8b6c98239 was
>>>>>>>> applied. In fact, the nopat option does *not* break the i915 driver
>>>>>>>> in 5.16. That is, with the nopat option, the i915 driver loads
>>>>>>>> normally on both the bare metal and on the Xen hypervisor.
>>>>>>>> That means your presumption (and the presumption of
>>>>>>>> the author of bdd8b6c98239) that the "nopat" option was
>>>>>>>> being observed by the i915 driver is incorrect. Setting "nopat"
>>>>>>>> had no effect on my system with Linux 5.16. So after doing these
>>>>>>>> tests, I am against the aggressive approach of breaking the i915
>>>>>>>> driver with the "nopat" option because prior to bdd8b6c98239,
>>>>>>>> nopat did not break the i915 driver. Why break it now?
>>>>>>> Because that's, in my understanding, is the purpose of "nopat"
>>>>>>> (not breaking the driver of course - that's a driver bug -, but
>>>>>>> having an effect on the driver).
>>>>>> I wouldn't call it a driver bug, but an incorrect configuration of the
>>>>>> kernel by the user.  I presume X86_FEATURE_PAT is required by the
>>>>>> i915 driver
>>>>> The driver ought to work fine without PAT (and hence without being
>>>>> able to make WC mappings). It would use UC instead and be slow, but
>>>>> it ought to work.
>>>>>
>>>>>> and therefore the driver should refuse to disable
>>>>>> it if the user requests to disable it and instead warn the user that
>>>>>> the driver did not disable the feature, contrary to what the user
>>>>>> requested with the nopat option.
>>>>>>
>>>>>> In any case, my test did not verify that when nopat is set in Linux
>>>>>> 5.16,
>>>>>> the thread takes the same code path as when nopat is not set,
>>>>>> so I am not totally sure that the reason nopat does not break the
>>>>>> i915 driver in 5.16 is that static_cpu_has(X86_FEATURE_PAT)
>>>>>> returns true even when nopat is set. I could test it with a custom
>>>>>> log message in 5.16 if that is necessary.
>>>>>>
>>>>>> Are you saying it was wrong for static_cpu_has(X86_FEATURE_PAT)
>>>>>> to return true in 5.16 when the user requests nopat?
>>>>> No, I'm not saying that. It was wrong for this construct to be used
>>>>> in the driver, which was fixed for 5.17 (and which had caused the
>>>>> regression I did observe, leading to the patch as a hopefully least
>>>>> bad option).
>>>>>
>>>>>> I think that is
>>>>>> just permitting a bad configuration to break the driver that a
>>>>>> well-written operating system should not allow. The i915 driver
>>>>>> was, in my opinion, correctly ignoring the nopat option in 5.16
>>>>>> because that option is not compatible with the hardware the
>>>>>> i915 driver is trying to initialize and setup at boot time. At least
>>>>>> that is my understanding now, but I will need to test it on 5.16
>>>>>> to be sure I understand it correctly.
>>>>>>
>>>>>> Also, AFAICT, your patch would break the driver when the nopat
>>>>>> option is set and only fix the regression introduced by bdd8b6c98239
>>>>>> when nopat is not set on my box, so your patch would
>>>>>> introduce a regression relative to Linux 5.16 and earlier for the
>>>>>> case when nopat is set on my box. I think your point would
>>>>>> be that it is not a regression if it is an incorrect user
>>>>>> configuration.
>>>>> Again no - my view is that there's a separate, pre-existing issue
>>>>> in the driver which was uncovered by the change. This may be a
>>>>> perceived regression, but is imo different from a real one.
>>> Sorry, for you maybe, but I'm pretty sure for Linus it's not when it
>>> comes to the "no regressions rule". Just took a quick look at quotes
>>> from Linus
>>> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>>> and found this statement from Linus to back this up:
>>>
>>> ```
>>> One _particularly_ last-minute revert is the top-most commit (ignoring
>>> the version change itself) done just before the release, and while
>>> it's very annoying, it's perhaps also instructive.
>>>
>>> What's instructive about it is that I reverted a commit that wasn't
>>> actually buggy. In fact, it was doing exactly what it set out to do,
>>> and did it very well. In fact it did it _so_ well that the much
>>> improved IO patterns it caused then ended up revealing a user-visible
>>> regression due to a real bug in a completely unrelated area.
>>> ```
>>>
>>> He said that here:
>>> https://www.kernel.org/doc/html/latest/process/handling-regressions.html
>>>
>>> The situation is of course different here, but similar enough.
>>>
>>>> Since it is a regression, I think for now bdd8b6c98239 should
>>>> be reverted and the fix backported to Linux 5.17 stable until
>>>> the underlying memory subsystem can provide the i915 driver
>>>> with an updated test for the PAT feature that also meets the
>>>> requirements of the author of bdd8b6c98239 without breaking
>>>> the i915 driver.
>>> I'm not a developer and I'm don't known the details of this thread and
>>> the backstory of the regression, but it sounds like that's the approach
>>> that is needed here until someone comes up with a fix for the regression
>>> exposed by bdd8b6c98239.
>>>
>>> But if I'm wrong, please tell me.
>> You are mostly right, I think. Reverting bdd8b6c98239 fixes
>> it. There is another way to fix it, though.
> Yeah, I'm aware of it. But it seems...
>
>> The patch proposed
>> by Jan Beulich also fixes the regression on my system, so as
>> the person reporting this is a regression, I would also be satisfied
>> with Jan's patch instead of reverting bdd8b6c98239 as a fix. Jan
>> posted his proposed patch here:
>>
>> https://lore.kernel.org/lkml/[e-mail]/
> ...that approach is not making any progress either?

Jan's approach does fix it on my system. There was some debate
about what the kernel nopat option should do, though. I don't
have a strong opinion on that and would accept Jan's patch
as a fix.

>
> Jan, can could provide a short status update here? I'd really like to
> get this regression fixed one way or another rather sooner than later,
> as this is taken way to long already IMHO.

I hope something is done soon also.

>
>> The only reservation I have about Jan's patch is that the commit
>> message does not clearly explain how the patch changes what
>> the nopat kernel boot option does. It doesn't affect me because
>> I don't use nopat, but it should probably be mentioned in the
>> commit message, as pointed out here:
>>
>> https://lore.kernel.org/lkml/[e-mail]/
>>
>>
>> Whatever fix for the regression exposed by bdd8b6c98239 also
>> needs to be backported to the stable versions 5.17 and 5.18.
> Sure.
>
> BTW, as you seem to be familiar with the issue: there is another report
> about a regression WRT to Xen and i915 (that is also not making really
> progress):
> https://lore.kernel.org/lkml/Yn%2FTgj1Ehs%2FBdpHp@itl-email/
>
> It's just a wild guess, but bould this somehow be related?

It could be, but I do not run a GUI in my Xen Dom0, so I have not
seen that issue.

Best regards,

Chuck

Patch

diff --git a/arch/x86/include/asm/memtype.h b/arch/x86/include/asm/memtype.h
index 9ca760e430b9..d00e0be854d4 100644
--- a/arch/x86/include/asm/memtype.h
+++ b/arch/x86/include/asm/memtype.h
@@ -25,6 +25,8 @@  extern void memtype_free_io(resource_size_t start, resource_size_t end);
 extern bool pat_pfn_immune_to_uc_mtrr(unsigned long pfn);
 
 bool x86_has_pat_wp(void);
+bool x86_has_pat_wc(void);
+bool x86_has_pat_uc_minus(void);
 enum page_cache_mode pgprot2cachemode(pgprot_t pgprot);
 
 #endif /* _ASM_X86_MEMTYPE_H */
diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index f3fd5928bcbb..a5742268dec1 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -94,7 +94,7 @@  int pcibios_set_irq_routing(struct pci_dev *dev, int pin, int irq);
 
 
 #define HAVE_PCI_MMAP
-#define arch_can_pci_mmap_wc()	pat_enabled()
+#define arch_can_pci_mmap_wc()	x86_has_pat_wc()
 #define ARCH_GENERIC_PCI_MMAP_RESOURCE
 
 #ifdef CONFIG_PCI
diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index 71e182ebced3..b6431f714dc2 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -77,12 +77,31 @@  static uint8_t __pte2cachemode_tbl[8] = {
 	[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
 };
 
-/* Check that the write-protect PAT entry is set for write-protect */
+static bool x86_has_pat_mode(unsigned int mode)
+{
+	return __pte2cachemode_tbl[__cachemode2pte_tbl[mode]] == mode;
+}
+
+/* Check that PAT supports write-protect */
 bool x86_has_pat_wp(void)
 {
-	return __pte2cachemode_tbl[__cachemode2pte_tbl[_PAGE_CACHE_MODE_WP]] ==
-	       _PAGE_CACHE_MODE_WP;
+	return x86_has_pat_mode(_PAGE_CACHE_MODE_WP);
+}
+EXPORT_SYMBOL_GPL(x86_has_pat_wp);
+
+/* Check that PAT supports WC */
+bool x86_has_pat_wc(void)
+{
+	return x86_has_pat_mode(_PAGE_CACHE_MODE_WC);
+}
+EXPORT_SYMBOL_GPL(x86_has_pat_wc);
+
+/* Check that PAT supports UC- */
+bool x86_has_pat_uc_minus(void)
+{
+	return x86_has_pat_mode(_PAGE_CACHE_MODE_UC_MINUS);
 }
+EXPORT_SYMBOL_GPL(x86_has_pat_uc_minus);
 
 enum page_cache_mode pgprot2cachemode(pgprot_t pgprot)
 {
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
index 0c5c43852e24..f43ecf3f63eb 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c
@@ -76,7 +76,7 @@  i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
 	if (args->flags & ~(I915_MMAP_WC))
 		return -EINVAL;
 
-	if (args->flags & I915_MMAP_WC && !pat_enabled())
+	if (args->flags & I915_MMAP_WC && !x86_has_pat_wc())
 		return -ENODEV;
 
 	obj = i915_gem_object_lookup(file, args->handle);
@@ -757,7 +757,7 @@  i915_gem_dumb_mmap_offset(struct drm_file *file,
 
 	if (HAS_LMEM(to_i915(dev)))
 		mmap_type = I915_MMAP_TYPE_FIXED;
-	else if (pat_enabled())
+	else if (x86_has_pat_wc())
 		mmap_type = I915_MMAP_TYPE_WC;
 	else if (!i915_ggtt_has_aperture(to_gt(i915)->ggtt))
 		return -ENODEV;
@@ -813,7 +813,7 @@  i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
 		break;
 
 	case I915_MMAP_OFFSET_WC:
-		if (!pat_enabled())
+		if (!x86_has_pat_wc())
 			return -ENODEV;
 		type = I915_MMAP_TYPE_WC;
 		break;
@@ -823,7 +823,7 @@  i915_gem_mmap_offset_ioctl(struct drm_device *dev, void *data,
 		break;
 
 	case I915_MMAP_OFFSET_UC:
-		if (!pat_enabled())
+		if (!x86_has_pat_uc_minus())
 			return -ENODEV;
 		type = I915_MMAP_TYPE_UC;
 		break;