[1/2] PCI: also set up legacy files only after sysfs init

Commit Message

Daniel Vetter Feb. 4, 2021, 4:58 p.m. UTC

We are already doing this for all the regular sysfs files on PCI
devices, but not yet on the legacy io files on the PCI buses. Thus far
now problem, but in the next patch I want to wire up iomem revoke
support. That needs the vfs up an running already to make so that
iomem_get_mapping() works.

Wire it up exactly like the existing code. Note that
pci_remove_legacy_files() doesn't need a check since the one for
pci_bus->legacy_io is sufficient.

Signed-off-by: Daniel Vetter <[e-mail]>
Cc: Stephen Rothwell <[e-mail]>
Cc: Jason Gunthorpe <[e-mail]>
Cc: Kees Cook <[e-mail]>
Cc: Dan Williams <[e-mail]>
Cc: Andrew Morton <[e-mail]>
Cc: John Hubbard <[e-mail]>
Cc: Jérôme Glisse <[e-mail]>
Cc: Jan Kara <[e-mail]>
Cc: Dan Williams <[e-mail]>
Cc: Greg Kroah-Hartman <[e-mail]>
Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]
Cc: Bjorn Helgaas <[e-mail]>
Cc: [e-mail]
---
 drivers/pci/pci-sysfs.c | 7 +++++++
 1 file changed, 7 insertions(+)

Comments

Bjorn Helgaas Feb. 4, 2021, 9:50 p.m. UTC | #1

[+cc Oliver, Pali, Krzysztof]

s/also/Also/ in subject

On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> We are already doing this for all the regular sysfs files on PCI
> devices, but not yet on the legacy io files on the PCI buses. Thus far
> now problem, but in the next patch I want to wire up iomem revoke
> support. That needs the vfs up an running already to make so that
> iomem_get_mapping() works.

s/now problem/no problem/
s/an running/and running/
s/so that/sure that/ ?

iomem_get_mapping() doesn't exist; I don't know what that should be.

> Wire it up exactly like the existing code. Note that
> pci_remove_legacy_files() doesn't need a check since the one for
> pci_bus->legacy_io is sufficient.

I'm not sure exactly what you mean by "the existing code."  I could
probably figure it out, but it would save time to mention the existing
function here.

This looks like another instance where we should really apply Oliver's
idea of converting these to attribute_groups [1].

The cover letter mentions options discussed with Greg in [2], but I
don't think the "sysfs_initialized" hack vs attribute_groups was part
of that discussion.

It's not absolutely a show-stopper, but it *is* a shame to extend the
sysfs_initialized hack if attribute_groups could do this more cleanly
and help solve more than one issue.

Bjorn

[1] https://lore.kernel.org/r/[e-mail]
[2] https://lore.kernel.org/dri-devel/[e-mail]/

> Signed-off-by: Daniel Vetter <[e-mail]>
> Cc: Stephen Rothwell <[e-mail]>
> Cc: Jason Gunthorpe <[e-mail]>
> Cc: Kees Cook <[e-mail]>
> Cc: Dan Williams <[e-mail]>
> Cc: Andrew Morton <[e-mail]>
> Cc: John Hubbard <[e-mail]>
> Cc: Jérôme Glisse <[e-mail]>
> Cc: Jan Kara <[e-mail]>
> Cc: Dan Williams <[e-mail]>
> Cc: Greg Kroah-Hartman <[e-mail]>
> Cc: [e-mail]
> Cc: [e-mail]
> Cc: [e-mail]
> Cc: [e-mail]
> Cc: Bjorn Helgaas <[e-mail]>
> Cc: [e-mail]
> ---
>  drivers/pci/pci-sysfs.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> index fb072f4b3176..0c45b4f7b214 100644
> --- a/drivers/pci/pci-sysfs.c
> +++ b/drivers/pci/pci-sysfs.c
> @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
>  {
>  	int error;
>  
> +	if (!sysfs_initialized)
> +		return;
> +
>  	b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
>  			       GFP_ATOMIC);
>  	if (!b->legacy_io)
> @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
>  static int __init pci_sysfs_init(void)
>  {
>  	struct pci_dev *pdev = NULL;
> +	struct pci_bus *pbus = NULL;
>  	int retval;
>  
>  	sysfs_initialized = 1;
> @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
>  		}
>  	}
>  
> +	while ((pbus = pci_find_next_bus(pbus)))
> +		pci_create_legacy_files(pbus);
> +
>  	return 0;
>  }
>  late_initcall(pci_sysfs_init);
> -- 
> 2.30.0
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> [e-mail]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Pali Rohár Feb. 4, 2021, 10:24 p.m. UTC | #2

On Thursday 04 February 2021 15:50:19 Bjorn Helgaas wrote:
> [+cc Oliver, Pali, Krzysztof]

Just to note that extending or using sysfs_initialized introduces
another race condition into kernel code which results in PCI fatal
errors. Details are in email discussion which Bjorn already sent.

> s/also/Also/ in subject
> 
> On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> > We are already doing this for all the regular sysfs files on PCI
> > devices, but not yet on the legacy io files on the PCI buses. Thus far
> > now problem, but in the next patch I want to wire up iomem revoke
> > support. That needs the vfs up an running already to make so that
> > iomem_get_mapping() works.
> 
> s/now problem/no problem/
> s/an running/and running/
> s/so that/sure that/ ?
> 
> iomem_get_mapping() doesn't exist; I don't know what that should be.
> 
> > Wire it up exactly like the existing code. Note that
> > pci_remove_legacy_files() doesn't need a check since the one for
> > pci_bus->legacy_io is sufficient.
> 
> I'm not sure exactly what you mean by "the existing code."  I could
> probably figure it out, but it would save time to mention the existing
> function here.
> 
> This looks like another instance where we should really apply Oliver's
> idea of converting these to attribute_groups [1].
> 
> The cover letter mentions options discussed with Greg in [2], but I
> don't think the "sysfs_initialized" hack vs attribute_groups was part
> of that discussion.
> 
> It's not absolutely a show-stopper, but it *is* a shame to extend the
> sysfs_initialized hack if attribute_groups could do this more cleanly
> and help solve more than one issue.
> 
> Bjorn
> 
> [1] https://lore.kernel.org/r/[e-mail]
> [2] https://lore.kernel.org/dri-devel/[e-mail]/
> 
> > Signed-off-by: Daniel Vetter <[e-mail]>
> > Cc: Stephen Rothwell <[e-mail]>
> > Cc: Jason Gunthorpe <[e-mail]>
> > Cc: Kees Cook <[e-mail]>
> > Cc: Dan Williams <[e-mail]>
> > Cc: Andrew Morton <[e-mail]>
> > Cc: John Hubbard <[e-mail]>
> > Cc: Jérôme Glisse <[e-mail]>
> > Cc: Jan Kara <[e-mail]>
> > Cc: Dan Williams <[e-mail]>
> > Cc: Greg Kroah-Hartman <[e-mail]>
> > Cc: [e-mail]
> > Cc: [e-mail]
> > Cc: [e-mail]
> > Cc: [e-mail]
> > Cc: Bjorn Helgaas <[e-mail]>
> > Cc: [e-mail]
> > ---
> >  drivers/pci/pci-sysfs.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > index fb072f4b3176..0c45b4f7b214 100644
> > --- a/drivers/pci/pci-sysfs.c
> > +++ b/drivers/pci/pci-sysfs.c
> > @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
> >  {
> >  	int error;
> >  
> > +	if (!sysfs_initialized)
> > +		return;
> > +
> >  	b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
> >  			       GFP_ATOMIC);
> >  	if (!b->legacy_io)
> > @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
> >  static int __init pci_sysfs_init(void)
> >  {
> >  	struct pci_dev *pdev = NULL;
> > +	struct pci_bus *pbus = NULL;
> >  	int retval;
> >  
> >  	sysfs_initialized = 1;
> > @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
> >  		}
> >  	}
> >  
> > +	while ((pbus = pci_find_next_bus(pbus)))
> > +		pci_create_legacy_files(pbus);
> > +
> >  	return 0;
> >  }
> >  late_initcall(pci_sysfs_init);
> > -- 
> > 2.30.0
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > [e-mail]
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Daniel Vetter Feb. 5, 2021, 9:23 a.m. UTC | #3

On Thu, Feb 4, 2021 at 10:50 PM Bjorn Helgaas <[e-mail]> wrote:
>
> [+cc Oliver, Pali, Krzysztof]
>
> s/also/Also/ in subject
>
> On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> > We are already doing this for all the regular sysfs files on PCI
> > devices, but not yet on the legacy io files on the PCI buses. Thus far
> > now problem, but in the next patch I want to wire up iomem revoke
> > support. That needs the vfs up an running already to make so that
> > iomem_get_mapping() works.
>
> s/now problem/no problem/
> s/an running/and running/
> s/so that/sure that/ ?
>
> iomem_get_mapping() doesn't exist; I don't know what that should be.

Series is based on top of linux-next, where iomem_get_mapping exists.
This patch fixes the 2nd patch in this series, which I had to take out
of my branch because it failed.

> > Wire it up exactly like the existing code. Note that
> > pci_remove_legacy_files() doesn't need a check since the one for
> > pci_bus->legacy_io is sufficient.
>
> I'm not sure exactly what you mean by "the existing code."  I could
> probably figure it out, but it would save time to mention the existing
> function here.

Sorry, I meant the existing code in pci_create_sysfs_dev_files().

> This looks like another instance where we should really apply Oliver's
> idea of converting these to attribute_groups [1].
>
> The cover letter mentions options discussed with Greg in [2], but I
> don't think the "sysfs_initialized" hack vs attribute_groups was part
> of that discussion.

Hm not sure the attribute_groups works. The problem is that I cant set
up the attributes before the vfs layer is initialized, because before
that point the iomem_get_mapping function doesn't return anything
useful (well it crashes), because it needs to have an inode available.

So if you want to set up the attributes earlier, we'd need some kind
of callback, which Greg didn't like.

> It's not absolutely a show-stopper, but it *is* a shame to extend the
> sysfs_initialized hack if attribute_groups could do this more cleanly
> and help solve more than one issue.

So I think I have yet another init ordering problem here, but not sure.
-Daniel

>
> Bjorn
>
> [1] https://lore.kernel.org/r/[e-mail]
> [2] https://lore.kernel.org/dri-devel/[e-mail]/
>
> > Signed-off-by: Daniel Vetter <[e-mail]>
> > Cc: Stephen Rothwell <[e-mail]>
> > Cc: Jason Gunthorpe <[e-mail]>
> > Cc: Kees Cook <[e-mail]>
> > Cc: Dan Williams <[e-mail]>
> > Cc: Andrew Morton <[e-mail]>
> > Cc: John Hubbard <[e-mail]>
> > Cc: Jérôme Glisse <[e-mail]>
> > Cc: Jan Kara <[e-mail]>
> > Cc: Dan Williams <[e-mail]>
> > Cc: Greg Kroah-Hartman <[e-mail]>
> > Cc: [e-mail]
> > Cc: [e-mail]
> > Cc: [e-mail]
> > Cc: [e-mail]
> > Cc: Bjorn Helgaas <[e-mail]>
> > Cc: [e-mail]
> > ---
> >  drivers/pci/pci-sysfs.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > index fb072f4b3176..0c45b4f7b214 100644
> > --- a/drivers/pci/pci-sysfs.c
> > +++ b/drivers/pci/pci-sysfs.c
> > @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
> >  {
> >       int error;
> >
> > +     if (!sysfs_initialized)
> > +             return;
> > +
> >       b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
> >                              GFP_ATOMIC);
> >       if (!b->legacy_io)
> > @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
> >  static int __init pci_sysfs_init(void)
> >  {
> >       struct pci_dev *pdev = NULL;
> > +     struct pci_bus *pbus = NULL;
> >       int retval;
> >
> >       sysfs_initialized = 1;
> > @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
> >               }
> >       }
> >
> > +     while ((pbus = pci_find_next_bus(pbus)))
> > +             pci_create_legacy_files(pbus);
> > +
> >       return 0;
> >  }
> >  late_initcall(pci_sysfs_init);
> > --
> > 2.30.0
> >
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > [e-mail]
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Daniel Vetter Feb. 5, 2021, 9:59 a.m. UTC | #4

On Thu, Feb 4, 2021 at 11:24 PM Pali Rohár <[e-mail]> wrote:
>
> On Thursday 04 February 2021 15:50:19 Bjorn Helgaas wrote:
> > [+cc Oliver, Pali, Krzysztof]
>
> Just to note that extending or using sysfs_initialized introduces
> another race condition into kernel code which results in PCI fatal
> errors. Details are in email discussion which Bjorn already sent.

Yeah I wondered why this doesn't race, but since the history goes back
to pre-git times I figured it would have been addressed somehow
already if it indeed does race.
-Daniel

> > s/also/Also/ in subject
> >
> > On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> > > We are already doing this for all the regular sysfs files on PCI
> > > devices, but not yet on the legacy io files on the PCI buses. Thus far
> > > now problem, but in the next patch I want to wire up iomem revoke
> > > support. That needs the vfs up an running already to make so that
> > > iomem_get_mapping() works.
> >
> > s/now problem/no problem/
> > s/an running/and running/
> > s/so that/sure that/ ?
> >
> > iomem_get_mapping() doesn't exist; I don't know what that should be.
> >
> > > Wire it up exactly like the existing code. Note that
> > > pci_remove_legacy_files() doesn't need a check since the one for
> > > pci_bus->legacy_io is sufficient.
> >
> > I'm not sure exactly what you mean by "the existing code."  I could
> > probably figure it out, but it would save time to mention the existing
> > function here.
> >
> > This looks like another instance where we should really apply Oliver's
> > idea of converting these to attribute_groups [1].
> >
> > The cover letter mentions options discussed with Greg in [2], but I
> > don't think the "sysfs_initialized" hack vs attribute_groups was part
> > of that discussion.
> >
> > It's not absolutely a show-stopper, but it *is* a shame to extend the
> > sysfs_initialized hack if attribute_groups could do this more cleanly
> > and help solve more than one issue.
> >
> > Bjorn
> >
> > [1] https://lore.kernel.org/r/[e-mail]
> > [2] https://lore.kernel.org/dri-devel/[e-mail]/
> >
> > > Signed-off-by: Daniel Vetter <[e-mail]>
> > > Cc: Stephen Rothwell <[e-mail]>
> > > Cc: Jason Gunthorpe <[e-mail]>
> > > Cc: Kees Cook <[e-mail]>
> > > Cc: Dan Williams <[e-mail]>
> > > Cc: Andrew Morton <[e-mail]>
> > > Cc: John Hubbard <[e-mail]>
> > > Cc: Jérôme Glisse <[e-mail]>
> > > Cc: Jan Kara <[e-mail]>
> > > Cc: Dan Williams <[e-mail]>
> > > Cc: Greg Kroah-Hartman <[e-mail]>
> > > Cc: [e-mail]
> > > Cc: [e-mail]
> > > Cc: [e-mail]
> > > Cc: [e-mail]
> > > Cc: Bjorn Helgaas <[e-mail]>
> > > Cc: [e-mail]
> > > ---
> > >  drivers/pci/pci-sysfs.c | 7 +++++++
> > >  1 file changed, 7 insertions(+)
> > >
> > > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > > index fb072f4b3176..0c45b4f7b214 100644
> > > --- a/drivers/pci/pci-sysfs.c
> > > +++ b/drivers/pci/pci-sysfs.c
> > > @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
> > >  {
> > >     int error;
> > >
> > > +   if (!sysfs_initialized)
> > > +           return;
> > > +
> > >     b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
> > >                            GFP_ATOMIC);
> > >     if (!b->legacy_io)
> > > @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
> > >  static int __init pci_sysfs_init(void)
> > >  {
> > >     struct pci_dev *pdev = NULL;
> > > +   struct pci_bus *pbus = NULL;
> > >     int retval;
> > >
> > >     sysfs_initialized = 1;
> > > @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
> > >             }
> > >     }
> > >
> > > +   while ((pbus = pci_find_next_bus(pbus)))
> > > +           pci_create_legacy_files(pbus);
> > > +
> > >     return 0;
> > >  }
> > >  late_initcall(pci_sysfs_init);
> > > --
> > > 2.30.0
> > >
> > >
> > > _______________________________________________
> > > linux-arm-kernel mailing list
> > > [e-mail]
> > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Pali Rohár Feb. 5, 2021, 10:04 a.m. UTC | #5

On Friday 05 February 2021 10:59:50 Daniel Vetter wrote:
> On Thu, Feb 4, 2021 at 11:24 PM Pali Rohár <[e-mail]> wrote:
> >
> > On Thursday 04 February 2021 15:50:19 Bjorn Helgaas wrote:
> > > [+cc Oliver, Pali, Krzysztof]
> >
> > Just to note that extending or using sysfs_initialized introduces
> > another race condition into kernel code which results in PCI fatal
> > errors. Details are in email discussion which Bjorn already sent.
> 
> Yeah I wondered why this doesn't race.

It races, but with smaller probability. I have not seen this race
condition on x86. But I was able to reproduce it with native PCIe
drivers on ARM64 (Marvell Armada 3720; pci-aardvark). In mentioned
discussion I wrote when this race condition happen. But I understand
that it is hard to simulate it.

> but since the history goes back
> to pre-git times I figured it would have been addressed somehow
> already if it indeed does race.
> -Daniel
> 
> > > s/also/Also/ in subject
> > >
> > > On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> > > > We are already doing this for all the regular sysfs files on PCI
> > > > devices, but not yet on the legacy io files on the PCI buses. Thus far
> > > > now problem, but in the next patch I want to wire up iomem revoke
> > > > support. That needs the vfs up an running already to make so that
> > > > iomem_get_mapping() works.
> > >
> > > s/now problem/no problem/
> > > s/an running/and running/
> > > s/so that/sure that/ ?
> > >
> > > iomem_get_mapping() doesn't exist; I don't know what that should be.
> > >
> > > > Wire it up exactly like the existing code. Note that
> > > > pci_remove_legacy_files() doesn't need a check since the one for
> > > > pci_bus->legacy_io is sufficient.
> > >
> > > I'm not sure exactly what you mean by "the existing code."  I could
> > > probably figure it out, but it would save time to mention the existing
> > > function here.
> > >
> > > This looks like another instance where we should really apply Oliver's
> > > idea of converting these to attribute_groups [1].
> > >
> > > The cover letter mentions options discussed with Greg in [2], but I
> > > don't think the "sysfs_initialized" hack vs attribute_groups was part
> > > of that discussion.
> > >
> > > It's not absolutely a show-stopper, but it *is* a shame to extend the
> > > sysfs_initialized hack if attribute_groups could do this more cleanly
> > > and help solve more than one issue.
> > >
> > > Bjorn
> > >
> > > [1] https://lore.kernel.org/r/[e-mail]
> > > [2] https://lore.kernel.org/dri-devel/[e-mail]/
> > >
> > > > Signed-off-by: Daniel Vetter <[e-mail]>
> > > > Cc: Stephen Rothwell <[e-mail]>
> > > > Cc: Jason Gunthorpe <[e-mail]>
> > > > Cc: Kees Cook <[e-mail]>
> > > > Cc: Dan Williams <[e-mail]>
> > > > Cc: Andrew Morton <[e-mail]>
> > > > Cc: John Hubbard <[e-mail]>
> > > > Cc: Jérôme Glisse <[e-mail]>
> > > > Cc: Jan Kara <[e-mail]>
> > > > Cc: Dan Williams <[e-mail]>
> > > > Cc: Greg Kroah-Hartman <[e-mail]>
> > > > Cc: [e-mail]
> > > > Cc: [e-mail]
> > > > Cc: [e-mail]
> > > > Cc: [e-mail]
> > > > Cc: Bjorn Helgaas <[e-mail]>
> > > > Cc: [e-mail]
> > > > ---
> > > >  drivers/pci/pci-sysfs.c | 7 +++++++
> > > >  1 file changed, 7 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > > > index fb072f4b3176..0c45b4f7b214 100644
> > > > --- a/drivers/pci/pci-sysfs.c
> > > > +++ b/drivers/pci/pci-sysfs.c
> > > > @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
> > > >  {
> > > >     int error;
> > > >
> > > > +   if (!sysfs_initialized)
> > > > +           return;
> > > > +
> > > >     b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
> > > >                            GFP_ATOMIC);
> > > >     if (!b->legacy_io)
> > > > @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
> > > >  static int __init pci_sysfs_init(void)
> > > >  {
> > > >     struct pci_dev *pdev = NULL;
> > > > +   struct pci_bus *pbus = NULL;
> > > >     int retval;
> > > >
> > > >     sysfs_initialized = 1;
> > > > @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
> > > >             }
> > > >     }
> > > >
> > > > +   while ((pbus = pci_find_next_bus(pbus)))
> > > > +           pci_create_legacy_files(pbus);
> > > > +
> > > >     return 0;
> > > >  }
> > > >  late_initcall(pci_sysfs_init);
> > > > --
> > > > 2.30.0
> > > >
> > > >
> > > > _______________________________________________
> > > > linux-arm-kernel mailing list
> > > > [e-mail]
> > > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> 
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

Daniel Vetter Feb. 5, 2021, 10:16 a.m. UTC | #6

On Fri, Feb 5, 2021 at 11:04 AM Pali Rohár <[e-mail]> wrote:
>
> On Friday 05 February 2021 10:59:50 Daniel Vetter wrote:
> > On Thu, Feb 4, 2021 at 11:24 PM Pali Rohár <[e-mail]> wrote:
> > >
> > > On Thursday 04 February 2021 15:50:19 Bjorn Helgaas wrote:
> > > > [+cc Oliver, Pali, Krzysztof]
> > >
> > > Just to note that extending or using sysfs_initialized introduces
> > > another race condition into kernel code which results in PCI fatal
> > > errors. Details are in email discussion which Bjorn already sent.
> >
> > Yeah I wondered why this doesn't race.
>
> It races, but with smaller probability. I have not seen this race
> condition on x86. But I was able to reproduce it with native PCIe
> drivers on ARM64 (Marvell Armada 3720; pci-aardvark). In mentioned
> discussion I wrote when this race condition happen. But I understand
> that it is hard to simulate it.

btw I looked at your patch, and isn't that just reducing the race window?

I think we have a very similar problem in drm, where the
drm_dev_register() for the overall device (which also registers all
drm_connector) can race with the hotplug of an individual connector in
drm_connector_register() which is hotplugged at runtime.

I went with a per-connector registered boolean + a lock to make sure
that really only one of the two call paths can end up registering the
connector. Part of registering connectors is setting up sysfs files,
so I think it's exactly the same problem as here.

Cheers, Daniel

>
> > but since the history goes back
> > to pre-git times I figured it would have been addressed somehow
> > already if it indeed does race.
> > -Daniel
> >
> > > > s/also/Also/ in subject
> > > >
> > > > On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> > > > > We are already doing this for all the regular sysfs files on PCI
> > > > > devices, but not yet on the legacy io files on the PCI buses. Thus far
> > > > > now problem, but in the next patch I want to wire up iomem revoke
> > > > > support. That needs the vfs up an running already to make so that
> > > > > iomem_get_mapping() works.
> > > >
> > > > s/now problem/no problem/
> > > > s/an running/and running/
> > > > s/so that/sure that/ ?
> > > >
> > > > iomem_get_mapping() doesn't exist; I don't know what that should be.
> > > >
> > > > > Wire it up exactly like the existing code. Note that
> > > > > pci_remove_legacy_files() doesn't need a check since the one for
> > > > > pci_bus->legacy_io is sufficient.
> > > >
> > > > I'm not sure exactly what you mean by "the existing code."  I could
> > > > probably figure it out, but it would save time to mention the existing
> > > > function here.
> > > >
> > > > This looks like another instance where we should really apply Oliver's
> > > > idea of converting these to attribute_groups [1].
> > > >
> > > > The cover letter mentions options discussed with Greg in [2], but I
> > > > don't think the "sysfs_initialized" hack vs attribute_groups was part
> > > > of that discussion.
> > > >
> > > > It's not absolutely a show-stopper, but it *is* a shame to extend the
> > > > sysfs_initialized hack if attribute_groups could do this more cleanly
> > > > and help solve more than one issue.
> > > >
> > > > Bjorn
> > > >
> > > > [1] https://lore.kernel.org/r/[e-mail]
> > > > [2] https://lore.kernel.org/dri-devel/[e-mail]/
> > > >
> > > > > Signed-off-by: Daniel Vetter <[e-mail]>
> > > > > Cc: Stephen Rothwell <[e-mail]>
> > > > > Cc: Jason Gunthorpe <[e-mail]>
> > > > > Cc: Kees Cook <[e-mail]>
> > > > > Cc: Dan Williams <[e-mail]>
> > > > > Cc: Andrew Morton <[e-mail]>
> > > > > Cc: John Hubbard <[e-mail]>
> > > > > Cc: Jérôme Glisse <[e-mail]>
> > > > > Cc: Jan Kara <[e-mail]>
> > > > > Cc: Dan Williams <[e-mail]>
> > > > > Cc: Greg Kroah-Hartman <[e-mail]>
> > > > > Cc: [e-mail]
> > > > > Cc: [e-mail]
> > > > > Cc: [e-mail]
> > > > > Cc: [e-mail]
> > > > > Cc: Bjorn Helgaas <[e-mail]>
> > > > > Cc: [e-mail]
> > > > > ---
> > > > >  drivers/pci/pci-sysfs.c | 7 +++++++
> > > > >  1 file changed, 7 insertions(+)
> > > > >
> > > > > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > > > > index fb072f4b3176..0c45b4f7b214 100644
> > > > > --- a/drivers/pci/pci-sysfs.c
> > > > > +++ b/drivers/pci/pci-sysfs.c
> > > > > @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
> > > > >  {
> > > > >     int error;
> > > > >
> > > > > +   if (!sysfs_initialized)
> > > > > +           return;
> > > > > +
> > > > >     b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
> > > > >                            GFP_ATOMIC);
> > > > >     if (!b->legacy_io)
> > > > > @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
> > > > >  static int __init pci_sysfs_init(void)
> > > > >  {
> > > > >     struct pci_dev *pdev = NULL;
> > > > > +   struct pci_bus *pbus = NULL;
> > > > >     int retval;
> > > > >
> > > > >     sysfs_initialized = 1;
> > > > > @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
> > > > >             }
> > > > >     }
> > > > >
> > > > > +   while ((pbus = pci_find_next_bus(pbus)))
> > > > > +           pci_create_legacy_files(pbus);
> > > > > +
> > > > >     return 0;
> > > > >  }
> > > > >  late_initcall(pci_sysfs_init);
> > > > > --
> > > > > 2.30.0
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > linux-arm-kernel mailing list
> > > > > [e-mail]
> > > > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> >
> >
> >
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > http://blog.ffwll.ch

Pali Rohár Feb. 5, 2021, 10:21 a.m. UTC | #7

On Friday 05 February 2021 11:16:00 Daniel Vetter wrote:
> On Fri, Feb 5, 2021 at 11:04 AM Pali Rohár <[e-mail]> wrote:
> >
> > On Friday 05 February 2021 10:59:50 Daniel Vetter wrote:
> > > On Thu, Feb 4, 2021 at 11:24 PM Pali Rohár <[e-mail]> wrote:
> > > >
> > > > On Thursday 04 February 2021 15:50:19 Bjorn Helgaas wrote:
> > > > > [+cc Oliver, Pali, Krzysztof]
> > > >
> > > > Just to note that extending or using sysfs_initialized introduces
> > > > another race condition into kernel code which results in PCI fatal
> > > > errors. Details are in email discussion which Bjorn already sent.
> > >
> > > Yeah I wondered why this doesn't race.
> >
> > It races, but with smaller probability. I have not seen this race
> > condition on x86. But I was able to reproduce it with native PCIe
> > drivers on ARM64 (Marvell Armada 3720; pci-aardvark). In mentioned
> > discussion I wrote when this race condition happen. But I understand
> > that it is hard to simulate it.
> 
> btw I looked at your patch, and isn't that just reducing the race window?

I probably have not wrote reply to that thread and only to Krzysztof on
IRC, but my "hack" really does not solve that race condition. And as you
wrote it only reduced occurrence on tested HW.

Krzysztof wrote that would look at this issue and try to solve it
properly. So I have not doing more investigation on that my "hack"
patch, race conditions are hard to catch and solve...

> I think we have a very similar problem in drm, where the
> drm_dev_register() for the overall device (which also registers all
> drm_connector) can race with the hotplug of an individual connector in
> drm_connector_register() which is hotplugged at runtime.
> 
> I went with a per-connector registered boolean + a lock to make sure
> that really only one of the two call paths can end up registering the
> connector. Part of registering connectors is setting up sysfs files,
> so I think it's exactly the same problem as here.
> 
> Cheers, Daniel
> 
> >
> > > but since the history goes back
> > > to pre-git times I figured it would have been addressed somehow
> > > already if it indeed does race.
> > > -Daniel
> > >
> > > > > s/also/Also/ in subject
> > > > >
> > > > > On Thu, Feb 04, 2021 at 05:58:30PM +0100, Daniel Vetter wrote:
> > > > > > We are already doing this for all the regular sysfs files on PCI
> > > > > > devices, but not yet on the legacy io files on the PCI buses. Thus far
> > > > > > now problem, but in the next patch I want to wire up iomem revoke
> > > > > > support. That needs the vfs up an running already to make so that
> > > > > > iomem_get_mapping() works.
> > > > >
> > > > > s/now problem/no problem/
> > > > > s/an running/and running/
> > > > > s/so that/sure that/ ?
> > > > >
> > > > > iomem_get_mapping() doesn't exist; I don't know what that should be.
> > > > >
> > > > > > Wire it up exactly like the existing code. Note that
> > > > > > pci_remove_legacy_files() doesn't need a check since the one for
> > > > > > pci_bus->legacy_io is sufficient.
> > > > >
> > > > > I'm not sure exactly what you mean by "the existing code."  I could
> > > > > probably figure it out, but it would save time to mention the existing
> > > > > function here.
> > > > >
> > > > > This looks like another instance where we should really apply Oliver's
> > > > > idea of converting these to attribute_groups [1].
> > > > >
> > > > > The cover letter mentions options discussed with Greg in [2], but I
> > > > > don't think the "sysfs_initialized" hack vs attribute_groups was part
> > > > > of that discussion.
> > > > >
> > > > > It's not absolutely a show-stopper, but it *is* a shame to extend the
> > > > > sysfs_initialized hack if attribute_groups could do this more cleanly
> > > > > and help solve more than one issue.
> > > > >
> > > > > Bjorn
> > > > >
> > > > > [1] https://lore.kernel.org/r/[e-mail]
> > > > > [2] https://lore.kernel.org/dri-devel/[e-mail]/
> > > > >
> > > > > > Signed-off-by: Daniel Vetter <[e-mail]>
> > > > > > Cc: Stephen Rothwell <[e-mail]>
> > > > > > Cc: Jason Gunthorpe <[e-mail]>
> > > > > > Cc: Kees Cook <[e-mail]>
> > > > > > Cc: Dan Williams <[e-mail]>
> > > > > > Cc: Andrew Morton <[e-mail]>
> > > > > > Cc: John Hubbard <[e-mail]>
> > > > > > Cc: Jérôme Glisse <[e-mail]>
> > > > > > Cc: Jan Kara <[e-mail]>
> > > > > > Cc: Dan Williams <[e-mail]>
> > > > > > Cc: Greg Kroah-Hartman <[e-mail]>
> > > > > > Cc: [e-mail]
> > > > > > Cc: [e-mail]
> > > > > > Cc: [e-mail]
> > > > > > Cc: [e-mail]
> > > > > > Cc: Bjorn Helgaas <[e-mail]>
> > > > > > Cc: [e-mail]
> > > > > > ---
> > > > > >  drivers/pci/pci-sysfs.c | 7 +++++++
> > > > > >  1 file changed, 7 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > > > > > index fb072f4b3176..0c45b4f7b214 100644
> > > > > > --- a/drivers/pci/pci-sysfs.c
> > > > > > +++ b/drivers/pci/pci-sysfs.c
> > > > > > @@ -927,6 +927,9 @@ void pci_create_legacy_files(struct pci_bus *b)
> > > > > >  {
> > > > > >     int error;
> > > > > >
> > > > > > +   if (!sysfs_initialized)
> > > > > > +           return;
> > > > > > +
> > > > > >     b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
> > > > > >                            GFP_ATOMIC);
> > > > > >     if (!b->legacy_io)
> > > > > > @@ -1448,6 +1451,7 @@ void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
> > > > > >  static int __init pci_sysfs_init(void)
> > > > > >  {
> > > > > >     struct pci_dev *pdev = NULL;
> > > > > > +   struct pci_bus *pbus = NULL;
> > > > > >     int retval;
> > > > > >
> > > > > >     sysfs_initialized = 1;
> > > > > > @@ -1459,6 +1463,9 @@ static int __init pci_sysfs_init(void)
> > > > > >             }
> > > > > >     }
> > > > > >
> > > > > > +   while ((pbus = pci_find_next_bus(pbus)))
> > > > > > +           pci_create_legacy_files(pbus);
> > > > > > +
> > > > > >     return 0;
> > > > > >  }
> > > > > >  late_initcall(pci_sysfs_init);
> > > > > > --
> > > > > > 2.30.0
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > linux-arm-kernel mailing list
> > > > > > [e-mail]
> > > > > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> > >
> > >
> > >
> > > --
> > > Daniel Vetter
> > > Software Engineer, Intel Corporation
> > > http://blog.ffwll.ch
> 
> 
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

Patch

diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index fb072f4b3176..0c45b4f7b214 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -927,6 +927,9 @@  void pci_create_legacy_files(struct pci_bus *b)
 {
 	int error;
 
+	if (!sysfs_initialized)
+		return;
+
 	b->legacy_io = kcalloc(2, sizeof(struct bin_attribute),
 			       GFP_ATOMIC);
 	if (!b->legacy_io)
@@ -1448,6 +1451,7 @@  void pci_remove_sysfs_dev_files(struct pci_dev *pdev)
 static int __init pci_sysfs_init(void)
 {
 	struct pci_dev *pdev = NULL;
+	struct pci_bus *pbus = NULL;
 	int retval;
 
 	sysfs_initialized = 1;
@@ -1459,6 +1463,9 @@  static int __init pci_sysfs_init(void)
 		}
 	}
 
+	while ((pbus = pci_find_next_bus(pbus)))
+		pci_create_legacy_files(pbus);
+
 	return 0;
 }
 late_initcall(pci_sysfs_init);