| Message ID | 20230619014715.3792883-1-maobibo@loongson.cn (mailing list archive) |
|---|---|
| State | New, archived |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | [v5] PCI: Align pci memory space base address with page size | expand |
gentle ping. 在 2023/6/19 09:47, Bibo Mao 写道: > Some PCI devices have only 4K memory space size, it is normal in general > machines and aligned with page size. However some architectures which > support different page size, default page size on LoongArch is 16K, and > ARM64 supports page size varying from 4K to 64K. On machines where larger > page size is use, memory space region of two different pci devices may be > in one page. It is not safe with mmu protection, also VFIO pci device > driver requires base address of pci memory space page aligned, so that it > can be memory mapped to qemu user space when it is passed-through to vm. > > It consumes more pci memory resource with page size alignment requirement, > here extra option PCI_MEMRES_PAGE_ALIGN is added, it can be enabled by > different architectures, currently arm64/loongarch enable this option. > > Signed-off-by: Bibo Mao <maobibo@loongson.cn> > Reviewed-by: Huacai Chen <chenhuacai@loongson.cn> > --- > Change history > v5: enable option PCI_MEMRES_PAGE_ALIGN on arm64. Verified on LoongArch > and pass to compile on arm64 with defconfig > > v4: add extra kernel option PCI_MEMRES_PAGE_ALIGN to set memory resource > page aligned > > v3: move alignment requirement to generic pci code > > v2: add pci resource alignment requirement in arch specified function > pcibios_align_resource on arm64/LoongArch platforms > > --- > arch/arm64/Kconfig | 1 + > arch/loongarch/Kconfig | 1 + > drivers/pci/Kconfig | 3 +++ > drivers/pci/setup-res.c | 7 +++++++ > 4 files changed, 12 insertions(+) > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index 343e1e1cae10..24858bbf2b72 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -232,6 +232,7 @@ config ARM64 > select OF_EARLY_FLATTREE > select PCI_DOMAINS_GENERIC if PCI > select PCI_ECAM if (ACPI && PCI) > + select PCI_MEMRES_PAGE_ALIGN if PCI > select PCI_SYSCALL if PCI > select POWER_RESET > select POWER_SUPPLY > diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig > index d38b066fc931..7dbde5e5b351 100644 > --- a/arch/loongarch/Kconfig > +++ b/arch/loongarch/Kconfig > @@ -140,6 +140,7 @@ config LOONGARCH > select PCI_DOMAINS_GENERIC > select PCI_ECAM if ACPI > select PCI_LOONGSON > + select PCI_MEMRES_PAGE_ALIGN > select PCI_MSI_ARCH_FALLBACKS > select PCI_QUIRKS > select PERF_USE_VMALLOC > diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig > index 9309f2469b41..9be5f85ff9dc 100644 > --- a/drivers/pci/Kconfig > +++ b/drivers/pci/Kconfig > @@ -128,6 +128,9 @@ config PCI_LOCKLESS_CONFIG > config PCI_BRIDGE_EMUL > bool > > +config PCI_MEMRES_PAGE_ALIGN > + bool > + > config PCI_IOV > bool "PCI IOV support" > select PCI_ATS > diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c > index 967f9a758923..6ad76734a670 100644 > --- a/drivers/pci/setup-res.c > +++ b/drivers/pci/setup-res.c > @@ -339,6 +339,13 @@ int pci_assign_resource(struct pci_dev *dev, int resno) > return -EINVAL; > } > > +#ifdef CONFIG_PCI_MEMRES_PAGE_ALIGN > + /* > + * force minimum page alignment for vfio pci usage > + */ > + if (res->flags & IORESOURCE_MEM) > + align = max_t(resource_size_t, PAGE_SIZE, align); > +#endif > size = resource_size(res); > ret = _pci_assign_resource(dev, resno, size, align); >
Bjourn, Hucai, ping again. The is such issue on only LoongArch system since 16K page size is used. Should we add code in general framework or in LoongArch specified code in v1? If you do not think that it is problem, I can give up. LoongArch system is not only for myself own, I do not care about it also since LoongArch Maintainer think it is not a issue. Regards Bibo Mao 在 2023/6/26 09:30, bibo mao 写道: > gentle ping. > > 在 2023/6/19 09:47, Bibo Mao 写道: >> Some PCI devices have only 4K memory space size, it is normal in general >> machines and aligned with page size. However some architectures which >> support different page size, default page size on LoongArch is 16K, and >> ARM64 supports page size varying from 4K to 64K. On machines where larger >> page size is use, memory space region of two different pci devices may be >> in one page. It is not safe with mmu protection, also VFIO pci device >> driver requires base address of pci memory space page aligned, so that it >> can be memory mapped to qemu user space when it is passed-through to vm. >> >> It consumes more pci memory resource with page size alignment requirement, >> here extra option PCI_MEMRES_PAGE_ALIGN is added, it can be enabled by >> different architectures, currently arm64/loongarch enable this option. >> >> Signed-off-by: Bibo Mao <maobibo@loongson.cn> >> Reviewed-by: Huacai Chen <chenhuacai@loongson.cn> >> --- >> Change history >> v5: enable option PCI_MEMRES_PAGE_ALIGN on arm64. Verified on LoongArch >> and pass to compile on arm64 with defconfig >> >> v4: add extra kernel option PCI_MEMRES_PAGE_ALIGN to set memory resource >> page aligned >> >> v3: move alignment requirement to generic pci code >> >> v2: add pci resource alignment requirement in arch specified function >> pcibios_align_resource on arm64/LoongArch platforms >> >> --- >> arch/arm64/Kconfig | 1 + >> arch/loongarch/Kconfig | 1 + >> drivers/pci/Kconfig | 3 +++ >> drivers/pci/setup-res.c | 7 +++++++ >> 4 files changed, 12 insertions(+) >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index 343e1e1cae10..24858bbf2b72 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -232,6 +232,7 @@ config ARM64 >> select OF_EARLY_FLATTREE >> select PCI_DOMAINS_GENERIC if PCI >> select PCI_ECAM if (ACPI && PCI) >> + select PCI_MEMRES_PAGE_ALIGN if PCI >> select PCI_SYSCALL if PCI >> select POWER_RESET >> select POWER_SUPPLY >> diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig >> index d38b066fc931..7dbde5e5b351 100644 >> --- a/arch/loongarch/Kconfig >> +++ b/arch/loongarch/Kconfig >> @@ -140,6 +140,7 @@ config LOONGARCH >> select PCI_DOMAINS_GENERIC >> select PCI_ECAM if ACPI >> select PCI_LOONGSON >> + select PCI_MEMRES_PAGE_ALIGN >> select PCI_MSI_ARCH_FALLBACKS >> select PCI_QUIRKS >> select PERF_USE_VMALLOC >> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig >> index 9309f2469b41..9be5f85ff9dc 100644 >> --- a/drivers/pci/Kconfig >> +++ b/drivers/pci/Kconfig >> @@ -128,6 +128,9 @@ config PCI_LOCKLESS_CONFIG >> config PCI_BRIDGE_EMUL >> bool >> >> +config PCI_MEMRES_PAGE_ALIGN >> + bool >> + >> config PCI_IOV >> bool "PCI IOV support" >> select PCI_ATS >> diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c >> index 967f9a758923..6ad76734a670 100644 >> --- a/drivers/pci/setup-res.c >> +++ b/drivers/pci/setup-res.c >> @@ -339,6 +339,13 @@ int pci_assign_resource(struct pci_dev *dev, int resno) >> return -EINVAL; >> } >> >> +#ifdef CONFIG_PCI_MEMRES_PAGE_ALIGN >> + /* >> + * force minimum page alignment for vfio pci usage >> + */ >> + if (res->flags & IORESOURCE_MEM) >> + align = max_t(resource_size_t, PAGE_SIZE, align); >> +#endif >> size = resource_size(res); >> ret = _pci_assign_resource(dev, resno, size, align); >> > > _______________________________________________ > Loongson-kernel mailing list -- loongson-kernel@lists.loongnix.cn > To unsubscribe send an email to loongson-kernel-leave@lists.loongnix.cn
On Fri, Jul 7, 2023 at 10:47 AM bibo mao <maobibo@loongson.cn> wrote: > > Bjourn, Hucai, > > ping again. > > The is such issue on only LoongArch system since 16K page size is used. > > Should we add code in general framework or in LoongArch specified code > in v1? Though I still don't know why others have no problems. But it is surely a bug on LoongArch. So if Bjorn has no objections, please use the old method (just like v1 but you can simply align to page size unconditionally). Huacai > > If you do not think that it is problem, I can give up. LoongArch system > is not only for myself own, I do not care about it also since LoongArch > Maintainer think it is not a issue. > > Regards > Bibo Mao > > 在 2023/6/26 09:30, bibo mao 写道: > > gentle ping. > > > > 在 2023/6/19 09:47, Bibo Mao 写道: > >> Some PCI devices have only 4K memory space size, it is normal in general > >> machines and aligned with page size. However some architectures which > >> support different page size, default page size on LoongArch is 16K, and > >> ARM64 supports page size varying from 4K to 64K. On machines where larger > >> page size is use, memory space region of two different pci devices may be > >> in one page. It is not safe with mmu protection, also VFIO pci device > >> driver requires base address of pci memory space page aligned, so that it > >> can be memory mapped to qemu user space when it is passed-through to vm. > >> > >> It consumes more pci memory resource with page size alignment requirement, > >> here extra option PCI_MEMRES_PAGE_ALIGN is added, it can be enabled by > >> different architectures, currently arm64/loongarch enable this option. > >> > >> Signed-off-by: Bibo Mao <maobibo@loongson.cn> > >> Reviewed-by: Huacai Chen <chenhuacai@loongson.cn> > >> --- > >> Change history > >> v5: enable option PCI_MEMRES_PAGE_ALIGN on arm64. Verified on LoongArch > >> and pass to compile on arm64 with defconfig > >> > >> v4: add extra kernel option PCI_MEMRES_PAGE_ALIGN to set memory resource > >> page aligned > >> > >> v3: move alignment requirement to generic pci code > >> > >> v2: add pci resource alignment requirement in arch specified function > >> pcibios_align_resource on arm64/LoongArch platforms > >> > >> --- > >> arch/arm64/Kconfig | 1 + > >> arch/loongarch/Kconfig | 1 + > >> drivers/pci/Kconfig | 3 +++ > >> drivers/pci/setup-res.c | 7 +++++++ > >> 4 files changed, 12 insertions(+) > >> > >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > >> index 343e1e1cae10..24858bbf2b72 100644 > >> --- a/arch/arm64/Kconfig > >> +++ b/arch/arm64/Kconfig > >> @@ -232,6 +232,7 @@ config ARM64 > >> select OF_EARLY_FLATTREE > >> select PCI_DOMAINS_GENERIC if PCI > >> select PCI_ECAM if (ACPI && PCI) > >> + select PCI_MEMRES_PAGE_ALIGN if PCI > >> select PCI_SYSCALL if PCI > >> select POWER_RESET > >> select POWER_SUPPLY > >> diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig > >> index d38b066fc931..7dbde5e5b351 100644 > >> --- a/arch/loongarch/Kconfig > >> +++ b/arch/loongarch/Kconfig > >> @@ -140,6 +140,7 @@ config LOONGARCH > >> select PCI_DOMAINS_GENERIC > >> select PCI_ECAM if ACPI > >> select PCI_LOONGSON > >> + select PCI_MEMRES_PAGE_ALIGN > >> select PCI_MSI_ARCH_FALLBACKS > >> select PCI_QUIRKS > >> select PERF_USE_VMALLOC > >> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig > >> index 9309f2469b41..9be5f85ff9dc 100644 > >> --- a/drivers/pci/Kconfig > >> +++ b/drivers/pci/Kconfig > >> @@ -128,6 +128,9 @@ config PCI_LOCKLESS_CONFIG > >> config PCI_BRIDGE_EMUL > >> bool > >> > >> +config PCI_MEMRES_PAGE_ALIGN > >> + bool > >> + > >> config PCI_IOV > >> bool "PCI IOV support" > >> select PCI_ATS > >> diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c > >> index 967f9a758923..6ad76734a670 100644 > >> --- a/drivers/pci/setup-res.c > >> +++ b/drivers/pci/setup-res.c > >> @@ -339,6 +339,13 @@ int pci_assign_resource(struct pci_dev *dev, int resno) > >> return -EINVAL; > >> } > >> > >> +#ifdef CONFIG_PCI_MEMRES_PAGE_ALIGN > >> + /* > >> + * force minimum page alignment for vfio pci usage > >> + */ > >> + if (res->flags & IORESOURCE_MEM) > >> + align = max_t(resource_size_t, PAGE_SIZE, align); > >> +#endif > >> size = resource_size(res); > >> ret = _pci_assign_resource(dev, resno, size, align); > >> > > > > _______________________________________________ > > Loongson-kernel mailing list -- loongson-kernel@lists.loongnix.cn > > To unsubscribe send an email to loongson-kernel-leave@lists.loongnix.cn > >
On Sun, Jul 09, 2023 at 05:19:57PM +0800, Huacai Chen wrote: > On Fri, Jul 7, 2023 at 10:47 AM bibo mao <maobibo@loongson.cn> wrote: > > > > Bjourn, Hucai, > > > > ping again. > > > > The is such issue on only LoongArch system since 16K page size is used. > > > > Should we add code in general framework or in LoongArch specified code > > in v1? > Though I still don't know why others have no problems. But it is > surely a bug on LoongArch. So if Bjorn has no objections, please use > the old method (just like v1 but you can simply align to page size > unconditionally). The initial assertion is that this is a problem for any architecture with a page-size > 4K, so there's nothing LoongArch-specific about that on paper. More likely, we've only just come out of the merge window so I suggest a little patience is probably all that is needed. Will
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10..24858bbf2b72 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -232,6 +232,7 @@ config ARM64 select OF_EARLY_FLATTREE select PCI_DOMAINS_GENERIC if PCI select PCI_ECAM if (ACPI && PCI) + select PCI_MEMRES_PAGE_ALIGN if PCI select PCI_SYSCALL if PCI select POWER_RESET select POWER_SUPPLY diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index d38b066fc931..7dbde5e5b351 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -140,6 +140,7 @@ config LOONGARCH select PCI_DOMAINS_GENERIC select PCI_ECAM if ACPI select PCI_LOONGSON + select PCI_MEMRES_PAGE_ALIGN select PCI_MSI_ARCH_FALLBACKS select PCI_QUIRKS select PERF_USE_VMALLOC diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 9309f2469b41..9be5f85ff9dc 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -128,6 +128,9 @@ config PCI_LOCKLESS_CONFIG config PCI_BRIDGE_EMUL bool +config PCI_MEMRES_PAGE_ALIGN + bool + config PCI_IOV bool "PCI IOV support" select PCI_ATS diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index 967f9a758923..6ad76734a670 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -339,6 +339,13 @@ int pci_assign_resource(struct pci_dev *dev, int resno) return -EINVAL; } +#ifdef CONFIG_PCI_MEMRES_PAGE_ALIGN + /* + * force minimum page alignment for vfio pci usage + */ + if (res->flags & IORESOURCE_MEM) + align = max_t(resource_size_t, PAGE_SIZE, align); +#endif size = resource_size(res); ret = _pci_assign_resource(dev, resno, size, align);