[V4,6/6] PCI: rcar: Fix 64bit MSI message address handling

Commit Message

Marek Vasut March 25, 2019, 11:41 a.m. UTC

From: Marek Vasut <[e-mail]>

The MSI message address in the RC address space can be 64 bit. The
R-Car PCIe RC supports such a 64bit MSI message address as well.
The code currently uses virt_to_phys(__get_free_pages()) to obtain
a reserved page for the MSI message address, and the return value
of which can be a 64 bit physical address on 64 bit system.

However, the driver only programs PCIEMSIALR register with the bottom
32 bits of the virt_to_phys(__get_free_pages()) return value and does
not program the top 32 bits into PCIEMSIAUR, but rather programs the
PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
SoCs, however may fail on new 64 bit R-Car SoCs.

Since from a PCIe controller perspective, an inbound MSI is a memory
write to a special address (in case of this controller, defined by
the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
never hits the DRAM _and_ because allocation of an MSI by a PCIe card
driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
cause memory corruption or other issues.

There is however the possibility that if virt_to_phys(__get_free_pages())
returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
to 0x0 _and_ if the system had physical RAM at the address matching the
value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
physical address matching the value of PCIEMSIALR and a remote write to
such a buffer by a PCIe card would trigger a spurious MSI.

Signed-off-by: Marek Vasut <[e-mail]>
Cc: Geert Uytterhoeven <[e-mail]>
Cc: Phil Edworthy <[e-mail]>
Cc: Simon Horman <[e-mail]>
Cc: Wolfram Sang <[e-mail]>
Cc: [e-mail]
To: [e-mail]
Reviewed-by: Geert Uytterhoeven <[e-mail]>
---
V2: - s/it's/its/ in commit message
    - Add R-B from Geert
V3: - Reworded commit message and thus dropped Geerts R-B
V4: - Add Geert's R-B again
---
 drivers/pci/controller/pcie-rcar.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comments

Simon Horman March 27, 2019, 11:30 a.m. UTC | #1

On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
> From: Marek Vasut <[e-mail]>
> 
> The MSI message address in the RC address space can be 64 bit. The
> R-Car PCIe RC supports such a 64bit MSI message address as well.
> The code currently uses virt_to_phys(__get_free_pages()) to obtain
> a reserved page for the MSI message address, and the return value
> of which can be a 64 bit physical address on 64 bit system.
> 
> However, the driver only programs PCIEMSIALR register with the bottom
> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
> not program the top 32 bits into PCIEMSIAUR, but rather programs the
> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
> SoCs, however may fail on new 64 bit R-Car SoCs.
> 
> Since from a PCIe controller perspective, an inbound MSI is a memory
> write to a special address (in case of this controller, defined by
> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
> cause memory corruption or other issues.
> 
> There is however the possibility that if virt_to_phys(__get_free_pages())
> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
> to 0x0 _and_ if the system had physical RAM at the address matching the
> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
> physical address matching the value of PCIEMSIALR and a remote write to
> such a buffer by a PCIe card would trigger a spurious MSI.
> 
> Signed-off-by: Marek Vasut <[e-mail]>
> Cc: Geert Uytterhoeven <[e-mail]>
> Cc: Phil Edworthy <[e-mail]>
> Cc: Simon Horman <[e-mail]>
> Cc: Wolfram Sang <[e-mail]>
> Cc: [e-mail]
> To: [e-mail]
> Reviewed-by: Geert Uytterhoeven <[e-mail]>

Does this warrant a Fixes tag?

That notwithstanding,

Reviewed-by: Simon Horman <[e-mail]>

> ---
> V2: - s/it's/its/ in commit message
>     - Add R-B from Geert
> V3: - Reworded commit message and thus dropped Geerts R-B
> V4: - Add Geert's R-B again
> ---
>  drivers/pci/controller/pcie-rcar.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
> index c6013f95bdb2..62d2de9fbf1c 100644
> --- a/drivers/pci/controller/pcie-rcar.c
> +++ b/drivers/pci/controller/pcie-rcar.c
> @@ -890,7 +890,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
>  {
>  	struct device *dev = pcie->dev;
>  	struct rcar_msi *msi = &pcie->msi;
> -	unsigned long base;
> +	phys_addr_t base;
>  	int err, i;
>  
>  	mutex_init(&msi->lock);
> @@ -932,7 +932,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
>  	base = virt_to_phys((void *)msi->pages);
>  
>  	rcar_pci_write_reg(pcie, base | MSIFE, PCIEMSIALR);
> -	rcar_pci_write_reg(pcie, 0, PCIEMSIAUR);
> +	rcar_pci_write_reg(pcie, base >> 32, PCIEMSIAUR);
>  
>  	/* enable all MSI interrupts */
>  	rcar_pci_write_reg(pcie, 0xffffffff, PCIEMSIIER);
> -- 
> 2.20.1
>

Geert Uytterhoeven March 27, 2019, 12:22 p.m. UTC | #2

On Wed, Mar 27, 2019 at 12:30 PM Simon Horman <[e-mail]> wrote:
> On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
> > From: Marek Vasut <[e-mail]>
> > The MSI message address in the RC address space can be 64 bit. The
> > R-Car PCIe RC supports such a 64bit MSI message address as well.
> > The code currently uses virt_to_phys(__get_free_pages()) to obtain
> > a reserved page for the MSI message address, and the return value
> > of which can be a 64 bit physical address on 64 bit system.
> >
> > However, the driver only programs PCIEMSIALR register with the bottom
> > 32 bits of the virt_to_phys(__get_free_pages()) return value and does
> > not program the top 32 bits into PCIEMSIAUR, but rather programs the
> > PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
> > SoCs, however may fail on new 64 bit R-Car SoCs.
> >
> > Since from a PCIe controller perspective, an inbound MSI is a memory
> > write to a special address (in case of this controller, defined by
> > the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
> > never hits the DRAM _and_ because allocation of an MSI by a PCIe card
> > driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
> > in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
> > cause memory corruption or other issues.
> >
> > There is however the possibility that if virt_to_phys(__get_free_pages())
> > returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
> > to 0x0 _and_ if the system had physical RAM at the address matching the
> > value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
> > physical address matching the value of PCIEMSIALR and a remote write to
> > such a buffer by a PCIe card would trigger a spurious MSI.
> >
> > Signed-off-by: Marek Vasut <[e-mail]>
> > Cc: Geert Uytterhoeven <[e-mail]>
> > Cc: Phil Edworthy <[e-mail]>
> > Cc: Simon Horman <[e-mail]>
> > Cc: Wolfram Sang <[e-mail]>
> > Cc: [e-mail]
> > To: [e-mail]
> > Reviewed-by: Geert Uytterhoeven <[e-mail]>
>
> Does this warrant a Fixes tag?

(digging in old sent email)
Fixes: 290c1fb358605402 ("PCI: rcar: Add MSI support for PCIe")

Gr{oetje,eeting}s,

                        Geert

Marek Vasut March 28, 2019, 3:03 a.m. UTC | #3

On 3/27/19 1:22 PM, Geert Uytterhoeven wrote:
> On Wed, Mar 27, 2019 at 12:30 PM Simon Horman <[e-mail]> wrote:
>> On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
>>> From: Marek Vasut <[e-mail]>
>>> The MSI message address in the RC address space can be 64 bit. The
>>> R-Car PCIe RC supports such a 64bit MSI message address as well.
>>> The code currently uses virt_to_phys(__get_free_pages()) to obtain
>>> a reserved page for the MSI message address, and the return value
>>> of which can be a 64 bit physical address on 64 bit system.
>>>
>>> However, the driver only programs PCIEMSIALR register with the bottom
>>> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
>>> not program the top 32 bits into PCIEMSIAUR, but rather programs the
>>> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
>>> SoCs, however may fail on new 64 bit R-Car SoCs.
>>>
>>> Since from a PCIe controller perspective, an inbound MSI is a memory
>>> write to a special address (in case of this controller, defined by
>>> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
>>> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
>>> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
>>> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
>>> cause memory corruption or other issues.
>>>
>>> There is however the possibility that if virt_to_phys(__get_free_pages())
>>> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
>>> to 0x0 _and_ if the system had physical RAM at the address matching the
>>> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
>>> physical address matching the value of PCIEMSIALR and a remote write to
>>> such a buffer by a PCIe card would trigger a spurious MSI.
>>>
>>> Signed-off-by: Marek Vasut <[e-mail]>
>>> Cc: Geert Uytterhoeven <[e-mail]>
>>> Cc: Phil Edworthy <[e-mail]>
>>> Cc: Simon Horman <[e-mail]>
>>> Cc: Wolfram Sang <[e-mail]>
>>> Cc: [e-mail]
>>> To: [e-mail]
>>> Reviewed-by: Geert Uytterhoeven <[e-mail]>
>>
>> Does this warrant a Fixes tag?
> 
> (digging in old sent email)
> Fixes: 290c1fb358605402 ("PCI: rcar: Add MSI support for PCIe")

But does it really fix that commit, given that on Gen2 and earlier, it
was not broken as those were 32bit platforms ?

Geert Uytterhoeven March 28, 2019, 8:02 a.m. UTC | #4

Hi Marek,

On Thu, Mar 28, 2019 at 4:19 AM Marek Vasut <[e-mail]> wrote:
> On 3/27/19 1:22 PM, Geert Uytterhoeven wrote:
> > On Wed, Mar 27, 2019 at 12:30 PM Simon Horman <[e-mail]> wrote:
> >> On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
> >>> From: Marek Vasut <[e-mail]>
> >>> The MSI message address in the RC address space can be 64 bit. The
> >>> R-Car PCIe RC supports such a 64bit MSI message address as well.
> >>> The code currently uses virt_to_phys(__get_free_pages()) to obtain
> >>> a reserved page for the MSI message address, and the return value
> >>> of which can be a 64 bit physical address on 64 bit system.
> >>>
> >>> However, the driver only programs PCIEMSIALR register with the bottom
> >>> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
> >>> not program the top 32 bits into PCIEMSIAUR, but rather programs the
> >>> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
> >>> SoCs, however may fail on new 64 bit R-Car SoCs.
> >>>
> >>> Since from a PCIe controller perspective, an inbound MSI is a memory
> >>> write to a special address (in case of this controller, defined by
> >>> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
> >>> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
> >>> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
> >>> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
> >>> cause memory corruption or other issues.
> >>>
> >>> There is however the possibility that if virt_to_phys(__get_free_pages())
> >>> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
> >>> to 0x0 _and_ if the system had physical RAM at the address matching the
> >>> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
> >>> physical address matching the value of PCIEMSIALR and a remote write to
> >>> such a buffer by a PCIe card would trigger a spurious MSI.
> >>>
> >>> Signed-off-by: Marek Vasut <[e-mail]>
> >>> Cc: Geert Uytterhoeven <[e-mail]>
> >>> Cc: Phil Edworthy <[e-mail]>
> >>> Cc: Simon Horman <[e-mail]>
> >>> Cc: Wolfram Sang <[e-mail]>
> >>> Cc: [e-mail]
> >>> To: [e-mail]
> >>> Reviewed-by: Geert Uytterhoeven <[e-mail]>
> >>
> >> Does this warrant a Fixes tag?
> >
> > (digging in old sent email)
> > Fixes: 290c1fb358605402 ("PCI: rcar: Add MSI support for PCIe")
>
> But does it really fix that commit, given that on Gen2 and earlier, it
> was not broken as those were 32bit platforms ?

It does not fix the bug on that commit, as the bug cannot happen on arm32.
It does fix that commit, in that that commit used "unsigned long" for a
physical address, which is wrong, even on arm32 (esp. with LPAE).
If you insist on having a Fixes tag for a commit where the bug could be
seen:
Fixes: e015f88c368da1e6 ("PCI: rcar: Add support for R-Car H3 to pcie-rcar")

Apart from that, drivers should use the DMA API instead of virt_to_phys().
However, now we have a better understanding of how MSI interrupts work,
we don't even need to allocate that page. All we need is the physical
address of a page that is guaranteed not to be backed by RAM
(i.e. not to be a valid target for a legitimate PCI bus mastering transaction).


Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [e-mail]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Lorenzo Pieralisi March 28, 2019, 4:28 p.m. UTC | #5

On Thu, Mar 28, 2019 at 09:02:00AM +0100, Geert Uytterhoeven wrote:
> Hi Marek,
> 
> On Thu, Mar 28, 2019 at 4:19 AM Marek Vasut <[e-mail]> wrote:
> > On 3/27/19 1:22 PM, Geert Uytterhoeven wrote:
> > > On Wed, Mar 27, 2019 at 12:30 PM Simon Horman <[e-mail]> wrote:
> > >> On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
> > >>> From: Marek Vasut <[e-mail]>
> > >>> The MSI message address in the RC address space can be 64 bit. The
> > >>> R-Car PCIe RC supports such a 64bit MSI message address as well.
> > >>> The code currently uses virt_to_phys(__get_free_pages()) to obtain
> > >>> a reserved page for the MSI message address, and the return value
> > >>> of which can be a 64 bit physical address on 64 bit system.
> > >>>
> > >>> However, the driver only programs PCIEMSIALR register with the bottom
> > >>> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
> > >>> not program the top 32 bits into PCIEMSIAUR, but rather programs the
> > >>> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
> > >>> SoCs, however may fail on new 64 bit R-Car SoCs.
> > >>>
> > >>> Since from a PCIe controller perspective, an inbound MSI is a memory
> > >>> write to a special address (in case of this controller, defined by
> > >>> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
> > >>> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
> > >>> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
> > >>> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
> > >>> cause memory corruption or other issues.
> > >>>
> > >>> There is however the possibility that if virt_to_phys(__get_free_pages())
> > >>> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
> > >>> to 0x0 _and_ if the system had physical RAM at the address matching the
> > >>> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
> > >>> physical address matching the value of PCIEMSIALR and a remote write to
> > >>> such a buffer by a PCIe card would trigger a spurious MSI.
> > >>>
> > >>> Signed-off-by: Marek Vasut <[e-mail]>
> > >>> Cc: Geert Uytterhoeven <[e-mail]>
> > >>> Cc: Phil Edworthy <[e-mail]>
> > >>> Cc: Simon Horman <[e-mail]>
> > >>> Cc: Wolfram Sang <[e-mail]>
> > >>> Cc: [e-mail]
> > >>> To: [e-mail]
> > >>> Reviewed-by: Geert Uytterhoeven <[e-mail]>
> > >>
> > >> Does this warrant a Fixes tag?
> > >
> > > (digging in old sent email)
> > > Fixes: 290c1fb358605402 ("PCI: rcar: Add MSI support for PCIe")
> >
> > But does it really fix that commit, given that on Gen2 and earlier, it
> > was not broken as those were 32bit platforms ?
> 
> It does not fix the bug on that commit, as the bug cannot happen on arm32.
> It does fix that commit, in that that commit used "unsigned long" for a
> physical address, which is wrong, even on arm32 (esp. with LPAE).
> If you insist on having a Fixes tag for a commit where the bug could be
> seen:
> Fixes: e015f88c368da1e6 ("PCI: rcar: Add support for R-Car H3 to pcie-rcar")
> 
> Apart from that, drivers should use the DMA API instead of virt_to_phys().
> However, now we have a better understanding of how MSI interrupts
> work, we don't even need to allocate that page. All we need is the
> physical address of a page that is guaranteed not to be backed by RAM
> (i.e. not to be a valid target for a legitimate PCI bus mastering
> transaction).

Agreed but I would merge this patch first since it is a fix
and update it later.

Shall I go with the Fixes: tag above ?

Thanks,
Lorenzo

> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [e-mail]
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds

Geert Uytterhoeven March 28, 2019, 4:31 p.m. UTC | #6

Hi Lorenzo,

On Thu, Mar 28, 2019 at 5:28 PM Lorenzo Pieralisi
<[e-mail]> wrote:
> On Thu, Mar 28, 2019 at 09:02:00AM +0100, Geert Uytterhoeven wrote:
> > On Thu, Mar 28, 2019 at 4:19 AM Marek Vasut <[e-mail]> wrote:
> > > On 3/27/19 1:22 PM, Geert Uytterhoeven wrote:
> > > > On Wed, Mar 27, 2019 at 12:30 PM Simon Horman <[e-mail]> wrote:
> > > >> On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
> > > >>> From: Marek Vasut <[e-mail]>
> > > >>> The MSI message address in the RC address space can be 64 bit. The
> > > >>> R-Car PCIe RC supports such a 64bit MSI message address as well.
> > > >>> The code currently uses virt_to_phys(__get_free_pages()) to obtain
> > > >>> a reserved page for the MSI message address, and the return value
> > > >>> of which can be a 64 bit physical address on 64 bit system.
> > > >>>
> > > >>> However, the driver only programs PCIEMSIALR register with the bottom
> > > >>> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
> > > >>> not program the top 32 bits into PCIEMSIAUR, but rather programs the
> > > >>> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
> > > >>> SoCs, however may fail on new 64 bit R-Car SoCs.
> > > >>>
> > > >>> Since from a PCIe controller perspective, an inbound MSI is a memory
> > > >>> write to a special address (in case of this controller, defined by
> > > >>> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
> > > >>> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
> > > >>> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
> > > >>> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
> > > >>> cause memory corruption or other issues.
> > > >>>
> > > >>> There is however the possibility that if virt_to_phys(__get_free_pages())
> > > >>> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
> > > >>> to 0x0 _and_ if the system had physical RAM at the address matching the
> > > >>> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
> > > >>> physical address matching the value of PCIEMSIALR and a remote write to
> > > >>> such a buffer by a PCIe card would trigger a spurious MSI.
> > > >>>
> > > >>> Signed-off-by: Marek Vasut <[e-mail]>
> > > >>> Cc: Geert Uytterhoeven <[e-mail]>
> > > >>> Cc: Phil Edworthy <[e-mail]>
> > > >>> Cc: Simon Horman <[e-mail]>
> > > >>> Cc: Wolfram Sang <[e-mail]>
> > > >>> Cc: [e-mail]
> > > >>> To: [e-mail]
> > > >>> Reviewed-by: Geert Uytterhoeven <[e-mail]>
> > > >>
> > > >> Does this warrant a Fixes tag?
> > > >
> > > > (digging in old sent email)
> > > > Fixes: 290c1fb358605402 ("PCI: rcar: Add MSI support for PCIe")
> > >
> > > But does it really fix that commit, given that on Gen2 and earlier, it
> > > was not broken as those were 32bit platforms ?
> >
> > It does not fix the bug on that commit, as the bug cannot happen on arm32.
> > It does fix that commit, in that that commit used "unsigned long" for a
> > physical address, which is wrong, even on arm32 (esp. with LPAE).
> > If you insist on having a Fixes tag for a commit where the bug could be
> > seen:
> > Fixes: e015f88c368da1e6 ("PCI: rcar: Add support for R-Car H3 to pcie-rcar")
> >
> > Apart from that, drivers should use the DMA API instead of virt_to_phys().
> > However, now we have a better understanding of how MSI interrupts
> > work, we don't even need to allocate that page. All we need is the
> > physical address of a page that is guaranteed not to be backed by RAM
> > (i.e. not to be a valid target for a legitimate PCI bus mastering
> > transaction).
>
> Agreed but I would merge this patch first since it is a fix
> and update it later.

Sure, definitely.

> Shall I go with the Fixes: tag above ?

Fine for me, thanks!

Gr{oetje,eeting}s,

                        Geert

Marek Vasut March 29, 2019, 9:53 a.m. UTC | #7

On 3/28/19 5:31 PM, Geert Uytterhoeven wrote:
> Hi Lorenzo,
> 
> On Thu, Mar 28, 2019 at 5:28 PM Lorenzo Pieralisi
> <[e-mail]> wrote:
>> On Thu, Mar 28, 2019 at 09:02:00AM +0100, Geert Uytterhoeven wrote:
>>> On Thu, Mar 28, 2019 at 4:19 AM Marek Vasut <[e-mail]> wrote:
>>>> On 3/27/19 1:22 PM, Geert Uytterhoeven wrote:
>>>>> On Wed, Mar 27, 2019 at 12:30 PM Simon Horman <[e-mail]> wrote:
>>>>>> On Mon, Mar 25, 2019 at 12:41:01PM +0100, [e-mail] wrote:
>>>>>>> From: Marek Vasut <[e-mail]>
>>>>>>> The MSI message address in the RC address space can be 64 bit. The
>>>>>>> R-Car PCIe RC supports such a 64bit MSI message address as well.
>>>>>>> The code currently uses virt_to_phys(__get_free_pages()) to obtain
>>>>>>> a reserved page for the MSI message address, and the return value
>>>>>>> of which can be a 64 bit physical address on 64 bit system.
>>>>>>>
>>>>>>> However, the driver only programs PCIEMSIALR register with the bottom
>>>>>>> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
>>>>>>> not program the top 32 bits into PCIEMSIAUR, but rather programs the
>>>>>>> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
>>>>>>> SoCs, however may fail on new 64 bit R-Car SoCs.
>>>>>>>
>>>>>>> Since from a PCIe controller perspective, an inbound MSI is a memory
>>>>>>> write to a special address (in case of this controller, defined by
>>>>>>> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
>>>>>>> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
>>>>>>> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
>>>>>>> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
>>>>>>> cause memory corruption or other issues.
>>>>>>>
>>>>>>> There is however the possibility that if virt_to_phys(__get_free_pages())
>>>>>>> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
>>>>>>> to 0x0 _and_ if the system had physical RAM at the address matching the
>>>>>>> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
>>>>>>> physical address matching the value of PCIEMSIALR and a remote write to
>>>>>>> such a buffer by a PCIe card would trigger a spurious MSI.
>>>>>>>
>>>>>>> Signed-off-by: Marek Vasut <[e-mail]>
>>>>>>> Cc: Geert Uytterhoeven <[e-mail]>
>>>>>>> Cc: Phil Edworthy <[e-mail]>
>>>>>>> Cc: Simon Horman <[e-mail]>
>>>>>>> Cc: Wolfram Sang <[e-mail]>
>>>>>>> Cc: [e-mail]
>>>>>>> To: [e-mail]
>>>>>>> Reviewed-by: Geert Uytterhoeven <[e-mail]>
>>>>>>
>>>>>> Does this warrant a Fixes tag?
>>>>>
>>>>> (digging in old sent email)
>>>>> Fixes: 290c1fb358605402 ("PCI: rcar: Add MSI support for PCIe")
>>>>
>>>> But does it really fix that commit, given that on Gen2 and earlier, it
>>>> was not broken as those were 32bit platforms ?
>>>
>>> It does not fix the bug on that commit, as the bug cannot happen on arm32.
>>> It does fix that commit, in that that commit used "unsigned long" for a
>>> physical address, which is wrong, even on arm32 (esp. with LPAE).
>>> If you insist on having a Fixes tag for a commit where the bug could be
>>> seen:
>>> Fixes: e015f88c368da1e6 ("PCI: rcar: Add support for R-Car H3 to pcie-rcar")
>>>
>>> Apart from that, drivers should use the DMA API instead of virt_to_phys().
>>> However, now we have a better understanding of how MSI interrupts
>>> work, we don't even need to allocate that page. All we need is the
>>> physical address of a page that is guaranteed not to be backed by RAM
>>> (i.e. not to be a valid target for a legitimate PCI bus mastering
>>> transaction).
>>
>> Agreed but I would merge this patch first since it is a fix
>> and update it later.
> 
> Sure, definitely.
> 
>> Shall I go with the Fixes: tag above ?
> 
> Fine for me, thanks!

I don't feel strongly either way.

Geert Uytterhoeven March 29, 2019, 7:32 p.m. UTC | #8

Hi Marek,

On Mon, Mar 25, 2019 at 12:41 PM <[e-mail]> wrote:
> From: Marek Vasut <[e-mail]>
>
> The MSI message address in the RC address space can be 64 bit. The
> R-Car PCIe RC supports such a 64bit MSI message address as well.
> The code currently uses virt_to_phys(__get_free_pages()) to obtain
> a reserved page for the MSI message address, and the return value
> of which can be a 64 bit physical address on 64 bit system.
>
> However, the driver only programs PCIEMSIALR register with the bottom
> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
> not program the top 32 bits into PCIEMSIAUR, but rather programs the
> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
> SoCs, however may fail on new 64 bit R-Car SoCs.
>
> Since from a PCIe controller perspective, an inbound MSI is a memory
> write to a special address (in case of this controller, defined by
> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
> cause memory corruption or other issues.
>
> There is however the possibility that if virt_to_phys(__get_free_pages())
> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
> to 0x0 _and_ if the system had physical RAM at the address matching the
> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
> physical address matching the value of PCIEMSIALR and a remote write to
> such a buffer by a PCIe card would trigger a spurious MSI.
>
> Signed-off-by: Marek Vasut <[e-mail]>
> Cc: Geert Uytterhoeven <[e-mail]>
> Cc: Phil Edworthy <[e-mail]>
> Cc: Simon Horman <[e-mail]>
> Cc: Wolfram Sang <[e-mail]>
> Cc: [e-mail]
> To: [e-mail]
> Reviewed-by: Geert Uytterhoeven <[e-mail]>
> ---
> V2: - s/it's/its/ in commit message
>     - Add R-B from Geert
> V3: - Reworded commit message and thus dropped Geerts R-B
> V4: - Add Geert's R-B again
> ---
>  drivers/pci/controller/pcie-rcar.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
> index c6013f95bdb2..62d2de9fbf1c 100644
> --- a/drivers/pci/controller/pcie-rcar.c
> +++ b/drivers/pci/controller/pcie-rcar.c
> @@ -890,7 +890,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
>  {
>         struct device *dev = pcie->dev;
>         struct rcar_msi *msi = &pcie->msi;
> -       unsigned long base;
> +       phys_addr_t base;
>         int err, i;
>
>         mutex_init(&msi->lock);
> @@ -932,7 +932,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
>         base = virt_to_phys((void *)msi->pages);
>
>         rcar_pci_write_reg(pcie, base | MSIFE, PCIEMSIALR);
> -       rcar_pci_write_reg(pcie, 0, PCIEMSIAUR);
> +       rcar_pci_write_reg(pcie, base >> 32, PCIEMSIAUR);

As reported by 0day, this causes a warning on arm32 without LPAE:

    drivers/pci/controller/pcie-rcar.c:935:32: warning: right shift
count >= width of type

Using upper_32_bits() instead of an explicit shift should fix that.


Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [e-mail]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Marek Vasut March 30, 2019, 7:45 a.m. UTC | #9

On 3/29/19 8:32 PM, Geert Uytterhoeven wrote:
> Hi Marek,
> 
> On Mon, Mar 25, 2019 at 12:41 PM <[e-mail]> wrote:
>> From: Marek Vasut <[e-mail]>
>>
>> The MSI message address in the RC address space can be 64 bit. The
>> R-Car PCIe RC supports such a 64bit MSI message address as well.
>> The code currently uses virt_to_phys(__get_free_pages()) to obtain
>> a reserved page for the MSI message address, and the return value
>> of which can be a 64 bit physical address on 64 bit system.
>>
>> However, the driver only programs PCIEMSIALR register with the bottom
>> 32 bits of the virt_to_phys(__get_free_pages()) return value and does
>> not program the top 32 bits into PCIEMSIAUR, but rather programs the
>> PCIEMSIAUR register with 0x0. This worked fine on older 32 bit R-Car
>> SoCs, however may fail on new 64 bit R-Car SoCs.
>>
>> Since from a PCIe controller perspective, an inbound MSI is a memory
>> write to a special address (in case of this controller, defined by
>> the value in PCIEMSIAUR:PCIEMSIALR), which triggers an interrupt, but
>> never hits the DRAM _and_ because allocation of an MSI by a PCIe card
>> driver obtains the MSI message address by reading PCIEMSIAUR:PCIEMSIALR
>> in rcar_msi_setup_irqs(), incorrectly programmed PCIEMSIAUR cannot
>> cause memory corruption or other issues.
>>
>> There is however the possibility that if virt_to_phys(__get_free_pages())
>> returned address above the 32bit boundary _and_ PCIEMSIAUR was programmed
>> to 0x0 _and_ if the system had physical RAM at the address matching the
>> value of PCIEMSIALR, a PCIe card driver could allocate a buffer with a
>> physical address matching the value of PCIEMSIALR and a remote write to
>> such a buffer by a PCIe card would trigger a spurious MSI.
>>
>> Signed-off-by: Marek Vasut <[e-mail]>
>> Cc: Geert Uytterhoeven <[e-mail]>
>> Cc: Phil Edworthy <[e-mail]>
>> Cc: Simon Horman <[e-mail]>
>> Cc: Wolfram Sang <[e-mail]>
>> Cc: [e-mail]
>> To: [e-mail]
>> Reviewed-by: Geert Uytterhoeven <[e-mail]>
>> ---
>> V2: - s/it's/its/ in commit message
>>     - Add R-B from Geert
>> V3: - Reworded commit message and thus dropped Geerts R-B
>> V4: - Add Geert's R-B again
>> ---
>>  drivers/pci/controller/pcie-rcar.c | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
>> index c6013f95bdb2..62d2de9fbf1c 100644
>> --- a/drivers/pci/controller/pcie-rcar.c
>> +++ b/drivers/pci/controller/pcie-rcar.c
>> @@ -890,7 +890,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
>>  {
>>         struct device *dev = pcie->dev;
>>         struct rcar_msi *msi = &pcie->msi;
>> -       unsigned long base;
>> +       phys_addr_t base;
>>         int err, i;
>>
>>         mutex_init(&msi->lock);
>> @@ -932,7 +932,7 @@ static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
>>         base = virt_to_phys((void *)msi->pages);
>>
>>         rcar_pci_write_reg(pcie, base | MSIFE, PCIEMSIALR);
>> -       rcar_pci_write_reg(pcie, 0, PCIEMSIAUR);
>> +       rcar_pci_write_reg(pcie, base >> 32, PCIEMSIAUR);
> 
> As reported by 0day, this causes a warning on arm32 without LPAE:
> 
>     drivers/pci/controller/pcie-rcar.c:935:32: warning: right shift
> count >= width of type
> 
> Using upper_32_bits() instead of an explicit shift should fix that.

I saw the report too.

Lorenzo, do you want a separate patch to squash with this or V5 ?

Patch

diff --git a/drivers/pci/controller/pcie-rcar.c b/drivers/pci/controller/pcie-rcar.c
index c6013f95bdb2..62d2de9fbf1c 100644
--- a/drivers/pci/controller/pcie-rcar.c
+++ b/drivers/pci/controller/pcie-rcar.c
@@ -890,7 +890,7 @@  static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
 	struct rcar_msi *msi = &pcie->msi;
-	unsigned long base;
+	phys_addr_t base;
 	int err, i;
 
 	mutex_init(&msi->lock);
@@ -932,7 +932,7 @@  static int rcar_pcie_enable_msi(struct rcar_pcie *pcie)
 	base = virt_to_phys((void *)msi->pages);
 
 	rcar_pci_write_reg(pcie, base | MSIFE, PCIEMSIALR);
-	rcar_pci_write_reg(pcie, 0, PCIEMSIAUR);
+	rcar_pci_write_reg(pcie, base >> 32, PCIEMSIAUR);
 
 	/* enable all MSI interrupts */
 	rcar_pci_write_reg(pcie, 0xffffffff, PCIEMSIIER);