| Message ID | 20180921195954.21574-1-marc.zyngier@arm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | GICv3 support for kexec/kdump on EFI systems | expand |
Hi, On 09/21/2018 02:59 PM, Marc Zyngier wrote: > The GICv3 architecture has the remarkable feature that once LPI tables > have been assigned to redistributors and that LPI delivery is enabled, > there is no guarantee that LPIs can be turned off (and most > implementations do not allow it), nor can it be reprogrammed to use > other tables. > > This is a bit of a problem for kexec, where the secondary kernel > completely looses track of the previous allocations. If the secondary > kernel doesn't allocate the tables exactly the same way, no LPIs will > be delivered by the GIC (which continues to use the old tables), and > memory previously allocated for the pending tables will be slowly > corrupted, one bit at a time. > > The workaround for this is based on a series[1] by Ard Biesheuvel, > which adds the required infrastructure for memory reservations to be > passed from one kernel to another using an EFI table. > > This infrastructure is then used to register the allocation of GIC > tables with EFI, and allow the GIC driver to safely reuse the existing > programming if it detects that the tables have been correctly > registered. On non-EFI systems, there is not much we can do. > > This has been tested on a TX2 system both as a host and a guest. I'd > welcome additional testing of different HW. For convenience, I've > stashed a branch containing the whole thing at [2]. When combined with Ard's patch set, this fixes kdump on a QC machine. Tested-by: Jeremy Linton <jeremy.linton@arm.com> Thanks, > > [1] https://marc.info/?l=linux-efi&m=153754757208163&w=2 > [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/gicv3-kdump > > Marc Zyngier (10): > irqchip/gic-v3-its: Change initialization ordering for LPIs > irqchip/gic-v3-its: Consolidate LPI_PENDBASE_SZ usage > irqchip/gic-v3-its: Split property table clearing from allocation > irqchip/gic-v3-its: Move pending table allocation to init time > irqchip/gic-v3-its: Keep track of property table's PA and VA > irqchip/gic-v3-its: Allow use of pre-programmed LPI tables > irqchip/gic-v3-its: Use pre-programmed redistributor tables with kdump > kernels > irqchip/gic-v3-its: Check that all RDs have the same property table > irqchip/gic-v3-its: Register LPI tables with EFI config table > irqchip/gic-v3-its: Allow use of LPI tables in reserved memory > > drivers/irqchip/irq-gic-v3-its.c | 249 ++++++++++++++++++++++------- > drivers/irqchip/irq-gic-v3.c | 20 ++- > include/linux/irqchip/arm-gic-v3.h | 4 +- > 3 files changed, 208 insertions(+), 65 deletions(-) >
Hi Marc, On 09/22/2018 01:29 AM, Marc Zyngier wrote: > The GICv3 architecture has the remarkable feature that once LPI tables > have been assigned to redistributors and that LPI delivery is enabled, > there is no guarantee that LPIs can be turned off (and most > implementations do not allow it), nor can it be reprogrammed to use > other tables. > > This is a bit of a problem for kexec, where the secondary kernel > completely looses track of the previous allocations. If the secondary > kernel doesn't allocate the tables exactly the same way, no LPIs will > be delivered by the GIC (which continues to use the old tables), and > memory previously allocated for the pending tables will be slowly > corrupted, one bit at a time. > > The workaround for this is based on a series[1] by Ard Biesheuvel, > which adds the required infrastructure for memory reservations to be > passed from one kernel to another using an EFI table. > > This infrastructure is then used to register the allocation of GIC > tables with EFI, and allow the GIC driver to safely reuse the existing > programming if it detects that the tables have been correctly > registered. On non-EFI systems, there is not much we can do. > > This has been tested on a TX2 system both as a host and a guest. I'd > welcome additional testing of different HW. For convenience, I've > stashed a branch containing the whole thing at [2]. > > [1] https://marc.info/?l=linux-efi&m=153754757208163&w=2 > [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/gicv3-kdump > > Marc Zyngier (10): > irqchip/gic-v3-its: Change initialization ordering for LPIs > irqchip/gic-v3-its: Consolidate LPI_PENDBASE_SZ usage > irqchip/gic-v3-its: Split property table clearing from allocation > irqchip/gic-v3-its: Move pending table allocation to init time > irqchip/gic-v3-its: Keep track of property table's PA and VA > irqchip/gic-v3-its: Allow use of pre-programmed LPI tables > irqchip/gic-v3-its: Use pre-programmed redistributor tables with kdump > kernels > irqchip/gic-v3-its: Check that all RDs have the same property table > irqchip/gic-v3-its: Register LPI tables with EFI config table > irqchip/gic-v3-its: Allow use of LPI tables in reserved memory > > drivers/irqchip/irq-gic-v3-its.c | 249 ++++++++++++++++++++++------- > drivers/irqchip/irq-gic-v3.c | 20 ++- > include/linux/irqchip/arm-gic-v3.h | 4 +- > 3 files changed, 208 insertions(+), 65 deletions(-) Thanks for the patchset. I can confirm that with Ard's patchset in [1] and this patchset applied on 'efi/next' branch, I see that the "Booted with LPIs enabled, memory probably corrupted" issue that I was seeing on gigabyte boards in kdump kernel is fixed. Here are some logs: without patchset applied: ========================= [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode [ 0.000000] GICv3: Distributor has no Range Selector support [ 0.000000] GICv3: no VLPI support, direct LPI support [ 0.000000] ITS [mem 0x801000020000-0x80100021ffff] [ 0.000000] ITS@0x0000801000020000: allocated 2097152 Devices @c1000000 (flat, esz 8, psz 64K, shr 1) [ 0.000000] GIC: using LPI property table @0x00000000c03b0000 [ 0.000000] GICv3: CPU0: found redistributor a region 0:0x0000801080140000 [ 0.000000] CPU0: Booted with LPIs enabled, memory probably corrupted [ 0.000000] CPU0: Failed to disable LPIs <..snip..> [ 198.702976] dracut-initqueue[298]: Warning: dracut-initqueue timeout - starting timeout scripts [ 199.332238] dracut-initqueue[298]: Warning: dracut-initqueue timeout - starting timeout scripts [ 199.922944] dracut-initqueue[298]: Warning: dracut-initqueue timeout - starting timeout scripts [ 200.512239] dracut-initqueue[298]: Warning: dracut-initqueue timeout - starting timeout scripts <..snip..> with patchset applied: ====================== [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode [ 0.000000] GICv3: Distributor has no Range Selector support [ 0.000000] GICv3: no VLPI support, direct LPI support [ 0.000000] GICv3: CPU0: found redistributor 109 region 0:0x0000801080320000 [ 0.000000] ITS [mem 0x801000020000-0x80100021ffff] [ 0.000000] ITS@0x0000801000020000: allocated 2097152 Devices @c1000000 (flat, esz 8, psz 64K, shr 1) [ 0.000000] GICv3: Using preallocated redistributor tables [ 0.000000] GICv3: using LPI property table @0x0000000fc0420000 [ 0.000000] GICv3: CPU0: using reserved LPI pending table @0x0000000fc05c0000 So, please feel to add: Tested-by: Bhupesh Sharma <bhsharma@redhat.com> Thanks, Bhupesh
Hi Marc > -----Original Message----- > From: linux-arm-kernel > [mailto:linux-arm-kernel-bounces@lists.infradead.org] On Behalf Of Marc > Zyngier > Sent: Saturday, September 22, 2018 5:00 AM > To: linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org > Cc: Jeffrey Hugo; Thomas Gleixner; Jason Cooper; Jeremy Linton; Ard > Biesheuvel > Subject: [PATCH 00/10] GICv3 support for kexec/kdump on EFI systems > > The GICv3 architecture has the remarkable feature that once LPI tables > have been assigned to redistributors and that LPI delivery is enabled, > there is no guarantee that LPIs can be turned off (and most > implementations do not allow it), nor can it be reprogrammed to use > other tables. > > This is a bit of a problem for kexec, where the secondary kernel > completely looses track of the previous allocations. If the secondary > kernel doesn't allocate the tables exactly the same way, no LPIs will > be delivered by the GIC (which continues to use the old tables), and > memory previously allocated for the pending tables will be slowly > corrupted, one bit at a time. > > The workaround for this is based on a series[1] by Ard Biesheuvel, > which adds the required infrastructure for memory reservations to be > passed from one kernel to another using an EFI table. > > This infrastructure is then used to register the allocation of GIC > tables with EFI, and allow the GIC driver to safely reuse the existing > programming if it detects that the tables have been correctly > registered. On non-EFI systems, there is not much we can do. > > This has been tested on a TX2 system both as a host and a guest. I'd > welcome additional testing of different HW. For convenience, I've > stashed a branch containing the whole thing at [2]. We have done the test on our chip A64FX that When a write changes EnableLPI bit from 0 to 1, this bit becomes RES1. The result is that the kexec operation successfully works on our chip, and PCI based on LPI also works after kexec. For detail: We did "kexec -e" command, and the message, "Using preallocated redistributor tables", was shown. After kexec, we can use our ssd normally. Test environment CPU: A64FX Kernel version: v4.19 rc4 base https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/gicv3-kdump 8bc67da irqchip/gic-v3-its: Allow use of LPI tables in reserved memory kexec version:kexec-tools-2.0.14-17.2.el7.aarch64 Tested-by: Lei Zhang <zhang.lei@jp.fujitsu.com> Thanks a lot. Best Regards, Lei,Zhang FUJITSU LIMITED.
Hi Marc On 9/21/2018 1:59 PM, Marc Zyngier wrote: > The GICv3 architecture has the remarkable feature that once LPI tables > have been assigned to redistributors and that LPI delivery is enabled, > there is no guarantee that LPIs can be turned off (and most > implementations do not allow it), nor can it be reprogrammed to use > other tables. > > This is a bit of a problem for kexec, where the secondary kernel > completely looses track of the previous allocations. If the secondary > kernel doesn't allocate the tables exactly the same way, no LPIs will > be delivered by the GIC (which continues to use the old tables), and > memory previously allocated for the pending tables will be slowly > corrupted, one bit at a time. > > The workaround for this is based on a series[1] by Ard Biesheuvel, > which adds the required infrastructure for memory reservations to be > passed from one kernel to another using an EFI table. > > This infrastructure is then used to register the allocation of GIC > tables with EFI, and allow the GIC driver to safely reuse the existing > programming if it detects that the tables have been correctly > registered. On non-EFI systems, there is not much we can do. > > This has been tested on a TX2 system both as a host and a guest. I'd > welcome additional testing of different HW. For convenience, I've > stashed a branch containing the whole thing at [2]. I tested [2] from the 4.19-rc4 set which included this series and [1]. Tested kexec on Centriq system with ITS support (46 core). On-board was a MLX CX5 NIC, verified MSIs are active in /proc/interrupts. Prior to this we used a workaround from Shanker to reuse the same tables in the kexec'ed kernel. Let me know if further testing is needed, and thanks for adding this support. > [1] https://marc.info/?l=linux-efi&m=153754757208163&w=2 > [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/gicv3-kdump > > Marc Zyngier (10): > irqchip/gic-v3-its: Change initialization ordering for LPIs > irqchip/gic-v3-its: Consolidate LPI_PENDBASE_SZ usage > irqchip/gic-v3-its: Split property table clearing from allocation > irqchip/gic-v3-its: Move pending table allocation to init time > irqchip/gic-v3-its: Keep track of property table's PA and VA > irqchip/gic-v3-its: Allow use of pre-programmed LPI tables > irqchip/gic-v3-its: Use pre-programmed redistributor tables with kdump > kernels > irqchip/gic-v3-its: Check that all RDs have the same property table > irqchip/gic-v3-its: Register LPI tables with EFI config table > irqchip/gic-v3-its: Allow use of LPI tables in reserved memory > > drivers/irqchip/irq-gic-v3-its.c | 249 ++++++++++++++++++++++------- > drivers/irqchip/irq-gic-v3.c | 20 ++- > include/linux/irqchip/arm-gic-v3.h | 4 +- > 3 files changed, 208 insertions(+), 65 deletions(-) >
Hi Richard, On 27/09/18 22:10, Richard Ruigrok wrote: > Hi Marc > > On 9/21/2018 1:59 PM, Marc Zyngier wrote: >> The GICv3 architecture has the remarkable feature that once LPI tables >> have been assigned to redistributors and that LPI delivery is enabled, >> there is no guarantee that LPIs can be turned off (and most >> implementations do not allow it), nor can it be reprogrammed to use >> other tables. >> >> This is a bit of a problem for kexec, where the secondary kernel >> completely looses track of the previous allocations. If the secondary >> kernel doesn't allocate the tables exactly the same way, no LPIs will >> be delivered by the GIC (which continues to use the old tables), and >> memory previously allocated for the pending tables will be slowly >> corrupted, one bit at a time. >> >> The workaround for this is based on a series[1] by Ard Biesheuvel, >> which adds the required infrastructure for memory reservations to be >> passed from one kernel to another using an EFI table. >> >> This infrastructure is then used to register the allocation of GIC >> tables with EFI, and allow the GIC driver to safely reuse the existing >> programming if it detects that the tables have been correctly >> registered. On non-EFI systems, there is not much we can do. >> >> This has been tested on a TX2 system both as a host and a guest. I'd >> welcome additional testing of different HW. For convenience, I've >> stashed a branch containing the whole thing at [2]. > I tested [2] from the 4.19-rc4 set which included this series and [1]. > Tested kexec on Centriq system with ITS support (46 core). On-board was a MLX CX5 NIC, verified MSIs are active in /proc/interrupts. > Prior to this we used a workaround from Shanker to reuse the same tables in the kexec'ed kernel. Yes, I remember seeing this workaround. Hopefully we're in a better place now that we can guarantee that the tables are not reused. > Let me know if further testing is needed, and thanks for adding this support. Good to know, thanks for having tested it. I've now put this code into -next for some more soaking. Hopefully nothing horrible will happen ;-) M.
Hi Xulin, On 01/02/2019 06:11, Sun Ted wrote: > Hi Marc, > > Marc Zyngier <marc.zyngier@arm.com <mailto:marc.zyngier@arm.com>> 于2018 > 年9月22日周六 上午4:03写道： > > The GICv3 architecture has the remarkable feature that once LPI tables > have been assigned to redistributors and that LPI delivery is enabled, > there is no guarantee that LPIs can be turned off (and most > implementations do not allow it), nor can it be reprogrammed to use > other tables. > > This is a bit of a problem for kexec, where the secondary kernel > completely looses track of the previous allocations. If the secondary > kernel doesn't allocate the tables exactly the same way, no LPIs will > be delivered by the GIC (which continues to use the old tables), and > memory previously allocated for the pending tables will be slowly > corrupted, one bit at a time. > > The workaround for this is based on a series[1] by Ard Biesheuvel, > which adds the required infrastructure for memory reservations to be > passed from one kernel to another using an EFI table. > > This infrastructure is then used to register the allocation of GIC > tables with EFI, and allow the GIC driver to safely reuse the existing > programming if it detects that the tables have been correctly > registered. On non-EFI systems, there is not much we can do. > > > Sorry to turn this question out again. > For others non-EFI systems, just as your said till now we did do > anything, right? We didn't do anything, because there is nothing we can do. > I did see the kexec booting failure since re-setting > the GICR_CTLR.EnableLPI from "1" to "0" unsuccessful. > > The below commit adds the judgement for disabling LPIs, and returned > error. Caused "kexec" booting failure. And I fully expected this. When I said "we don't do anything", I meant "we don't do anything to make LPIs work". > > 6eb486b66a (irqchip/gic-v3: Ensure GICR_CTLR.EnableLPI=0 is observed > before enabling). > > <snip patch> > int its_cpu_init(void) > { > if (!list_empty(&its_nodes)) { > - if (!gic_rdists_supports_plpis()) { > - pr_info("CPU%d: LPIs not supported\n", > smp_processor_id()); > - return -ENXIO; > - } > + int ret; > + > + ret = redist_disable_lpis(); > + if (ret) > + return ret; > + And I maintain that this is the right thing to do. If LPIs are on and the memory has not been reserved, it is then likely that this memory is now being used by the kernel for something else. The system is thus going to see single-bit corruption in some random places. At that stage, your system is horribly unsafe, and I will not let it boot under these conditions. If it was working before, that's because you were lucky, and I place no faith in luck. Now you have two alternatives: - You switch to an EFI-based firmware. These days, even u-boot has an EFI implementation. COnsider doing that if you can. - If there is no EFI implementation for your SoC, you can pass the "irqchip.gicv3_nolpi=1" option to the first kernel. This will keep LPI disabled, and you'll be able to kexec a secondary kernel (and get working LPIs there). This is what I do on my Chromebook. Thanks, M.
On 02/01/2019 05:15 PM, Marc Zyngier wrote: > Hi Xulin, > > On 01/02/2019 06:11, Sun Ted wrote: >> Hi Marc, >> >> Marc Zyngier <marc.zyngier@arm.com <mailto:marc.zyngier@arm.com>> 于2018 >> 年9月22日周六 上午4:03写道： >> >> The GICv3 architecture has the remarkable feature that once LPI tables >> have been assigned to redistributors and that LPI delivery is enabled, >> there is no guarantee that LPIs can be turned off (and most >> implementations do not allow it), nor can it be reprogrammed to use >> other tables. >> >> This is a bit of a problem for kexec, where the secondary kernel >> completely looses track of the previous allocations. If the secondary >> kernel doesn't allocate the tables exactly the same way, no LPIs will >> be delivered by the GIC (which continues to use the old tables), and >> memory previously allocated for the pending tables will be slowly >> corrupted, one bit at a time. >> >> The workaround for this is based on a series[1] by Ard Biesheuvel, >> which adds the required infrastructure for memory reservations to be >> passed from one kernel to another using an EFI table. >> >> This infrastructure is then used to register the allocation of GIC >> tables with EFI, and allow the GIC driver to safely reuse the existing >> programming if it detects that the tables have been correctly >> registered. On non-EFI systems, there is not much we can do. >> >> >> Sorry to turn this question out again. >> For others non-EFI systems, just as your said till now we did do >> anything, right? > We didn't do anything, because there is nothing we can do. > >> I did see the kexec booting failure since re-setting >> the GICR_CTLR.EnableLPI from "1" to "0" unsuccessful. >> >> The below commit adds the judgement for disabling LPIs, and returned >> error. Caused "kexec" booting failure. > And I fully expected this. When I said "we don't do anything", I meant > "we don't do anything to make LPIs work". > >> 6eb486b66a (irqchip/gic-v3: Ensure GICR_CTLR.EnableLPI=0 is observed >> before enabling). >> >> <snip patch> >> int its_cpu_init(void) >> { >> if (!list_empty(&its_nodes)) { >> - if (!gic_rdists_supports_plpis()) { >> - pr_info("CPU%d: LPIs not supported\n", >> smp_processor_id()); >> - return -ENXIO; >> - } >> + int ret; >> + >> + ret = redist_disable_lpis(); >> + if (ret) >> + return ret; >> + > And I maintain that this is the right thing to do. If LPIs are on and > the memory has not been reserved, it is then likely that this memory is > now being used by the kernel for something else. The system is thus > going to see single-bit corruption in some random places. > > At that stage, your system is horribly unsafe, and I will not let it > boot under these conditions. If it was working before, that's because > you were lucky, and I place no faith in luck. > > Now you have two alternatives: > > - You switch to an EFI-based firmware. These days, even u-boot has an > EFI implementation. COnsider doing that if you can. > > - If there is no EFI implementation for your SoC, you can pass the > "irqchip.gicv3_nolpi=1" option to the first kernel. This will keep LPI > disabled, and you'll be able to kexec a secondary kernel (and get > working LPIs there). This is what I do on my Chromebook. Hi Marc, Thanks for detailed explanation. Thanks Xulin > > Thanks, > > M.
The GICv3 architecture has the remarkable feature that once LPI tables have been assigned to redistributors and that LPI delivery is enabled, there is no guarantee that LPIs can be turned off (and most implementations do not allow it), nor can it be reprogrammed to use other tables. This is a bit of a problem for kexec, where the secondary kernel completely looses track of the previous allocations. If the secondary kernel doesn't allocate the tables exactly the same way, no LPIs will be delivered by the GIC (which continues to use the old tables), and memory previously allocated for the pending tables will be slowly corrupted, one bit at a time. The workaround for this is based on a series[1] by Ard Biesheuvel, which adds the required infrastructure for memory reservations to be passed from one kernel to another using an EFI table. This infrastructure is then used to register the allocation of GIC tables with EFI, and allow the GIC driver to safely reuse the existing programming if it detects that the tables have been correctly registered. On non-EFI systems, there is not much we can do. This has been tested on a TX2 system both as a host and a guest. I'd welcome additional testing of different HW. For convenience, I've stashed a branch containing the whole thing at [2]. [1] https://marc.info/?l=linux-efi&m=153754757208163&w=2 [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/gicv3-kdump Marc Zyngier (10): irqchip/gic-v3-its: Change initialization ordering for LPIs irqchip/gic-v3-its: Consolidate LPI_PENDBASE_SZ usage irqchip/gic-v3-its: Split property table clearing from allocation irqchip/gic-v3-its: Move pending table allocation to init time irqchip/gic-v3-its: Keep track of property table's PA and VA irqchip/gic-v3-its: Allow use of pre-programmed LPI tables irqchip/gic-v3-its: Use pre-programmed redistributor tables with kdump kernels irqchip/gic-v3-its: Check that all RDs have the same property table irqchip/gic-v3-its: Register LPI tables with EFI config table irqchip/gic-v3-its: Allow use of LPI tables in reserved memory drivers/irqchip/irq-gic-v3-its.c | 249 ++++++++++++++++++++++------- drivers/irqchip/irq-gic-v3.c | 20 ++- include/linux/irqchip/arm-gic-v3.h | 4 +- 3 files changed, 208 insertions(+), 65 deletions(-)