| Message ID | 1604317487-14543-1-git-send-email-sumit.garg@linaro.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | arm64: Add framework to turn an IPI as NMI | expand |
Hi Marc, On Mon, 2 Nov 2020 at 17:15, Sumit Garg <sumit.garg@linaro.org> wrote: > > With pseudo NMIs support available its possible to configure SGIs to be > triggered as pseudo NMIs running in NMI context. And kernel features > such as: > - NMI backtrace can leverage IPI turned as NMI to get a backtrace of CPU > stuck in hard lockup using magic SYSRQ. > - kgdb relies on NMI support to round up CPUs which are stuck in hard > lockup state with interrupts disabled. > > This patch-set adds framework to turn an IPI as NMI which can be triggered > as a pseudo NMI which in turn invokes registered NMI handlers. > > After this patch-set we should be able to get a backtrace for a CPU > stuck in HARDLOCKUP. Have a look at an examples below from a hard lockup > testcase run on Developerbox: > > $ echo HARDLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT > > NMI backtrace: > ============== > > # Issue Magic SysRq to dump backtrace > > [ 376.894502] NMI backtrace for cpu 8 > [ 376.894506] CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > [ 376.894510] Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > [ 376.894514] pstate: 40000005 (nZcv daif -PAN -UAO BTYPE=--) > [ 376.894517] pc : lkdtm_HARDLOCKUP+0x8/0x18 > [ 376.894520] lr : lkdtm_do_action+0x24/0x30 > [ 376.894524] sp : ffff800012cebd20 > [ 376.894527] pmr_save: 00000060 > [ 376.894530] x29: ffff800012cebd20 x28: ffff000875ae8000 > [ 376.894540] x27: 0000000000000000 x26: 0000000000000000 > [ 376.894550] x25: 000000000000001a x24: ffff800012cebe40 > [ 376.894560] x23: 000000000000000b x22: ffff800010fc5040 > [ 376.894569] x21: ffff000878b61000 x20: ffff8000113b2870 > [ 376.894579] x19: 000000000000001b x18: 0000000000000010 > [ 376.894588] x17: 0000000000000000 x16: 0000000000000000 > [ 376.894598] x15: ffff000875ae8470 x14: 00000000000002ad > [ 376.894613] x13: 0000000000000000 x12: 0000000000000000 > [ 376.894622] x11: 0000000000000007 x10: 00000000000009c0 > [ 376.894631] x9 : ffff800012ceba80 x8 : ffff000875ae8a20 > [ 376.894641] x7 : ffff00087f6b3280 x6 : ffff00087f6b3200 > [ 376.894651] x5 : 0000000000000000 x4 : ffff00087f6a91f8 > [ 376.894660] x3 : ffff00087f6b0120 x2 : 1aa310cec69eb500 > [ 376.894670] x1 : 0000000000000000 x0 : 0000000000000060 > [ 376.894679] Call trace: > [ 376.894683] lkdtm_HARDLOCKUP+0x8/0x18 > [ 376.894686] direct_entry+0x124/0x1c0 > [ 376.894689] full_proxy_write+0x60/0xb0 > [ 376.894693] vfs_write+0xf0/0x230 > [ 376.894696] ksys_write+0x6c/0xf8 > [ 376.894699] __arm64_sys_write+0x1c/0x28 > [ 376.894703] el0_svc_common.constprop.0+0x74/0x1f0 > [ 376.894707] do_el0_svc+0x24/0x90 > [ 376.894710] el0_sync_handler+0x180/0x2f8 > [ 376.894713] el0_sync+0x158/0x180 > > KGDB: > ===== > > # Enter kdb via Magic SysRq > > [6]kdb> btc > btc: cpu status: Currently on cpu 6 > Available cpus: 0-5(I), 6, 7(I), 8, 9-23(I) > <snip> > Stack traceback for pid 555 > 0xffff000875ae8000 555 554 1 8 R 0xffff000875ae89c0 bash > CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > Call trace: > dump_backtrace+0x0/0x1a0 > show_stack+0x18/0x28 > dump_stack+0xc0/0x11c > kgdb_cpu_enter+0x648/0x660 > kgdb_nmicallback+0xa0/0xa8 > ipi_kgdb_nmicallback+0x24/0x30 > ipi_nmi_handler+0x48/0x60 > handle_percpu_devid_fasteoi_ipi+0x74/0x88 > generic_handle_irq+0x30/0x48 > handle_domain_nmi+0x48/0x80 > gic_handle_irq+0x18c/0x34c > el1_irq+0xcc/0x180 > lkdtm_HARDLOCKUP+0x8/0x18 > direct_entry+0x124/0x1c0 > full_proxy_write+0x60/0xb0 > vfs_write+0xf0/0x230 > ksys_write+0x6c/0xf8 > __arm64_sys_write+0x1c/0x28 > el0_svc_common.constprop.0+0x74/0x1f0 > do_el0_svc+0x24/0x90 > el0_sync_handler+0x180/0x2f8 > el0_sync+0x158/0x180 > <snip> > > Changes in v7: > - Add a new library function: kgdb_smp_call_nmi_hook() to expose fallback > mechanism to arch specific code. > - Addressed other misc comments from Daniel. > > Changes in v6: > - Two new patches: #4 and #6 which adds runtime fallback framework for > sysrq backtrace and kgdb roundup features. > - Reversed order of NMI backtrace and kgdb roundup feaure patches. > - Addressed other misc. comments from Marc. > - I haven't picked any tags from v5 since I think there is major rework > involved. Masayoshi, could you please confirm if these features still > work for you? > > Changes in v5: > - Rebased to head of upstream master. > - Remove redundant invocation of ipi_nmi_setup(). > - Addressed misc. comments. > > Changes in v4: > - Move IPI NMI framework to a separate file. > - Get rid of hard-coded IPI_CALL_NMI_FUNC allocation. > - Add NMI backtrace support leveraged via magic SYSRQ. > > Changes in v3: > - Rebased to Marc's latest IPIs patch-set [1]. > > [1] https://lkml.org/lkml/2020/9/1/603 > > Changes since RFC version [1]: > - Switch to use generic interrupt framework to turn an IPI as NMI. > - Dependent on Marc's patch-set [2] which turns IPIs into normal > interrupts. > - Addressed misc. comments from Doug on patch #4. > - Posted kgdb NMI printk() fixup separately which has evolved since > to be solved using different approach via changing kgdb interception > of printk() in common printk() code (see patch [3]). > > [1] https://lkml.org/lkml/2020/4/24/328 > [2] https://lkml.org/lkml/2020/5/19/710 > [3] https://lkml.org/lkml/2020/5/20/418 > > Sumit Garg (7): > arm64: Add framework to turn IPI as NMI > irqchip/gic-v3: Enable support for SGIs to act as NMIs > arm64: smp: Assign and setup an IPI as NMI > nmi: backtrace: Allow runtime arch specific override > arm64: ipi_nmi: Add support for NMI backtrace > kgdb: Expose default CPUs roundup fallback mechanism > arm64: kgdb: Roundup cpus using IPI as NMI > Do you have any further feedback on this patch-set? -Sumit > arch/arm/include/asm/irq.h | 2 +- > arch/arm/kernel/smp.c | 3 +- > arch/arm64/include/asm/irq.h | 6 +++ > arch/arm64/include/asm/nmi.h | 17 ++++++++ > arch/arm64/kernel/Makefile | 2 +- > arch/arm64/kernel/ipi_nmi.c | 84 ++++++++++++++++++++++++++++++++++++++++ > arch/arm64/kernel/kgdb.c | 18 +++++++++ > arch/arm64/kernel/smp.c | 8 ++++ > arch/mips/include/asm/irq.h | 2 +- > arch/mips/kernel/process.c | 3 +- > arch/powerpc/include/asm/nmi.h | 2 +- > arch/powerpc/kernel/stacktrace.c | 3 +- > arch/sparc/include/asm/irq_64.h | 2 +- > arch/sparc/kernel/process_64.c | 4 +- > arch/x86/include/asm/irq.h | 2 +- > arch/x86/kernel/apic/hw_nmi.c | 3 +- > drivers/irqchip/irq-gic-v3.c | 29 ++++++++++---- > include/linux/kgdb.h | 12 ++++++ > include/linux/nmi.h | 12 ++---- > kernel/debug/debug_core.c | 8 +++- > 20 files changed, 194 insertions(+), 28 deletions(-) > create mode 100644 arch/arm64/include/asm/nmi.h > create mode 100644 arch/arm64/kernel/ipi_nmi.c > > -- > 2.7.4 >
On 2021-01-05 10:34, Sumit Garg wrote:
> Do you have any further feedback on this patch-set?
None at the moment. We have tons of issues to solve with the arm64
interrupt entry code vs instrumentation at the moment, so it is
pretty much at the bottom of the priority list for now.
Thanks,
M.
On Mon, Nov 2, 2020 at 7:46 PM Sumit Garg <sumit.garg@linaro.org> wrote: > > With pseudo NMIs support available its possible to configure SGIs to be > triggered as pseudo NMIs running in NMI context. And kernel features > such as: > - NMI backtrace can leverage IPI turned as NMI to get a backtrace of CPU > stuck in hard lockup using magic SYSRQ. > - kgdb relies on NMI support to round up CPUs which are stuck in hard > lockup state with interrupts disabled. > > This patch-set adds framework to turn an IPI as NMI which can be triggered > as a pseudo NMI which in turn invokes registered NMI handlers. > > After this patch-set we should be able to get a backtrace for a CPU > stuck in HARDLOCKUP. Have a look at an examples below from a hard lockup > testcase run on Developerbox: > > $ echo HARDLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT > > NMI backtrace: > ============== > > # Issue Magic SysRq to dump backtrace > > [ 376.894502] NMI backtrace for cpu 8 > [ 376.894506] CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > [ 376.894510] Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > [ 376.894514] pstate: 40000005 (nZcv daif -PAN -UAO BTYPE=--) > [ 376.894517] pc : lkdtm_HARDLOCKUP+0x8/0x18 > [ 376.894520] lr : lkdtm_do_action+0x24/0x30 > [ 376.894524] sp : ffff800012cebd20 > [ 376.894527] pmr_save: 00000060 > [ 376.894530] x29: ffff800012cebd20 x28: ffff000875ae8000 > [ 376.894540] x27: 0000000000000000 x26: 0000000000000000 > [ 376.894550] x25: 000000000000001a x24: ffff800012cebe40 > [ 376.894560] x23: 000000000000000b x22: ffff800010fc5040 > [ 376.894569] x21: ffff000878b61000 x20: ffff8000113b2870 > [ 376.894579] x19: 000000000000001b x18: 0000000000000010 > [ 376.894588] x17: 0000000000000000 x16: 0000000000000000 > [ 376.894598] x15: ffff000875ae8470 x14: 00000000000002ad > [ 376.894613] x13: 0000000000000000 x12: 0000000000000000 > [ 376.894622] x11: 0000000000000007 x10: 00000000000009c0 > [ 376.894631] x9 : ffff800012ceba80 x8 : ffff000875ae8a20 > [ 376.894641] x7 : ffff00087f6b3280 x6 : ffff00087f6b3200 > [ 376.894651] x5 : 0000000000000000 x4 : ffff00087f6a91f8 > [ 376.894660] x3 : ffff00087f6b0120 x2 : 1aa310cec69eb500 > [ 376.894670] x1 : 0000000000000000 x0 : 0000000000000060 > [ 376.894679] Call trace: > [ 376.894683] lkdtm_HARDLOCKUP+0x8/0x18 > [ 376.894686] direct_entry+0x124/0x1c0 > [ 376.894689] full_proxy_write+0x60/0xb0 > [ 376.894693] vfs_write+0xf0/0x230 > [ 376.894696] ksys_write+0x6c/0xf8 > [ 376.894699] __arm64_sys_write+0x1c/0x28 > [ 376.894703] el0_svc_common.constprop.0+0x74/0x1f0 > [ 376.894707] do_el0_svc+0x24/0x90 > [ 376.894710] el0_sync_handler+0x180/0x2f8 > [ 376.894713] el0_sync+0x158/0x180 > > KGDB: > ===== > > # Enter kdb via Magic SysRq > > [6]kdb> btc > btc: cpu status: Currently on cpu 6 > Available cpus: 0-5(I), 6, 7(I), 8, 9-23(I) > <snip> > Stack traceback for pid 555 > 0xffff000875ae8000 555 554 1 8 R 0xffff000875ae89c0 bash > CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > Call trace: > dump_backtrace+0x0/0x1a0 > show_stack+0x18/0x28 > dump_stack+0xc0/0x11c > kgdb_cpu_enter+0x648/0x660 > kgdb_nmicallback+0xa0/0xa8 > ipi_kgdb_nmicallback+0x24/0x30 > ipi_nmi_handler+0x48/0x60 > handle_percpu_devid_fasteoi_ipi+0x74/0x88 > generic_handle_irq+0x30/0x48 > handle_domain_nmi+0x48/0x80 > gic_handle_irq+0x18c/0x34c > el1_irq+0xcc/0x180 > lkdtm_HARDLOCKUP+0x8/0x18 > direct_entry+0x124/0x1c0 > full_proxy_write+0x60/0xb0 > vfs_write+0xf0/0x230 > ksys_write+0x6c/0xf8 > __arm64_sys_write+0x1c/0x28 > el0_svc_common.constprop.0+0x74/0x1f0 > do_el0_svc+0x24/0x90 > el0_sync_handler+0x180/0x2f8 > el0_sync+0x158/0x180 > <snip> > > Changes in v7: > - Add a new library function: kgdb_smp_call_nmi_hook() to expose fallback > mechanism to arch specific code. > - Addressed other misc comments from Daniel. > > Changes in v6: > - Two new patches: #4 and #6 which adds runtime fallback framework for > sysrq backtrace and kgdb roundup features. > - Reversed order of NMI backtrace and kgdb roundup feaure patches. > - Addressed other misc. comments from Marc. > - I haven't picked any tags from v5 since I think there is major rework > involved. Masayoshi, could you please confirm if these features still > work for you? > > Changes in v5: > - Rebased to head of upstream master. > - Remove redundant invocation of ipi_nmi_setup(). > - Addressed misc. comments. > > Changes in v4: > - Move IPI NMI framework to a separate file. > - Get rid of hard-coded IPI_CALL_NMI_FUNC allocation. > - Add NMI backtrace support leveraged via magic SYSRQ. > > Changes in v3: > - Rebased to Marc's latest IPIs patch-set [1]. > > [1] https://lkml.org/lkml/2020/9/1/603 > > Changes since RFC version [1]: > - Switch to use generic interrupt framework to turn an IPI as NMI. > - Dependent on Marc's patch-set [2] which turns IPIs into normal > interrupts. > - Addressed misc. comments from Doug on patch #4. > - Posted kgdb NMI printk() fixup separately which has evolved since > to be solved using different approach via changing kgdb interception > of printk() in common printk() code (see patch [3]). > > [1] https://lkml.org/lkml/2020/4/24/328 > [2] https://lkml.org/lkml/2020/5/19/710 > [3] https://lkml.org/lkml/2020/5/20/418 > > Sumit Garg (7): > arm64: Add framework to turn IPI as NMI > irqchip/gic-v3: Enable support for SGIs to act as NMIs > arm64: smp: Assign and setup an IPI as NMI > nmi: backtrace: Allow runtime arch specific override > arm64: ipi_nmi: Add support for NMI backtrace > kgdb: Expose default CPUs roundup fallback mechanism > arm64: kgdb: Roundup cpus using IPI as NMI Tested-by: Chen-Yu Tsai <wens@csie.org> on an ROC-RK3399-PC.
On Fri, 25 Jun 2021 at 20:26, Chen-Yu Tsai <wens@kernel.org> wrote: > > On Mon, Nov 2, 2020 at 7:46 PM Sumit Garg <sumit.garg@linaro.org> wrote: > > > > With pseudo NMIs support available its possible to configure SGIs to be > > triggered as pseudo NMIs running in NMI context. And kernel features > > such as: > > - NMI backtrace can leverage IPI turned as NMI to get a backtrace of CPU > > stuck in hard lockup using magic SYSRQ. > > - kgdb relies on NMI support to round up CPUs which are stuck in hard > > lockup state with interrupts disabled. > > > > This patch-set adds framework to turn an IPI as NMI which can be triggered > > as a pseudo NMI which in turn invokes registered NMI handlers. > > > > After this patch-set we should be able to get a backtrace for a CPU > > stuck in HARDLOCKUP. Have a look at an examples below from a hard lockup > > testcase run on Developerbox: > > > > $ echo HARDLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT > > > > NMI backtrace: > > ============== > > > > # Issue Magic SysRq to dump backtrace > > > > [ 376.894502] NMI backtrace for cpu 8 > > [ 376.894506] CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > > [ 376.894510] Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > > [ 376.894514] pstate: 40000005 (nZcv daif -PAN -UAO BTYPE=--) > > [ 376.894517] pc : lkdtm_HARDLOCKUP+0x8/0x18 > > [ 376.894520] lr : lkdtm_do_action+0x24/0x30 > > [ 376.894524] sp : ffff800012cebd20 > > [ 376.894527] pmr_save: 00000060 > > [ 376.894530] x29: ffff800012cebd20 x28: ffff000875ae8000 > > [ 376.894540] x27: 0000000000000000 x26: 0000000000000000 > > [ 376.894550] x25: 000000000000001a x24: ffff800012cebe40 > > [ 376.894560] x23: 000000000000000b x22: ffff800010fc5040 > > [ 376.894569] x21: ffff000878b61000 x20: ffff8000113b2870 > > [ 376.894579] x19: 000000000000001b x18: 0000000000000010 > > [ 376.894588] x17: 0000000000000000 x16: 0000000000000000 > > [ 376.894598] x15: ffff000875ae8470 x14: 00000000000002ad > > [ 376.894613] x13: 0000000000000000 x12: 0000000000000000 > > [ 376.894622] x11: 0000000000000007 x10: 00000000000009c0 > > [ 376.894631] x9 : ffff800012ceba80 x8 : ffff000875ae8a20 > > [ 376.894641] x7 : ffff00087f6b3280 x6 : ffff00087f6b3200 > > [ 376.894651] x5 : 0000000000000000 x4 : ffff00087f6a91f8 > > [ 376.894660] x3 : ffff00087f6b0120 x2 : 1aa310cec69eb500 > > [ 376.894670] x1 : 0000000000000000 x0 : 0000000000000060 > > [ 376.894679] Call trace: > > [ 376.894683] lkdtm_HARDLOCKUP+0x8/0x18 > > [ 376.894686] direct_entry+0x124/0x1c0 > > [ 376.894689] full_proxy_write+0x60/0xb0 > > [ 376.894693] vfs_write+0xf0/0x230 > > [ 376.894696] ksys_write+0x6c/0xf8 > > [ 376.894699] __arm64_sys_write+0x1c/0x28 > > [ 376.894703] el0_svc_common.constprop.0+0x74/0x1f0 > > [ 376.894707] do_el0_svc+0x24/0x90 > > [ 376.894710] el0_sync_handler+0x180/0x2f8 > > [ 376.894713] el0_sync+0x158/0x180 > > > > KGDB: > > ===== > > > > # Enter kdb via Magic SysRq > > > > [6]kdb> btc > > btc: cpu status: Currently on cpu 6 > > Available cpus: 0-5(I), 6, 7(I), 8, 9-23(I) > > <snip> > > Stack traceback for pid 555 > > 0xffff000875ae8000 555 554 1 8 R 0xffff000875ae89c0 bash > > CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > > Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > > Call trace: > > dump_backtrace+0x0/0x1a0 > > show_stack+0x18/0x28 > > dump_stack+0xc0/0x11c > > kgdb_cpu_enter+0x648/0x660 > > kgdb_nmicallback+0xa0/0xa8 > > ipi_kgdb_nmicallback+0x24/0x30 > > ipi_nmi_handler+0x48/0x60 > > handle_percpu_devid_fasteoi_ipi+0x74/0x88 > > generic_handle_irq+0x30/0x48 > > handle_domain_nmi+0x48/0x80 > > gic_handle_irq+0x18c/0x34c > > el1_irq+0xcc/0x180 > > lkdtm_HARDLOCKUP+0x8/0x18 > > direct_entry+0x124/0x1c0 > > full_proxy_write+0x60/0xb0 > > vfs_write+0xf0/0x230 > > ksys_write+0x6c/0xf8 > > __arm64_sys_write+0x1c/0x28 > > el0_svc_common.constprop.0+0x74/0x1f0 > > do_el0_svc+0x24/0x90 > > el0_sync_handler+0x180/0x2f8 > > el0_sync+0x158/0x180 > > <snip> > > > > Changes in v7: > > - Add a new library function: kgdb_smp_call_nmi_hook() to expose fallback > > mechanism to arch specific code. > > - Addressed other misc comments from Daniel. > > > > Changes in v6: > > - Two new patches: #4 and #6 which adds runtime fallback framework for > > sysrq backtrace and kgdb roundup features. > > - Reversed order of NMI backtrace and kgdb roundup feaure patches. > > - Addressed other misc. comments from Marc. > > - I haven't picked any tags from v5 since I think there is major rework > > involved. Masayoshi, could you please confirm if these features still > > work for you? > > > > Changes in v5: > > - Rebased to head of upstream master. > > - Remove redundant invocation of ipi_nmi_setup(). > > - Addressed misc. comments. > > > > Changes in v4: > > - Move IPI NMI framework to a separate file. > > - Get rid of hard-coded IPI_CALL_NMI_FUNC allocation. > > - Add NMI backtrace support leveraged via magic SYSRQ. > > > > Changes in v3: > > - Rebased to Marc's latest IPIs patch-set [1]. > > > > [1] https://lkml.org/lkml/2020/9/1/603 > > > > Changes since RFC version [1]: > > - Switch to use generic interrupt framework to turn an IPI as NMI. > > - Dependent on Marc's patch-set [2] which turns IPIs into normal > > interrupts. > > - Addressed misc. comments from Doug on patch #4. > > - Posted kgdb NMI printk() fixup separately which has evolved since > > to be solved using different approach via changing kgdb interception > > of printk() in common printk() code (see patch [3]). > > > > [1] https://lkml.org/lkml/2020/4/24/328 > > [2] https://lkml.org/lkml/2020/5/19/710 > > [3] https://lkml.org/lkml/2020/5/20/418 > > > > Sumit Garg (7): > > arm64: Add framework to turn IPI as NMI > > irqchip/gic-v3: Enable support for SGIs to act as NMIs > > arm64: smp: Assign and setup an IPI as NMI > > nmi: backtrace: Allow runtime arch specific override > > arm64: ipi_nmi: Add support for NMI backtrace > > kgdb: Expose default CPUs roundup fallback mechanism > > arm64: kgdb: Roundup cpus using IPI as NMI > > Tested-by: Chen-Yu Tsai <wens@csie.org> > > on an ROC-RK3399-PC. Thanks for testing this feature. -Sumit
On Tue, 5 Jan 2021 at 16:13, Marc Zyngier <maz@kernel.org> wrote: > > On 2021-01-05 10:34, Sumit Garg wrote: > > > Do you have any further feedback on this patch-set? > > None at the moment. We have tons of issues to solve with the arm64 > interrupt entry code vs instrumentation at the moment, so it is > pretty much at the bottom of the priority list for now. > I hope you were able to resolve most of the issues. Is it an appropriate time to have a relook at this pseudo NMI feature? -Sumit > Thanks, > > M. > -- > Jazz is not dead. It just smells funny...
On Wed, Jul 07, 2021 at 11:29:29AM +0530, Sumit Garg wrote: > On Fri, 25 Jun 2021 at 20:26, Chen-Yu Tsai <wens@kernel.org> wrote: > > > > On Mon, Nov 2, 2020 at 7:46 PM Sumit Garg <sumit.garg@linaro.org> wrote: > > > > > > With pseudo NMIs support available its possible to configure SGIs to be > > > triggered as pseudo NMIs running in NMI context. And kernel features > > > such as: > > > - NMI backtrace can leverage IPI turned as NMI to get a backtrace of CPU > > > stuck in hard lockup using magic SYSRQ. > > > - kgdb relies on NMI support to round up CPUs which are stuck in hard > > > lockup state with interrupts disabled. > > > > > > This patch-set adds framework to turn an IPI as NMI which can be triggered > > > as a pseudo NMI which in turn invokes registered NMI handlers. > > > > > > After this patch-set we should be able to get a backtrace for a CPU > > > stuck in HARDLOCKUP. Have a look at an examples below from a hard lockup > > > testcase run on Developerbox: > > > > > > $ echo HARDLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT > > > > > > NMI backtrace: > > > ============== > > > > > > # Issue Magic SysRq to dump backtrace > > > > > > [ 376.894502] NMI backtrace for cpu 8 > > > [ 376.894506] CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > > > [ 376.894510] Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > > > [ 376.894514] pstate: 40000005 (nZcv daif -PAN -UAO BTYPE=--) > > > [ 376.894517] pc : lkdtm_HARDLOCKUP+0x8/0x18 > > > [ 376.894520] lr : lkdtm_do_action+0x24/0x30 > > > [ 376.894524] sp : ffff800012cebd20 > > > [ 376.894527] pmr_save: 00000060 > > > [ 376.894530] x29: ffff800012cebd20 x28: ffff000875ae8000 > > > [ 376.894540] x27: 0000000000000000 x26: 0000000000000000 > > > [ 376.894550] x25: 000000000000001a x24: ffff800012cebe40 > > > [ 376.894560] x23: 000000000000000b x22: ffff800010fc5040 > > > [ 376.894569] x21: ffff000878b61000 x20: ffff8000113b2870 > > > [ 376.894579] x19: 000000000000001b x18: 0000000000000010 > > > [ 376.894588] x17: 0000000000000000 x16: 0000000000000000 > > > [ 376.894598] x15: ffff000875ae8470 x14: 00000000000002ad > > > [ 376.894613] x13: 0000000000000000 x12: 0000000000000000 > > > [ 376.894622] x11: 0000000000000007 x10: 00000000000009c0 > > > [ 376.894631] x9 : ffff800012ceba80 x8 : ffff000875ae8a20 > > > [ 376.894641] x7 : ffff00087f6b3280 x6 : ffff00087f6b3200 > > > [ 376.894651] x5 : 0000000000000000 x4 : ffff00087f6a91f8 > > > [ 376.894660] x3 : ffff00087f6b0120 x2 : 1aa310cec69eb500 > > > [ 376.894670] x1 : 0000000000000000 x0 : 0000000000000060 > > > [ 376.894679] Call trace: > > > [ 376.894683] lkdtm_HARDLOCKUP+0x8/0x18 > > > [ 376.894686] direct_entry+0x124/0x1c0 > > > [ 376.894689] full_proxy_write+0x60/0xb0 > > > [ 376.894693] vfs_write+0xf0/0x230 > > > [ 376.894696] ksys_write+0x6c/0xf8 > > > [ 376.894699] __arm64_sys_write+0x1c/0x28 > > > [ 376.894703] el0_svc_common.constprop.0+0x74/0x1f0 > > > [ 376.894707] do_el0_svc+0x24/0x90 > > > [ 376.894710] el0_sync_handler+0x180/0x2f8 > > > [ 376.894713] el0_sync+0x158/0x180 > > > > > > KGDB: > > > ===== > > > > > > # Enter kdb via Magic SysRq > > > > > > [6]kdb> btc > > > btc: cpu status: Currently on cpu 6 > > > Available cpus: 0-5(I), 6, 7(I), 8, 9-23(I) > > > <snip> > > > Stack traceback for pid 555 > > > 0xffff000875ae8000 555 554 1 8 R 0xffff000875ae89c0 bash > > > CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > > > Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > > > Call trace: > > > dump_backtrace+0x0/0x1a0 > > > show_stack+0x18/0x28 > > > dump_stack+0xc0/0x11c > > > kgdb_cpu_enter+0x648/0x660 > > > kgdb_nmicallback+0xa0/0xa8 > > > ipi_kgdb_nmicallback+0x24/0x30 > > > ipi_nmi_handler+0x48/0x60 > > > handle_percpu_devid_fasteoi_ipi+0x74/0x88 > > > generic_handle_irq+0x30/0x48 > > > handle_domain_nmi+0x48/0x80 > > > gic_handle_irq+0x18c/0x34c > > > el1_irq+0xcc/0x180 > > > lkdtm_HARDLOCKUP+0x8/0x18 > > > direct_entry+0x124/0x1c0 > > > full_proxy_write+0x60/0xb0 > > > vfs_write+0xf0/0x230 > > > ksys_write+0x6c/0xf8 > > > __arm64_sys_write+0x1c/0x28 > > > el0_svc_common.constprop.0+0x74/0x1f0 > > > do_el0_svc+0x24/0x90 > > > el0_sync_handler+0x180/0x2f8 > > > el0_sync+0x158/0x180 > > > <snip> > > > > > > Changes in v7: > > > - Add a new library function: kgdb_smp_call_nmi_hook() to expose fallback > > > mechanism to arch specific code. > > > - Addressed other misc comments from Daniel. > > > > > > Changes in v6: > > > - Two new patches: #4 and #6 which adds runtime fallback framework for > > > sysrq backtrace and kgdb roundup features. > > > - Reversed order of NMI backtrace and kgdb roundup feaure patches. > > > - Addressed other misc. comments from Marc. > > > - I haven't picked any tags from v5 since I think there is major rework > > > involved. Masayoshi, could you please confirm if these features still > > > work for you? > > > > > > Changes in v5: > > > - Rebased to head of upstream master. > > > - Remove redundant invocation of ipi_nmi_setup(). > > > - Addressed misc. comments. > > > > > > Changes in v4: > > > - Move IPI NMI framework to a separate file. > > > - Get rid of hard-coded IPI_CALL_NMI_FUNC allocation. > > > - Add NMI backtrace support leveraged via magic SYSRQ. > > > > > > Changes in v3: > > > - Rebased to Marc's latest IPIs patch-set [1]. > > > > > > [1] https://lkml.org/lkml/2020/9/1/603 > > > > > > Changes since RFC version [1]: > > > - Switch to use generic interrupt framework to turn an IPI as NMI. > > > - Dependent on Marc's patch-set [2] which turns IPIs into normal > > > interrupts. > > > - Addressed misc. comments from Doug on patch #4. > > > - Posted kgdb NMI printk() fixup separately which has evolved since > > > to be solved using different approach via changing kgdb interception > > > of printk() in common printk() code (see patch [3]). > > > > > > [1] https://lkml.org/lkml/2020/4/24/328 > > > [2] https://lkml.org/lkml/2020/5/19/710 > > > [3] https://lkml.org/lkml/2020/5/20/418 > > > > > > Sumit Garg (7): > > > arm64: Add framework to turn IPI as NMI > > > irqchip/gic-v3: Enable support for SGIs to act as NMIs > > > arm64: smp: Assign and setup an IPI as NMI > > > nmi: backtrace: Allow runtime arch specific override > > > arm64: ipi_nmi: Add support for NMI backtrace > > > kgdb: Expose default CPUs roundup fallback mechanism > > > arm64: kgdb: Roundup cpus using IPI as NMI > > > > Tested-by: Chen-Yu Tsai <wens@csie.org> > > > > on an ROC-RK3399-PC. > > Thanks for testing this feature. > > -Sumit Hello Sumit, How is this patch series going? - Masa
Hi Masa, On Thu, 13 Jan 2022 at 01:01, Masayoshi Mizuma <msys.mizuma@gmail.com> wrote: > > On Wed, Jul 07, 2021 at 11:29:29AM +0530, Sumit Garg wrote: > > On Fri, 25 Jun 2021 at 20:26, Chen-Yu Tsai <wens@kernel.org> wrote: > > > > > > On Mon, Nov 2, 2020 at 7:46 PM Sumit Garg <sumit.garg@linaro.org> wrote: > > > > > > > > With pseudo NMIs support available its possible to configure SGIs to be > > > > triggered as pseudo NMIs running in NMI context. And kernel features > > > > such as: > > > > - NMI backtrace can leverage IPI turned as NMI to get a backtrace of CPU > > > > stuck in hard lockup using magic SYSRQ. > > > > - kgdb relies on NMI support to round up CPUs which are stuck in hard > > > > lockup state with interrupts disabled. > > > > > > > > This patch-set adds framework to turn an IPI as NMI which can be triggered > > > > as a pseudo NMI which in turn invokes registered NMI handlers. > > > > > > > > After this patch-set we should be able to get a backtrace for a CPU > > > > stuck in HARDLOCKUP. Have a look at an examples below from a hard lockup > > > > testcase run on Developerbox: > > > > > > > > $ echo HARDLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT > > > > > > > > NMI backtrace: > > > > ============== > > > > > > > > # Issue Magic SysRq to dump backtrace > > > > > > > > [ 376.894502] NMI backtrace for cpu 8 > > > > [ 376.894506] CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > > > > [ 376.894510] Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > > > > [ 376.894514] pstate: 40000005 (nZcv daif -PAN -UAO BTYPE=--) > > > > [ 376.894517] pc : lkdtm_HARDLOCKUP+0x8/0x18 > > > > [ 376.894520] lr : lkdtm_do_action+0x24/0x30 > > > > [ 376.894524] sp : ffff800012cebd20 > > > > [ 376.894527] pmr_save: 00000060 > > > > [ 376.894530] x29: ffff800012cebd20 x28: ffff000875ae8000 > > > > [ 376.894540] x27: 0000000000000000 x26: 0000000000000000 > > > > [ 376.894550] x25: 000000000000001a x24: ffff800012cebe40 > > > > [ 376.894560] x23: 000000000000000b x22: ffff800010fc5040 > > > > [ 376.894569] x21: ffff000878b61000 x20: ffff8000113b2870 > > > > [ 376.894579] x19: 000000000000001b x18: 0000000000000010 > > > > [ 376.894588] x17: 0000000000000000 x16: 0000000000000000 > > > > [ 376.894598] x15: ffff000875ae8470 x14: 00000000000002ad > > > > [ 376.894613] x13: 0000000000000000 x12: 0000000000000000 > > > > [ 376.894622] x11: 0000000000000007 x10: 00000000000009c0 > > > > [ 376.894631] x9 : ffff800012ceba80 x8 : ffff000875ae8a20 > > > > [ 376.894641] x7 : ffff00087f6b3280 x6 : ffff00087f6b3200 > > > > [ 376.894651] x5 : 0000000000000000 x4 : ffff00087f6a91f8 > > > > [ 376.894660] x3 : ffff00087f6b0120 x2 : 1aa310cec69eb500 > > > > [ 376.894670] x1 : 0000000000000000 x0 : 0000000000000060 > > > > [ 376.894679] Call trace: > > > > [ 376.894683] lkdtm_HARDLOCKUP+0x8/0x18 > > > > [ 376.894686] direct_entry+0x124/0x1c0 > > > > [ 376.894689] full_proxy_write+0x60/0xb0 > > > > [ 376.894693] vfs_write+0xf0/0x230 > > > > [ 376.894696] ksys_write+0x6c/0xf8 > > > > [ 376.894699] __arm64_sys_write+0x1c/0x28 > > > > [ 376.894703] el0_svc_common.constprop.0+0x74/0x1f0 > > > > [ 376.894707] do_el0_svc+0x24/0x90 > > > > [ 376.894710] el0_sync_handler+0x180/0x2f8 > > > > [ 376.894713] el0_sync+0x158/0x180 > > > > > > > > KGDB: > > > > ===== > > > > > > > > # Enter kdb via Magic SysRq > > > > > > > > [6]kdb> btc > > > > btc: cpu status: Currently on cpu 6 > > > > Available cpus: 0-5(I), 6, 7(I), 8, 9-23(I) > > > > <snip> > > > > Stack traceback for pid 555 > > > > 0xffff000875ae8000 555 554 1 8 R 0xffff000875ae89c0 bash > > > > CPU: 8 PID: 555 Comm: bash Not tainted 5.9.0-rc3-00740-g06ff047-dirty #242 > > > > Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build #73 Apr 6 2020 > > > > Call trace: > > > > dump_backtrace+0x0/0x1a0 > > > > show_stack+0x18/0x28 > > > > dump_stack+0xc0/0x11c > > > > kgdb_cpu_enter+0x648/0x660 > > > > kgdb_nmicallback+0xa0/0xa8 > > > > ipi_kgdb_nmicallback+0x24/0x30 > > > > ipi_nmi_handler+0x48/0x60 > > > > handle_percpu_devid_fasteoi_ipi+0x74/0x88 > > > > generic_handle_irq+0x30/0x48 > > > > handle_domain_nmi+0x48/0x80 > > > > gic_handle_irq+0x18c/0x34c > > > > el1_irq+0xcc/0x180 > > > > lkdtm_HARDLOCKUP+0x8/0x18 > > > > direct_entry+0x124/0x1c0 > > > > full_proxy_write+0x60/0xb0 > > > > vfs_write+0xf0/0x230 > > > > ksys_write+0x6c/0xf8 > > > > __arm64_sys_write+0x1c/0x28 > > > > el0_svc_common.constprop.0+0x74/0x1f0 > > > > do_el0_svc+0x24/0x90 > > > > el0_sync_handler+0x180/0x2f8 > > > > el0_sync+0x158/0x180 > > > > <snip> > > > > > > > > Changes in v7: > > > > - Add a new library function: kgdb_smp_call_nmi_hook() to expose fallback > > > > mechanism to arch specific code. > > > > - Addressed other misc comments from Daniel. > > > > > > > > Changes in v6: > > > > - Two new patches: #4 and #6 which adds runtime fallback framework for > > > > sysrq backtrace and kgdb roundup features. > > > > - Reversed order of NMI backtrace and kgdb roundup feaure patches. > > > > - Addressed other misc. comments from Marc. > > > > - I haven't picked any tags from v5 since I think there is major rework > > > > involved. Masayoshi, could you please confirm if these features still > > > > work for you? > > > > > > > > Changes in v5: > > > > - Rebased to head of upstream master. > > > > - Remove redundant invocation of ipi_nmi_setup(). > > > > - Addressed misc. comments. > > > > > > > > Changes in v4: > > > > - Move IPI NMI framework to a separate file. > > > > - Get rid of hard-coded IPI_CALL_NMI_FUNC allocation. > > > > - Add NMI backtrace support leveraged via magic SYSRQ. > > > > > > > > Changes in v3: > > > > - Rebased to Marc's latest IPIs patch-set [1]. > > > > > > > > [1] https://lkml.org/lkml/2020/9/1/603 > > > > > > > > Changes since RFC version [1]: > > > > - Switch to use generic interrupt framework to turn an IPI as NMI. > > > > - Dependent on Marc's patch-set [2] which turns IPIs into normal > > > > interrupts. > > > > - Addressed misc. comments from Doug on patch #4. > > > > - Posted kgdb NMI printk() fixup separately which has evolved since > > > > to be solved using different approach via changing kgdb interception > > > > of printk() in common printk() code (see patch [3]). > > > > > > > > [1] https://lkml.org/lkml/2020/4/24/328 > > > > [2] https://lkml.org/lkml/2020/5/19/710 > > > > [3] https://lkml.org/lkml/2020/5/20/418 > > > > > > > > Sumit Garg (7): > > > > arm64: Add framework to turn IPI as NMI > > > > irqchip/gic-v3: Enable support for SGIs to act as NMIs > > > > arm64: smp: Assign and setup an IPI as NMI > > > > nmi: backtrace: Allow runtime arch specific override > > > > arm64: ipi_nmi: Add support for NMI backtrace > > > > kgdb: Expose default CPUs roundup fallback mechanism > > > > arm64: kgdb: Roundup cpus using IPI as NMI > > > > > > Tested-by: Chen-Yu Tsai <wens@csie.org> > > > > > > on an ROC-RK3399-PC. > > > > Thanks for testing this feature. > > > > -Sumit > > Hello Sumit, > > How is this patch series going? I am still awaiting feedback from arm64 maintainers. arm64 maintainers, Do you believe that features leveraging pseudo NMIs on arm64 can make it to the mainline? Or the pseudo NMIs feature itself isn't mature enough that we don't want to have any further users at this point? -Sumit > > - Masa