| Message ID | 20231026143122.279437-1-cleger@rivosinc.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | riscv: add support for SBI Supervisor Software Events | expand |
Pardon. It seems that the code in opensbi[1] is not complete for PMU NMI now. For example, the pmu ovf irq is still delegated to supervisor mode and thus can not really play a role as NMI. And neither the kernel nor opensbi will inject a pmu event. To complete the work, we think maybe 'enable_cb' and 'disable_cb' functions can be supplied for sbi_sse_cb_ops. When sbi_sse_enable is called to enable pmu event, the enable_cb will be called to revoke the delegation of pmu ovf irq and enable this irq in CSR_MIE. When pmu ovf irq occurs, kernel traps into opensbi and opensbi injects the event via sbi_sse_inject_event and eret back to kernel. Please point it out if we have any misunderstanding. By the way, how is SSE going now? We will appreciate it very much if we can participate in some development work in kernel or opensbi and be of any help. Regards! Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [1] On Thu, Oct 26, 2023 at 10:31 PM Clément Léger <cleger@rivosinc.com> wrote: > > The SBI Supervisor Software Events (SSE) extensions provides a mechanism > to inject software events from an SBI implementation to supervisor > software such that it preempts all other supervisor level traps and > interrupts [1]. > > Various events are defined and can be send asynchronously to supervisor > software (RAS, PMU, DEBUG, Asynchronous page fault) from SBI as well > as platform specific events. Events can be either local (per-hart) or > global. Events can be nested on top of each other based on priority and > can interrupt the kernel at any time. > > First patch adds the SSE definitions. Second one adds support for SSE > itself. Implementation is split between arch specific code and generic > part (similarly to what is done for ARM SDEI). Finally, the last patch > add support fro SSE event in the SBI PMU driver. If the SSE event is > available from the SBI then, it will be used instead of the normal > interrupt. > > Amongst the specific points that needs to be handle is the interruption > at any point of the kernel execution and more specifically during > exception handling. Due to the fact that the exception entry > implementation uses the SCRATCH CSR as both the current task struct and > as the temporary register to switch the stack and save register, it is > difficult to reliably get the current task struct if we get interrupted > at this specific moment. A fixup-like mechanism allows to mark the > location of the current task struct depending on the entry level > (user/kernel) and the location. This is then used in the SSE assembly to > determine where is located the current task_struct. > > Contrary to pseudo NMI [2], SSE does not modifies the way interrupts are > handled and does not adds any overhead to existing code. Moreover, it > provides "true" NMI-like interrupts which can interrupt the kernel at > any time (even in exception handling). This is particularly crucial for > RAS errors which needs to be handled as fast as possible to avoid any > fault propagation. Additionally, SSE event handling is faster that the > standard IRQ handling path with almost half executed instruction (700 vs > 1590). Some complementary tests/perf measurements will be done. > > For testing purpose, one can use the provided SBI implementation at [3]. > This series also needs patch [4] to fix a bug in the PMU driver. > > Link: https://lists.riscv.org/g/tech-prs/message/515 [1] > Link: https://lore.kernel.org/lkml/20231023082911.23242-10-luxu.kernel@bytedance.com/T/ [2] > Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [3] > Link: https://lore.kernel.org/linux-arm-kernel/20231026084010.11888-1-alexghiti@rivosinc.com/ [4] > > --- > > Clément Léger (3): > riscv: add SBI SSE extension definitions > riscv: add support for SBI Supervisor Software Events extension > perf: RISC-V: add support for SSE event > > arch/riscv/include/asm/asm-prototypes.h | 5 + > arch/riscv/include/asm/sbi.h | 40 ++ > arch/riscv/include/asm/sse.h | 94 +++++ > arch/riscv/kernel/Makefile | 1 + > arch/riscv/kernel/asm-offsets.c | 17 + > arch/riscv/kernel/entry.S | 156 ++++++++ > arch/riscv/kernel/sbi.c | 4 + > arch/riscv/kernel/sse.c | 97 +++++ > arch/riscv/kernel/stacktrace.c | 13 + > arch/riscv/kernel/vmlinux.lds.S | 6 + > drivers/firmware/Kconfig | 10 + > drivers/firmware/Makefile | 1 + > drivers/firmware/riscv_sse.c | 496 ++++++++++++++++++++++++ > drivers/perf/riscv_pmu_sbi.c | 51 ++- > include/linux/riscv_sse.h | 56 +++ > 15 files changed, 1038 insertions(+), 9 deletions(-) > create mode 100644 arch/riscv/include/asm/sse.h > create mode 100644 arch/riscv/kernel/sse.c > create mode 100644 drivers/firmware/riscv_sse.c > create mode 100644 include/linux/riscv_sse.h > > -- > 2.42.0 >
On 07/12/2023 10:09, Xu Lu wrote: > Pardon. It seems that the code in opensbi[1] is not complete for PMU NMI now. > For example, the pmu ovf irq is still delegated to supervisor mode and > thus can not really play a role as NMI. And neither the kernel nor > opensbi will inject a pmu event. > > To complete the work, we think maybe 'enable_cb' and 'disable_cb' > functions can be supplied for sbi_sse_cb_ops. > When sbi_sse_enable is called to enable pmu event, the enable_cb will > be called to revoke the delegation of pmu ovf irq and enable this irq > in CSR_MIE. Hi Xu, Indeed, this part has been developed but was left out for the RFC. But your understanding is correct. > When pmu ovf irq occurs, kernel traps into opensbi and opensbi injects > the event via sbi_sse_inject_event and eret back to kernel. > > Please point it out if we have any misunderstanding. > > By the way, how is SSE going now? We will appreciate it very much if > we can participate in some development work in kernel or opensbi and > be of any help. The development is almost complete, Anup/Himanchu will send a new revision of the spec addressing various comments and I'll resend the RFC following that spec update. Regards, Clément > > Regards! > > Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [1] > > On Thu, Oct 26, 2023 at 10:31 PM Clément Léger <cleger@rivosinc.com> wrote: >> >> The SBI Supervisor Software Events (SSE) extensions provides a mechanism >> to inject software events from an SBI implementation to supervisor >> software such that it preempts all other supervisor level traps and >> interrupts [1]. >> >> Various events are defined and can be send asynchronously to supervisor >> software (RAS, PMU, DEBUG, Asynchronous page fault) from SBI as well >> as platform specific events. Events can be either local (per-hart) or >> global. Events can be nested on top of each other based on priority and >> can interrupt the kernel at any time. >> >> First patch adds the SSE definitions. Second one adds support for SSE >> itself. Implementation is split between arch specific code and generic >> part (similarly to what is done for ARM SDEI). Finally, the last patch >> add support fro SSE event in the SBI PMU driver. If the SSE event is >> available from the SBI then, it will be used instead of the normal >> interrupt. >> >> Amongst the specific points that needs to be handle is the interruption >> at any point of the kernel execution and more specifically during >> exception handling. Due to the fact that the exception entry >> implementation uses the SCRATCH CSR as both the current task struct and >> as the temporary register to switch the stack and save register, it is >> difficult to reliably get the current task struct if we get interrupted >> at this specific moment. A fixup-like mechanism allows to mark the >> location of the current task struct depending on the entry level >> (user/kernel) and the location. This is then used in the SSE assembly to >> determine where is located the current task_struct. >> >> Contrary to pseudo NMI [2], SSE does not modifies the way interrupts are >> handled and does not adds any overhead to existing code. Moreover, it >> provides "true" NMI-like interrupts which can interrupt the kernel at >> any time (even in exception handling). This is particularly crucial for >> RAS errors which needs to be handled as fast as possible to avoid any >> fault propagation. Additionally, SSE event handling is faster that the >> standard IRQ handling path with almost half executed instruction (700 vs >> 1590). Some complementary tests/perf measurements will be done. >> >> For testing purpose, one can use the provided SBI implementation at [3]. >> This series also needs patch [4] to fix a bug in the PMU driver. >> >> Link: https://lists.riscv.org/g/tech-prs/message/515 [1] >> Link: https://lore.kernel.org/lkml/20231023082911.23242-10-luxu.kernel@bytedance.com/T/ [2] >> Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [3] >> Link: https://lore.kernel.org/linux-arm-kernel/20231026084010.11888-1-alexghiti@rivosinc.com/ [4] >> >> --- >> >> Clément Léger (3): >> riscv: add SBI SSE extension definitions >> riscv: add support for SBI Supervisor Software Events extension >> perf: RISC-V: add support for SSE event >> >> arch/riscv/include/asm/asm-prototypes.h | 5 + >> arch/riscv/include/asm/sbi.h | 40 ++ >> arch/riscv/include/asm/sse.h | 94 +++++ >> arch/riscv/kernel/Makefile | 1 + >> arch/riscv/kernel/asm-offsets.c | 17 + >> arch/riscv/kernel/entry.S | 156 ++++++++ >> arch/riscv/kernel/sbi.c | 4 + >> arch/riscv/kernel/sse.c | 97 +++++ >> arch/riscv/kernel/stacktrace.c | 13 + >> arch/riscv/kernel/vmlinux.lds.S | 6 + >> drivers/firmware/Kconfig | 10 + >> drivers/firmware/Makefile | 1 + >> drivers/firmware/riscv_sse.c | 496 ++++++++++++++++++++++++ >> drivers/perf/riscv_pmu_sbi.c | 51 ++- >> include/linux/riscv_sse.h | 56 +++ >> 15 files changed, 1038 insertions(+), 9 deletions(-) >> create mode 100644 arch/riscv/include/asm/sse.h >> create mode 100644 arch/riscv/kernel/sse.c >> create mode 100644 drivers/firmware/riscv_sse.c >> create mode 100644 include/linux/riscv_sse.h >> >> -- >> 2.42.0 >>