| Message ID | 20230410081438.1750-1-xin3.li@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | x86: enable FRED for x86-64 | expand |
On 4/10/23 01:14, Xin Li wrote:
> This patch set enables FRED for x86-64.
I'm worried we're just in a patchbomb-once-a-week mode with FRED at this
point. There wasn't a single comment on v7 so, of course, here's v8 a
week later.
FRED is a rare CPU feature because it's universally wanted. Us software
folks have been begging for it for a looooooooong time. Is there anyone
out there that has any doubts that the kernel will support (this) FRED
eventually?
The code also looks pretty reasonable.
I do think it's missing some Documentation, and the cover letter is bit
sparse. It would be nice to see some high-level information about
things like, for instance, why there needs to be FRED refactoring for
NMI/#PF/#DB/#MC specifically, but not other exceptions.
There also aren't any new selftests. I faintly recall some tweak to the
selftests recently that was FRED-oriented, but I'd still expect all the
selftests that poke at the entry code to be perturbed by this a bit.
Basically, if anyone else has been procrastinating on reviewing this
set, now is probably the time to dig in. (I'll include myself in that
category, btw)
Oh, and one other nit. We do have a specific maintainer for the entry code: > X86 ENTRY CODE > M: Andy Lutomirski <luto@kernel.org> > L: linux-kernel@vger.kernel.org > S: Maintained > T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/asm > F: arch/x86/entry/ Please make sure to cc Andy on FRED going forward. This is probably also a good cue to go and make sure you didn't miss any other folks that need to see this series.
> I do think it's missing some Documentation, and the cover letter is bit sparse. It > would be nice to see some high-level information about things like, for instance, > why there needs to be FRED refactoring for NMI/#PF/#DB/#MC specifically, but > not other exceptions. We do have some comments in the commit messages or around the code changes. However a high level document in the Documentation/x86/ directory probably works better, I can do that. > > There also aren't any new selftests. I faintly recall some tweak to the selftests > recently that was FRED-oriented, but I'd still expect all the selftests that poke at > the entry code to be perturbed by this a bit. Because FRED code majorly replaces the IDT entry/dispatch/return code, and makes few changes to the event handlers, our focus was more of to check if all the event handlers are properly called and returned, which is very well covered by the existing IDT selftests. One area we need to add selftests to is the FRED event dispatch framework, to make sure we cover all the possible dispatch paths. > Basically, if anyone else has been procrastinating on reviewing this set, now is > probably the time to dig in. (I'll include myself in that category, btw) I really appreciate it! Thanks! Xin
> Oh, and one other nit. We do have a specific maintainer for the entry code: > > > X86 ENTRY CODE > > M: Andy Lutomirski <luto@kernel.org> > > L: linux-kernel@vger.kernel.org > > S: Maintained > > T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/asm > > F: arch/x86/entry/ > > Please make sure to cc Andy on FRED going forward. > > This is probably also a good cue to go and make sure you didn't miss any other > folks that need to see this series. My bad, and surely will add Andy.
On Mon, Apr 10, 2023 at 07:14:21PM +0000, Li, Xin3 wrote: > > Basically, if anyone else has been procrastinating on reviewing this set, now is > > probably the time to dig in. (I'll include myself in that category, btw) > > I really appreciate it! That doesn't mean that you should patch-bomb people once a week even without any review happening. Is FRED in any hardware incarnation to rush it? If no, be patient, address the review comments once you have them and do not spam once a week just because. As Dave said, this is wanted by all and it will get reviewed eventually. But it is not something that needs to go in now so you don't have to create unnecessary pressure. Thx.
On 4/10/23 12:32, Borislav Petkov wrote: > On Mon, Apr 10, 2023 at 07:14:21PM +0000, Li, Xin3 wrote: >>> Basically, if anyone else has been procrastinating on reviewing this set, now is >>> probably the time to dig in. (I'll include myself in that category, btw) >> I really appreciate it! > That doesn't mean that you should patch-bomb people once a week even > without any review happening. > > Is FRED in any hardware incarnation to rush it? Which reminds me... It is always appreciated when hardware vendors can put a stake in the ground about where and how a feature will show up. It's great that you said that it is SIMICS-only for now. That's a start. But, if you could, for instance say things like (and I'm totally pulling these out of my rear end): FRED will be available only on server CPUs or FRED will be available across all Intel CPUs or FRED will start showing up in CPUs that are released in roughly 2033 it would be nice to know that. Here's what I said for protection keys, for example: > https://lore.kernel.org/linux-mm/20160212210152.9CAD15B0@viggo.jf.intel.com/
> > Is FRED in any hardware incarnation to rush it? > > Which reminds me... It is always appreciated when hardware vendors can put a > stake in the ground about where and how a feature will show up. > It's great that you said that it is SIMICS-only for now. That's a start. > > But, if you could, for instance say things like (and I'm totally pulling these out of > my rear end): > > FRED will be available only on server CPUs or > FRED will be available across all Intel CPUs or > FRED will start showing up in CPUs that are released > in roughly 2033 > > it would be nice to know that. > > Here's what I said for protection keys, for example: > > > https://lore.kernel.org/linux-mm/20160212210152.9CAD15B0@viggo.jf.intel.com/ Good point, I will find out what I could provide in LKML and update later. Thanks! Xin
> Is FRED in any hardware incarnation to rush it? Current plan is to have FRED HW available in Intel developer Cloud in 1H'2024. As the cover letter mentioned, for now, it's available through Intel Simics https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html , and a developer has to install it on a dev machine. FRED will also be available publicly through a Simics cloud service in Q2'23, in which a Simics instance is created for development/QA use. FRED is a baseline feature, and there are some HW features on top of it. Some of the new features might be launched with the first FRED HW. In addition, the KVM FRED patch set is on top of this patch set. > If no, be patient, address the review comments once you have them and do not > spam once a week just because. As Dave said, this is wanted by all and it will get > reviewed eventually. But it is not something that needs to go in now so you don't > have to create unnecessary pressure. It wasn't my intension to patch-bomb the list. But I will pay attention regarding this concern. Thanks! Xin
On Mon, Apr 10 2023 at 19:16, Li, Xin3 wrote: >> Oh, and one other nit. We do have a specific maintainer for the entry code: >> >> > X86 ENTRY CODE >> > M: Andy Lutomirski <luto@kernel.org> >> > L: linux-kernel@vger.kernel.org >> > S: Maintained >> > T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/asm >> > F: arch/x86/entry/ >> >> Please make sure to cc Andy on FRED going forward. >> >> This is probably also a good cue to go and make sure you didn't miss any other >> folks that need to see this series. > > My bad, and surely will add Andy. He's on CC indirectly via the x86@kernel.org mail exploder.
So I feel obliged to throw in some defending of Xin here, mostly because in some of these cases I'm the perkeleen vittupää[1] and not Xin. The other thing is that this patchset is from April and predates the recent FRED changes (which were unanticipated at the time); I explicitly asked Xin to hold off making updates until the spec had restabilized. -hpa [1] https://lkml.org/lkml/2013/7/13/132
On Mon, Jun 05 2023 at 10:22, H. Peter Anvin wrote: > So I feel obliged to throw in some defending of Xin here, mostly because > in some of these cases I'm the perkeleen vittupää[1] and not Xin. That's clear from the authorship and the style of the patches and not all was criticism was directed at Xin obviously. He's just the messenger in that case. Thanks, tglx
This patch set enables FRED for x86-64. The Intel flexible return and event delivery (FRED) architecture defines simple new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals: 1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by the IRET instruction with lower latency transitions. 2) Improve software robustness by ensuring that event delivery establishes the full supervisor context and that event return establishes the full user context. The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0. Search for the latest FRED spec in most search engines with this search pattern: site:intel.com FRED (flexible return and event delivery) specification As of now there is no publicly avaiable CPU supporting FRED, thus the Intel Simics® Simulator is used as software development and testing vehicles. And it can be downloaded from: https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html To enable FRED, the Simics package 8112 QSP-CPU needs to be installed with CPU model configured as: $cpu_comp_class = "x86-experimental-fred" Longer term, we should refactor common code shared by FRED and IDT into common shared files, and contain IDT code using a new config CONFIG_X86_IDT. Changes since v7: * Always call external_interrupt() for VMX IRQ handling on x86_64, thus avoid re-entering the noinstr code. * Create a FRED stack frame when FRED is compiled-in but not enabled, which uses some extra stack space but simplifies the code. * Add a log message when FRED is enabled. Changes since v6: * Add a comment to explain why it is safe to write to a previous FRED stack frame. (Lai Jiangshan). * Export fred_entrypoint_kernel(), required when kvm-intel built as a module. * Reserve a REDZONE for CALL emulation and Align RSP to a 64-byte boundary before pushing a new FRED stack frame. * Replace pt_regs csx flags prefix FRED_CSL_ with FRED_CSX_. Changes since v5: * Initialize system_interrupt_handlers with dispatch_table_spurious_interrupt() instead of NULL to get rid of a branch (Peter Zijlstra). * Disallow #DB inside #MCE for robustness sake (Peter Zijlstra). * Add a comment for FRED stack level settings (Lai Jiangshan). * Move the NMI bit from an invalid stack frame, which caused ERETU to fault, to the fault handler's stack frame, thus to unblock NMI ASAP if NMI is blocked (Lai Jiangshan). * Refactor VMX_DO_EVENT_IRQOFF to handle IRQ/NMI in IRQ/NMI induced VM exits when FRED is enabled (Sean Christopherson). Changes since v4: * Do NOT use the term "injection", which in the KVM context means to reinject an event into the guest (Sean Christopherson). * Add the explanation of why to execute "int $2" to invoke the NMI handler in NMI caused VM exits (Sean Christopherson). * Use cs/ss instead of csx/ssx when initializing the pt_regs structure for calling external_interrupt(), otherwise it breaks i386 build. Changes since v3: * Call external_interrupt() to handle IRQ in IRQ caused VM exits. * Execute "int $2" to handle NMI in NMI caused VM exits. * Rename csl/ssl of the pt_regs structure to csx/ssx (x for extended) (Andrew Cooper). Changes since v2: * Improve comments for changes in arch/x86/include/asm/idtentry.h. Changes since v1: * call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel handler (Peter Zijlstra). * Initialize a FRED exception handler to fred_bad_event() instead of NULL if no FRED handler defined for an exception vector (Peter Zijlstra). * Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}() down into individual FRED exception handlers, instead of in the dispatch framework (Peter Zijlstra). H. Peter Anvin (Intel) (24): x86/traps: let common_interrupt() handle IRQ_MOVE_CLEANUP_VECTOR x86/fred: make unions for the cs and ss fields in struct pt_regs x86/traps: add a system interrupt table for system interrupt dispatch x86/traps: add external_interrupt() to dispatch external interrupts x86/cpufeature: add the cpu feature bit for FRED x86/opcode: add ERETU, ERETS instructions to x86-opcode-map x86/objtool: teach objtool about ERETU and ERETS x86/cpu: add X86_CR4_FRED macro x86/fred: add Kconfig option for FRED (CONFIG_X86_FRED) x86/fred: if CONFIG_X86_FRED is disabled, disable FRED support x86/cpu: add MSR numbers for FRED configuration x86/fred: header file with FRED definitions x86/fred: reserve space for the FRED stack frame x86/fred: add a page fault entry stub for FRED x86/fred: add a debug fault entry stub for FRED x86/fred: add a NMI entry stub for FRED x86/fred: FRED entry/exit and dispatch code x86/fred: FRED initialization code x86/fred: update MSR_IA32_FRED_RSP0 during task switch x86/fred: let ret_from_fork() jmp to fred_exit_user when FRED is enabled x86/fred: disallow the swapgs instruction when FRED is enabled x86/fred: no ESPFIX needed when FRED is enabled x86/fred: allow single-step trap and NMI when starting a new thread x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1f Xin Li (9): x86/traps: add install_system_interrupt_handler() x86/fred: header file for event types x86/fred: add a machine check entry stub for FRED x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user x86/ia32: do not modify the DPL bits for a null selector x86/fred: allow dynamic stack frame size x86/fred: BUG() when ERETU with %rsp not equal to that when the ring 3 event was just delivered x86/fred: disable FRED by default in its early stage KVM: x86/vmx: refactor VMX_DO_EVENT_IRQOFF to generate FRED stack frames .../admin-guide/kernel-parameters.txt | 4 + arch/x86/Kconfig | 9 + arch/x86/entry/Makefile | 5 +- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 5 + arch/x86/entry/entry_64_fred.S | 62 +++++ arch/x86/entry/entry_fred.c | 222 ++++++++++++++++++ arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/asm-prototypes.h | 1 + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/event-type.h | 17 ++ arch/x86/include/asm/extable_fixup_types.h | 4 +- arch/x86/include/asm/fred.h | 150 ++++++++++++ arch/x86/include/asm/idtentry.h | 76 +++++- arch/x86/include/asm/irq.h | 5 + arch/x86/include/asm/irq_vectors.h | 15 +- arch/x86/include/asm/msr-index.h | 13 +- arch/x86/include/asm/processor.h | 12 +- arch/x86/include/asm/ptrace.h | 34 ++- arch/x86/include/asm/switch_to.h | 10 +- arch/x86/include/asm/thread_info.h | 35 +-- arch/x86/include/asm/traps.h | 13 + arch/x86/include/asm/vmx.h | 17 +- arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/apic/apic.c | 11 +- arch/x86/kernel/apic/vector.c | 8 +- arch/x86/kernel/cpu/acrn.c | 7 +- arch/x86/kernel/cpu/common.c | 88 ++++--- arch/x86/kernel/cpu/mce/core.c | 15 ++ arch/x86/kernel/cpu/mshyperv.c | 22 +- arch/x86/kernel/espfix_64.c | 8 + arch/x86/kernel/fred.c | 81 +++++++ arch/x86/kernel/head_32.S | 3 +- arch/x86/kernel/idt.c | 6 +- arch/x86/kernel/irq.c | 6 +- arch/x86/kernel/irqinit.c | 7 +- arch/x86/kernel/kvm.c | 4 +- arch/x86/kernel/nmi.c | 19 ++ arch/x86/kernel/process.c | 5 + arch/x86/kernel/process_64.c | 21 +- arch/x86/kernel/signal_32.c | 21 +- arch/x86/kernel/traps.c | 175 ++++++++++++-- arch/x86/kvm/vmx/vmenter.S | 78 +++++- arch/x86/kvm/vmx/vmx.c | 12 +- arch/x86/lib/x86-opcode-map.txt | 2 +- arch/x86/mm/extable.c | 84 +++++++ arch/x86/mm/fault.c | 20 +- drivers/xen/events/events_base.c | 5 +- kernel/fork.c | 6 + tools/arch/x86/include/asm/cpufeatures.h | 1 + .../arch/x86/include/asm/disabled-features.h | 8 +- tools/arch/x86/include/asm/msr-index.h | 13 +- tools/arch/x86/lib/x86-opcode-map.txt | 2 +- tools/objtool/arch/x86/decode.c | 19 +- 57 files changed, 1306 insertions(+), 179 deletions(-) create mode 100644 arch/x86/entry/entry_64_fred.S create mode 100644 arch/x86/entry/entry_fred.c create mode 100644 arch/x86/include/asm/event-type.h create mode 100644 arch/x86/include/asm/fred.h create mode 100644 arch/x86/kernel/fred.c