| Message ID | 20211215145633.5238-1-dwmw2@infradead.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Parallel CPU bringup for x86_64 | expand |
On 12/15/21 8:56 AM, David Woodhouse wrote: > Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for > them shaves about 80% off the AP bringup time on a 96-thread socket > Skylake box (EC2 c5.metal) — from about 500ms to 100ms. > > There are more wins to be had with further parallelisation, but this is > the simple part. I applied this series and began booting a regular non-SEV guest and hit a failure at 39 vCPUs. No panic or warning, just a reset and OVMF was executing again. I'll try to debug what's going, but not sure how quickly I'll arrive at anything. Thanks, Tom
On Thu, 2021-12-16 at 10:27 -0600, Tom Lendacky wrote: > On 12/15/21 8:56 AM, David Woodhouse wrote: > > Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for > > them shaves about 80% off the AP bringup time on a 96-thread socket > > Skylake box (EC2 c5.metal) — from about 500ms to 100ms. > > > > There are more wins to be had with further parallelisation, but this is > > the simple part. > > I applied this series and began booting a regular non-SEV guest and hit a > failure at 39 vCPUs. No panic or warning, just a reset and OVMF was > executing again. I'll try to debug what's going, but not sure how quickly > I'll arrive at anything. Thanks for testing. This is working for me with BIOS and EFI boots in qemu and real hardware but it's mostly been Intel so far. I'll try harder on an AMD box. Anything else special about your setup, kernel config or qemu invocation that might help me reproduce? If it can repro without KVM, 'qemu -d in_asm' can be extremely useful for this kind of thing btw.
On Thu, 2021-12-16 at 10:27 -0600, Tom Lendacky wrote: > On 12/15/21 8:56 AM, David Woodhouse wrote: > > > Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for > > them shaves about 80% off the AP bringup time on a 96-thread socket > > Skylake box (EC2 c5.metal) — from about 500ms to 100ms. > > > > There are more wins to be had with further parallelisation, but this is > > the simple part. > > I applied this series and began booting a regular non-SEV guest and hit a > failure at 39 vCPUs. No panic or warning, just a reset and OVMF was > executing again. I'll try to debug what's going, but not sure how quickly > I'll arrive at anything. I've pushed the SEV-ES fix to https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-5.16 and in doing so I've moved the 'no_parallel_bringup' command line argument earlier in the series, to Thomas's "Support parallel startup of secondary CPUs" commit (now 191f0899757). It would be interesting to see if you can reproduce with just that much, both with and with no_parallel_bringup. And then whether the subsequent commit that actually enables the parallel INIT/SIPI/SIPI actually makes the difference? Thanks!
On 12/16/21 1:52 PM, David Woodhouse wrote: > On Thu, 2021-12-16 at 10:27 -0600, Tom Lendacky wrote: >> On 12/15/21 8:56 AM, David Woodhouse wrote: >> >>> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for >>> them shaves about 80% off the AP bringup time on a 96-thread socket >>> Skylake box (EC2 c5.metal) — from about 500ms to 100ms. >>> >>> There are more wins to be had with further parallelisation, but this is >>> the simple part. >> >> I applied this series and began booting a regular non-SEV guest and hit a >> failure at 39 vCPUs. No panic or warning, just a reset and OVMF was >> executing again. I'll try to debug what's going, but not sure how quickly >> I'll arrive at anything. > > I've pushed the SEV-ES fix to > https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-5.16 > and in doing so I've moved the 'no_parallel_bringup' command line > argument earlier in the series, to Thomas's "Support parallel startup > of secondary CPUs" commit (now 191f0899757). It would be interesting to > see if you can reproduce with just that much, both with and with > no_parallel_bringup. And then whether the subsequent commit that > actually enables the parallel INIT/SIPI/SIPI actually makes the > difference? > I'll pull it down and give it try. Thanks, Tom > Thanks! >
On 16 December 2021 19:55:36 GMT, Tom Lendacky <thomas.lendacky@amd.com> wrote: >On 12/16/21 1:52 PM, David Woodhouse wrote: >> On Thu, 2021-12-16 at 10:27 -0600, Tom Lendacky wrote: >>> On 12/15/21 8:56 AM, David Woodhouse wrote: >>> >>>> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for >>>> them shaves about 80% off the AP bringup time on a 96-thread socket >>>> Skylake box (EC2 c5.metal) — from about 500ms to 100ms. >>>> >>>> There are more wins to be had with further parallelisation, but this is >>>> the simple part. >>> >>> I applied this series and began booting a regular non-SEV guest and hit a >>> failure at 39 vCPUs. No panic or warning, just a reset and OVMF was >>> executing again. I'll try to debug what's going, but not sure how quickly >>> I'll arrive at anything. >> >> I've pushed the SEV-ES fix to >> https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-5.16 >> and in doing so I've moved the 'no_parallel_bringup' command line >> argument earlier in the series, to Thomas's "Support parallel startup >> of secondary CPUs" commit (now 191f0899757). It would be interesting to >> see if you can reproduce with just that much, both with and with >> no_parallel_bringup. And then whether the subsequent commit that >> actually enables the parallel INIT/SIPI/SIPI actually makes the >> difference? >> > >I'll pull it down and give it try. Thanks. Note: don't use the whole thing; that last "parallel part2" patch in particular isn't ready. And probably isn't where the next low-hanging fruit is anyway.
On 12/16/21 1:24 PM, David Woodhouse wrote: > On Thu, 2021-12-16 at 10:27 -0600, Tom Lendacky wrote: >> On 12/15/21 8:56 AM, David Woodhouse wrote: >>> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for >>> them shaves about 80% off the AP bringup time on a 96-thread socket >>> Skylake box (EC2 c5.metal) — from about 500ms to 100ms. >>> >>> There are more wins to be had with further parallelisation, but this is >>> the simple part. >> >> I applied this series and began booting a regular non-SEV guest and hit a >> failure at 39 vCPUs. No panic or warning, just a reset and OVMF was >> executing again. I'll try to debug what's going, but not sure how quickly >> I'll arrive at anything. > > Thanks for testing. This is working for me with BIOS and EFI boots in > qemu and real hardware but it's mostly been Intel so far. I'll try > harder on an AMD box. On baremetal, I haven't seen an issue. This only seems to have a problem with Qemu/KVM. With 191f08997577 I could boot without issues with and without the no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I jumped to 128 vCPUs it failed again. When I moved the series to df9726cb7178, then 64 vCPUs also failed pretty consistently. Strange thing is it is random. Sometimes (rarely) it works on the first boot and then sometimes it doesn't, at which point it will reset and reboot 3 or 4 times and then make it past the failure and fully boot. > > Anything else special about your setup, kernel config or qemu > invocation that might help me reproduce? Shouldn't be anything special that I'm aware of: - EPYC 3rd Gen (Milan) - Qemu 6.1.0 - OVMF edk2-stable202111 The qemu command line is: qemu-system-x86_64 -enable-kvm -cpu EPYC,host-phys-bits=true -smp 128 -m 1G -machine type=q35 -drive if=pflash,format=raw,unit=0,file=/root/kernels/qemu-install/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=./diskless.fd -nographic -kernel /root/kernels/linux-build-x86_64/arch/x86/boot/bzImage -append "console=ttyS0,115200n8" -monitor pty -monitor unix:monitor,server,nowait I can send the kernel config to you offlist if you're unable to repro with yours. > > If it can repro without KVM, 'qemu -d in_asm' can be extremely useful > for this kind of thing btw. I didn't repro the failure without KVM. Thanks, Tom >
On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: > On baremetal, I haven't seen an issue. This only seems to have a problem > with Qemu/KVM. > > With 191f08997577 I could boot without issues with and without the > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I > jumped to 128 vCPUs it failed again. When I moved the series to > df9726cb7178, then 64 vCPUs also failed pretty consistently. > > Strange thing is it is random. Sometimes (rarely) it works on the first > boot and then sometimes it doesn't, at which point it will reset and > reboot 3 or 4 times and then make it past the failure and fully boot. Hm, some of that is just artifacts of timing, I'm sure. But now I'm staring at the way that early_setup_idt() can run in parallel on all CPUs, rewriting bringup_idt_descr and loading it. To start with, let's try unlocking the trampoline_lock much later, after cpu_init_exception_handling() has loaded the real IDT. I think we can probably make secondaries load the real IDT early and never use bringup_idt_descr at all, can't we? But let's see if this makes it go away, to start with... diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 0cd6373bc3f2..2307f7575ab4 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -59,7 +59,7 @@ #include <asm/cpu_device_id.h> #include <asm/uv/uv.h> #include <asm/sigframe.h> - +#include <asm/realmode.h> #include "cpu.h" u32 elf_hwcap2 __read_mostly; @@ -2060,6 +2060,7 @@ void cpu_init_secondary(void) * on this CPU in cpu_init_exception_handling(). */ cpu_init_exception_handling(); + clear_bit(0, (unsigned long *)trampoline_lock); cpu_init(); } #endif diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 3e4c3c416bce..db01b56574cd 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -273,14 +273,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ movq initial_stack(%rip), %rsp - /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ - movq trampoline_lock(%rip), %rax - testq %rax, %rax - jz .Lsetup_idt - lock - btrl $0, (%rax) - -.Lsetup_idt: /* Setup and Load IDT */ pushq %rsi call early_setup_idt
On Fri, 17 Dec 2021 00:13:16 +0000 David Woodhouse <dwmw2@infradead.org> wrote: > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: > > On baremetal, I haven't seen an issue. This only seems to have a problem > > with Qemu/KVM. > > > > With 191f08997577 I could boot without issues with and without the > > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. > > > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I > > jumped to 128 vCPUs it failed again. When I moved the series to > > df9726cb7178, then 64 vCPUs also failed pretty consistently. > > > > Strange thing is it is random. Sometimes (rarely) it works on the first > > boot and then sometimes it doesn't, at which point it will reset and > > reboot 3 or 4 times and then make it past the failure and fully boot. > > Hm, some of that is just artifacts of timing, I'm sure. But now I'm that's most likely the case (there is a race somewhere left). To trigger CPU bringup (hotplug) races, I used to run QEMU guest with heavy vCPU overcommit. It helps to induce unexpected delays at CPU bringup time. > staring at the way that early_setup_idt() can run in parallel on all > CPUs, rewriting bringup_idt_descr and loading it. > > To start with, let's try unlocking the trampoline_lock much later, > after cpu_init_exception_handling() has loaded the real IDT. > > I think we can probably make secondaries load the real IDT early and > never use bringup_idt_descr at all, can't we? But let's see if this > makes it go away, to start with... > > diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c > index 0cd6373bc3f2..2307f7575ab4 100644 > --- a/arch/x86/kernel/cpu/common.c > +++ b/arch/x86/kernel/cpu/common.c > @@ -59,7 +59,7 @@ > #include <asm/cpu_device_id.h> > #include <asm/uv/uv.h> > #include <asm/sigframe.h> > - > +#include <asm/realmode.h> > #include "cpu.h" > > u32 elf_hwcap2 __read_mostly; > @@ -2060,6 +2060,7 @@ void cpu_init_secondary(void) > * on this CPU in cpu_init_exception_handling(). > */ > cpu_init_exception_handling(); > + clear_bit(0, (unsigned long *)trampoline_lock); > cpu_init(); > } > #endif > diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S > index 3e4c3c416bce..db01b56574cd 100644 > --- a/arch/x86/kernel/head_64.S > +++ b/arch/x86/kernel/head_64.S > @@ -273,14 +273,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) > */ > movq initial_stack(%rip), %rsp > > - /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ > - movq trampoline_lock(%rip), %rax > - testq %rax, %rax > - jz .Lsetup_idt > - lock > - btrl $0, (%rax) > - > -.Lsetup_idt: > /* Setup and Load IDT */ > pushq %rsi > call early_setup_idt
On Fri, 2021-12-17 at 11:09 +0100, Igor Mammedov wrote: > that's most likely the case (there is a race somewhere left). > To trigger CPU bringup (hotplug) races, I used to run QEMU guest with > heavy vCPU overcommit. It helps to induce unexpected delays at CPU bringup > time. Yeah, I've been doing a fair amount of that but even with Tom's config I can't reproduce a crash. Have seen this one now though. It's hard to reproduce, and I suspect it was there already and I've only tweaked the timing to expose it (or not even that, and just done enough tests that I've seen it when it's extremely sporadic). [ 0.061937] kvm-clock: cpu 24, msr 31801601, secondary cpu clock [ 0.668842] kvm-guest: stealtime: cpu 24, msr 37c31080 [ 0.061937] kvm-clock: cpu 25, msr 31801641, secondary cpu clock [ 0.670557] kvm-guest: stealtime: cpu 25, msr 37c71080 [ 0.670557] ------------[ cut here ]------------ [ 0.670557] cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg [ 0.670557] WARNING: CPU: 25 PID: 140 at kernel/sched/fair.c:3299 __update_blocked_fair+0x4b7/0x4d0 [ 0.061937] kvm-clock: cpu 26, msr 31801681, secondary cpu clock [ 0.670740] Modules linked in: [ 0.670740] CPU: 25 PID: 140 Comm: kworker/25:0H Not tainted 5.16.0-rc2-sos-testing+ #963 [ 0.670740] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 [ 0.670740] RIP: 0010:__update_blocked_fair+0x4b7/0x4d0 [ 0.670740] Code: 4f fd ff ff 49 8b 96 48 01 00 00 48 89 90 60 09 00 00 e9 e3 fc ff ff 48 c7 c7 30 36 34 b6 c6 05 77 78 ae 01 01 e8 d5 58 96 00 <0f> 0b 41 8b 86 38 01 00 00 e9 aa fc ff ff 66 66 2e 0f 1f 84 00 00 [ 0.670740] RSP: 0018:ffffc90000cc7d30 EFLAGS: 00010086 [ 0.670740] RAX: 0000000000000000 RBX: 00000000000000c8 RCX: 0000000000000000 [ 0.670740] RDX: 0000000000000003 RSI: ffff88803bbfffe8 RDI: 00000000ffffffff [ 0.670740] RBP: ffff888037c6f800 R08: 00000000ffffffea R09: 0000000000000000 [ 0.670740] R10: 0000000000000003 R11: 3fffffffffffffff R12: ffff888037c6ff90 [ 0.670740] R13: ffff888037c6fe50 R14: ffff888037c6f6c0 R15: 0000000000000000 [ 0.670740] FS: 0000000000000000(0000) GS:ffff888037c40000(0000) knlGS:0000000000000000 [ 0.670740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.670740] CR2: 0000000000000000 CR3: 000000003060a000 CR4: 0000000000350ee0 [ 0.670740] Call Trace: [ 0.670740] <TASK> [ 0.670740] update_blocked_averages+0x98/0x160 [ 0.670740] newidle_balance+0x117/0x390 [ 0.670740] pick_next_task_fair+0x39/0x3c0 [ 0.670740] __schedule+0x156/0x6f0 [ 0.670740] schedule+0x4e/0xc0 [ 0.670740] worker_thread+0xb1/0x300 [ 0.670740] ? rescuer_thread+0x370/0x370 [ 0.687790] kvm-guest: stealtime: cpu 26, msr 37cb1080 [ 0.061937] kvm-clock: cpu 27, msr 318016c1, secondary cpu clock [ 0.690740] kvm-guest: stealtime: cpu 27, msr 37cf1080 [ 0.061937] kvm-clock: cpu 28, msr 31801701, secondary cpu clock [ 0.693781] kvm-guest: stealtime: cpu 28, msr 37d31080 [ 0.670740] kthread+0x158/0x180 [ 0.061937] kvm-clock: cpu 29, msr 31801741, secondary cpu clock [ 0.670740] ? set_kthread_struct+0x40/0x40 [ 0.670740] ret_from_fork+0x22/0x30 [ 0.670740] </TASK> [ 0.670740] ---[ end trace ac8562dd64da6bb5 ]--- [ 0.747785] kvm-guest: stealtime: cpu 29, msr 37d71080 [ 0.061937] kvm-clock: cpu 30, msr 31801781, secondary cpu clock [ 0.756784] kvm-guest: stealtime: cpu 30, msr 37db1080
On 12/16/21 6:13 PM, David Woodhouse wrote: > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: >> On baremetal, I haven't seen an issue. This only seems to have a problem >> with Qemu/KVM. >> >> With 191f08997577 I could boot without issues with and without the >> no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. >> >> With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I >> jumped to 128 vCPUs it failed again. When I moved the series to >> df9726cb7178, then 64 vCPUs also failed pretty consistently. >> >> Strange thing is it is random. Sometimes (rarely) it works on the first >> boot and then sometimes it doesn't, at which point it will reset and >> reboot 3 or 4 times and then make it past the failure and fully boot. > > Hm, some of that is just artifacts of timing, I'm sure. But now I'm > staring at the way that early_setup_idt() can run in parallel on all > CPUs, rewriting bringup_idt_descr and loading it. > > To start with, let's try unlocking the trampoline_lock much later, > after cpu_init_exception_handling() has loaded the real IDT. > > I think we can probably make secondaries load the real IDT early and > never use bringup_idt_descr at all, can't we? But let's see if this > makes it go away, to start with... > This still fails. I ran with -d cpu_reset on the command line and will forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c" and got: 128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0 128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 These are before running any of the vCPUs. 1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 This is where vCPU0 is at the time of the reset. This address tends to be different all the time and so I think it is just where it happens to be when the reset occurs and isn't contributing to the reset. 5 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 1 RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 15 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 These are some of the APs and all are in wait_for_master_cpu(). 1 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 This seems ok because: CS =9900 00099000 0000ffff 00009b00 So likely in the trampoline code. 1 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 This one seems odd... could it be the one causing the reset? CS =f000 ffff0000 0000ffff 00009a00 3 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 2 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 99 EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1 Thanks, Tom
On Fri, 2021-12-17 at 11:48 -0600, Tom Lendacky wrote: > On 12/16/21 6:13 PM, David Woodhouse wrote: > > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: > > > On baremetal, I haven't seen an issue. This only seems to have a problem > > > with Qemu/KVM. > > > > > > With 191f08997577 I could boot without issues with and without the > > > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. > > > > > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I > > > jumped to 128 vCPUs it failed again. When I moved the series to > > > df9726cb7178, then 64 vCPUs also failed pretty consistently. > > > > > > Strange thing is it is random. Sometimes (rarely) it works on the first > > > boot and then sometimes it doesn't, at which point it will reset and > > > reboot 3 or 4 times and then make it past the failure and fully boot. > > > > Hm, some of that is just artifacts of timing, I'm sure. But now I'm > > staring at the way that early_setup_idt() can run in parallel on all > > CPUs, rewriting bringup_idt_descr and loading it. > > > > To start with, let's try unlocking the trampoline_lock much later, > > after cpu_init_exception_handling() has loaded the real IDT. > > > > I think we can probably make secondaries load the real IDT early and > > never use bringup_idt_descr at all, can't we? But let's see if this > > makes it go away, to start with... > > > > This still fails. I ran with -d cpu_reset on the command line and will > forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c" > and got: > > 128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0 > 128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 > These are before running any of the vCPUs. > > 1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > This is where vCPU0 is at the time of the reset. This address tends to > be different all the time and so I think it is just where it happens to > be when the reset occurs and isn't contributing to the reset. I note that one is in native_write_msr() though. I wonder what it's writing? Do you have console output (perhaps with earlyprintk=ttyS0) to go with this? > 5 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > 1 RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > 15 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > These are some of the APs and all are in wait_for_master_cpu(). As is right and proper. They should be coming up to that point and waiting for the... erm... controlling CPU to tell them to go any further. > 1 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 > This seems ok because: CS =9900 00099000 0000ffff 00009b00 > So likely in the trampoline code. Yeah, that'll be in the bitlock waiting for its turn through the real mode stack. 1010: 66 0f ba 26 18 btw $0x18,(%esi) 1015: 40 inc %eax 1016: 00 73 04 add %dh,0x4(%ebx) 1019: f3 90 pause 101b: eb f3 jmp 1010 <trampoline_start+0x10> > 1 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 > This one seems odd... could it be the one causing the reset? > CS =f000 ffff0000 0000ffff 00009a00 Yeah. I'm finding it slightly easier without the 'uniq'... > CPU Reset (CPU 0) > RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 1) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 2) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 3) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 4) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 5) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 6) > RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 7) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 8) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 9) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 10) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 11) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 12) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 13) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 14) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 15) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 16) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 17) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 18) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 19) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 20) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 21) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 All those came up and are waiting in wait_for_master_cpu() as they should. > CPU Reset (CPU 22) > EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 That one's in the bitlock, also waiting. > CPU Reset (CPU 23) > EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 This one we suspect. Is this what a triple-fault would look like? Not if it's *already* at f000:fff0, surely? CPU Reset (CPU 23) EAX=00000000 EBX=00000000 ECX=00000000 EDX=00800f12 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000 0000ffff 00009a00 SS =0000 00000000 0000ffff 00009200 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008300 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 CCS=00000000 CCD=00000000 CCO=DYNAMIC EFER=0000000000000000 FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80 FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=0000000000000000 0000000000000000 XMM01=0000000000000000 0000000000000000 XMM02=0000000000000000 0000000000000000 XMM03=0000000000000000 0000000000000000 XMM04=0000000000000000 0000000000000000 XMM05=0000000000000000 0000000000000000 XMM06=0000000000000000 0000000000000000 XMM07=0000000000000000 0000000000000000 > CPU Reset (CPU 24) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 25) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 26) > RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 These ones made it through the real mode first and are also waiting. > CPU Reset (CPU 27) > EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 28) > EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0 > CPU Reset (CPU 29) Still in the real mode bitlock. And after this point they are still halted in presumably 32-bit BIOS code because the BSP hasn't even touched them yet. > EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1 > CPU Reset (CPU 30) > EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1 > CPU Reset (CPU 31) > EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1 > CPU Reset (CPU 32) > ... > CPU Reset (CPU 127) > EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
On Fri, 2021-12-17 at 19:11 +0000, David Woodhouse wrote: > I note that one is in native_write_msr() though. I wonder what it's > writing? CPU Reset (CPU 0) RAX=0000000000000000 RBX=0000000000000202 RCX=0000000000000828 RDX=0000000000000000 RSI=0000000000000000 RDI=0000000000000828 RBP=0000000000000000 RSP=ffffc90000023ce0 R8 =0000000000000000 R9 =ffffc90000023b60 R10=0000000000000001 R11=0000000000000001 R12=000000000000069a R13=0000000000000005 R14=000000000000001c R15=0000000000000001 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 It's writing zero (%rax/%rsi) to MSR 0x828 (%rcx/%rdi) which is the X2APIC's APIC_ESR. Can you reproduce this without the guest being in X2APIC mode? You'll have to cut it back to only 254 vCPUs for that test.
On 12/17/21 1:11 PM, David Woodhouse wrote: > On Fri, 2021-12-17 at 11:48 -0600, Tom Lendacky wrote: >> On 12/16/21 6:13 PM, David Woodhouse wrote: >>> On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: >>>> On baremetal, I haven't seen an issue. This only seems to have a problem >>>> with Qemu/KVM. >>>> >>>> With 191f08997577 I could boot without issues with and without the >>>> no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. >>>> >>>> With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I >>>> jumped to 128 vCPUs it failed again. When I moved the series to >>>> df9726cb7178, then 64 vCPUs also failed pretty consistently. >>>> >>>> Strange thing is it is random. Sometimes (rarely) it works on the first >>>> boot and then sometimes it doesn't, at which point it will reset and >>>> reboot 3 or 4 times and then make it past the failure and fully boot. >>> >>> Hm, some of that is just artifacts of timing, I'm sure. But now I'm >>> staring at the way that early_setup_idt() can run in parallel on all >>> CPUs, rewriting bringup_idt_descr and loading it. >>> >>> To start with, let's try unlocking the trampoline_lock much later, >>> after cpu_init_exception_handling() has loaded the real IDT. >>> >>> I think we can probably make secondaries load the real IDT early and >>> never use bringup_idt_descr at all, can't we? But let's see if this >>> makes it go away, to start with... >>> >> >> This still fails. I ran with -d cpu_reset on the command line and will >> forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c" >> and got: >> >> 128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0 >> 128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 >> These are before running any of the vCPUs. >> >> 1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 >> This is where vCPU0 is at the time of the reset. This address tends to >> be different all the time and so I think it is just where it happens to >> be when the reset occurs and isn't contributing to the reset. > > I note that one is in native_write_msr() though. I wonder what it's writing? > > Do you have console output (perhaps with earlyprintk=ttyS0) to go with this? Yes, but it's not really much help... [ 0.146318] Freeing SMP alternatives memory: 36K [ 0.249121] smpboot: CPU0: AMD EPYC Processor (family: 0x17, model: 0x1, stepping: 0x2) [ 0.249291] Performance Events: AMD PMU driver. [ 0.249771] ... version: 0 [ 0.250170] ... bit width: 48 [ 0.250258] ... generic registers: 4 [ 0.250662] ... value mask: 0000ffffffffffff [ 0.251258] ... max period: 00007fffffffffff [ 0.251790] ... fixed-purpose events: 0 [ 0.252258] ... event mask: 000000000000000f [ 0.252972] rcu: Hierarchical SRCU implementation. [ 0.255797] smp: Bringing up secondary CPUs ... [ 0.256372] x86: Booting SMP configuration: SecCoreStartupWithStack(0xFFFCC000, 0x820000) Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000 There's no WARN or PANIC, just a reset. I can look to try and capture some KVM trace data if that would help. If so, let me know what events you'd like captured. > >> CPU Reset (CPU 23) >> EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 > > This one we suspect. Is this what a triple-fault would look like? Not > if it's *already* at f000:fff0, surely? Good question. The APM doesn't really document it. I'll see if I can find some h/w folks to check with. Thanks, Tom > > CPU Reset (CPU 23) > EAX=00000000 EBX=00000000 ECX=00000000 EDX=00800f12 > ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 > EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 > ES =0000 00000000 0000ffff 00009300 > CS =f000 ffff0000 0000ffff 00009a00 > SS =0000 00000000 0000ffff 00009200 > DS =0000 00000000 0000ffff 00009300 > FS =0000 00000000 0000ffff 00009300 > GS =0000 00000000 0000ffff 00009300 > LDT=0000 00000000 0000ffff 00008200 > TR =0000 00000000 0000ffff 00008300 > GDT= 00000000 0000ffff > IDT= 00000000 0000ffff > CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 > DR6=00000000ffff0ff0 DR7=0000000000000400 > CCS=00000000 CCD=00000000 CCO=DYNAMIC > EFER=0000000000000000 > FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80 > FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 > FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 > FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 > FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 > XMM00=0000000000000000 0000000000000000 XMM01=0000000000000000 0000000000000000 > XMM02=0000000000000000 0000000000000000 XMM03=0000000000000000 0000000000000000 > XMM04=0000000000000000 0000000000000000 XMM05=0000000000000000 0000000000000000 > XMM06=0000000000000000 0000000000000000 XMM07=0000000000000000 0000000000000000 >
On Fri, 2021-12-17 at 13:46 -0600, Tom Lendacky wrote: > There's no WARN or PANIC, just a reset. I can look to try and capture some > KVM trace data if that would help. If so, let me know what events you'd > like captured. Could start with just kvm_run_exit? Reason 8 would be KVM_EXIT_SHUTDOWN and would potentially indicate a triple fault. Failing that, I'd want to start littering the real mode code with outputting 'a' 'b' 'c' etc. to the serial port and see if the offending CPU is really in the trampoline somewhere when something goes wrong. I can knock up an example patch to do that (not tonight) but this would be somewhat easier if I could find a machine I can reproduce on. Sadly I only seem to have access to Milan *guests* without nested virt, not bare metal. Got a machine I can log in to?
On 12/17/21 1:26 PM, David Woodhouse wrote: > On Fri, 2021-12-17 at 19:11 +0000, David Woodhouse wrote: >> I note that one is in native_write_msr() though. I wonder what it's >> writing? > > CPU Reset (CPU 0) > RAX=0000000000000000 RBX=0000000000000202 RCX=0000000000000828 RDX=0000000000000000 > RSI=0000000000000000 RDI=0000000000000828 RBP=0000000000000000 RSP=ffffc90000023ce0 > R8 =0000000000000000 R9 =ffffc90000023b60 R10=0000000000000001 R11=0000000000000001 > R12=000000000000069a R13=0000000000000005 R14=000000000000001c R15=0000000000000001 > RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > > It's writing zero (%rax/%rsi) to MSR 0x828 (%rcx/%rdi) which is the > X2APIC's APIC_ESR. > > Can you reproduce this without the guest being in X2APIC mode? You'll > have to cut it back to only 254 vCPUs for that test. Yes, reproducible with guest in xAPIC mode. Thanks, Tom >
On 12/17/21 2:13 PM, David Woodhouse wrote: > On Fri, 2021-12-17 at 13:46 -0600, Tom Lendacky wrote: >> There's no WARN or PANIC, just a reset. I can look to try and capture some >> KVM trace data if that would help. If so, let me know what events you'd >> like captured. > > > Could start with just kvm_run_exit? > > Reason 8 would be KVM_EXIT_SHUTDOWN and would potentially indicate a > triple fault. qemu-system-x86-24093 [005] ..... 1601.759486: kvm_exit: vcpu 112 reason shutdown rip 0xffffffff81070574 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x80000b08 error_code 0x00000000 # addr2line -e woodhouse-build-x86_64/vmlinux 0xffffffff81070574 /root/kernels/woodhouse-build-x86_64/./arch/x86/include/asm/desc.h:272 Which is: asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); Thanks, Tom > > Failing that, I'd want to start littering the real mode code with > outputting 'a' 'b' 'c' etc. to the serial port and see if the offending > CPU is really in the trampoline somewhere when something goes wrong. > > I can knock up an example patch to do that (not tonight) but this would > be somewhat easier if I could find a machine I can reproduce on. Sadly > I only seem to have access to Milan *guests* without nested virt, not > bare metal. Got a machine I can log in to? > > >
On Fri, 2021-12-17 at 14:55 -0600, Tom Lendacky wrote: > On 12/17/21 2:13 PM, David Woodhouse wrote: > > On Fri, 2021-12-17 at 13:46 -0600, Tom Lendacky wrote: > > > There's no WARN or PANIC, just a reset. I can look to try and capture some > > > KVM trace data if that would help. If so, let me know what events you'd > > > like captured. > > > > > > Could start with just kvm_run_exit? > > > > Reason 8 would be KVM_EXIT_SHUTDOWN and would potentially indicate a > > triple fault. > > qemu-system-x86-24093 [005] ..... 1601.759486: kvm_exit: vcpu 112 reason shutdown rip 0xffffffff81070574 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x80000b08 error_code 0x00000000 > > # addr2line -e woodhouse-build-x86_64/vmlinux 0xffffffff81070574 > /root/kernels/woodhouse-build-x86_64/./arch/x86/include/asm/desc.h:272 > > Which is: asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); OK, that seems like enough of a smoking gun; I'll stare at that harder. Thanks. /* * The LTR instruction marks the TSS GDT entry as busy. On 64-bit, the GDT is * a read-only remapping. To prevent a page fault, the GDT is switched to the * original writeable version when needed. */ #ifdef CONFIG_X86_64 static inline void native_load_tr_desc(void) { struct desc_ptr gdt; int cpu = raw_smp_processor_id(); bool restore = 0; struct desc_struct *fixmap_gdt; native_store_gdt(&gdt); fixmap_gdt = get_cpu_gdt_ro(cpu); /* * If the current GDT is the read-only fixmap, swap to the original * writeable version. Swap back at the end. */ if (gdt.address == (unsigned long)fixmap_gdt) { load_direct_gdt(cpu); restore = 1; } asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); if (restore) load_fixmap_gdt(cpu); }
On Fri, 2021-12-17 at 11:09 +0100, Igor Mammedov wrote: > On Fri, 17 Dec 2021 00:13:16 +0000 > David Woodhouse <dwmw2@infradead.org> wrote: > > > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: > > > On baremetal, I haven't seen an issue. This only seems to have a problem > > > with Qemu/KVM. > > > > > > With 191f08997577 I could boot without issues with and without the > > > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. > > > > > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I > > > jumped to 128 vCPUs it failed again. When I moved the series to > > > df9726cb7178, then 64 vCPUs also failed pretty consistently. > > > > > > Strange thing is it is random. Sometimes (rarely) it works on the first > > > boot and then sometimes it doesn't, at which point it will reset and > > > reboot 3 or 4 times and then make it past the failure and fully boot. > > > > Hm, some of that is just artifacts of timing, I'm sure. But now I'm > > that's most likely the case (there is a race somewhere left). > To trigger CPU bringup (hotplug) races, I used to run QEMU guest with > heavy vCPU overcommit. It helps to induce unexpected delays at CPU bringup > time. That last commit which actually enables parallel bringup does *two* things. It makes the generic cpuhp code bring all the CPUs through all the CPUHP_*_PREPARE stages and then actually brings them up. With that test patch I sent, the bringup basically *wasn't* parallel any more; they were using the trampoline lock all the way to the point where they start waiting on cpu_callin_mask. So maybe it's the 'prepare' ordering, like the x2apic one I already fixed... but some weirdness that only triggers on some CPUs. Can we back out the actual pseudo-parallel bringup and do *only* the prepare part, by doing something like this on top... --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1337,7 +1337,7 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) int ret; /* If parallel AP bringup isn't enabled, perform the first steps now. */ - if (!do_parallel_bringup) { + if (1 || !do_parallel_bringup) { ret = do_cpu_up(cpu, tidle); if (ret) return ret; @@ -1366,7 +1366,8 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) /* Bringup step one: Send INIT/SIPI to the target AP */ static int native_cpu_kick(unsigned int cpu) { - return do_cpu_up(cpu, idle_thread_get(cpu)); + return 0; + // return do_cpu_up(cpu, idle_thread_get(cpu)); } /**
On 12/20/21 11:10 AM, David Woodhouse wrote: > On Fri, 2021-12-17 at 11:09 +0100, Igor Mammedov wrote: >> On Fri, 17 Dec 2021 00:13:16 +0000 >> David Woodhouse <dwmw2@infradead.org> wrote: >> >>> On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote: >>>> On baremetal, I haven't seen an issue. This only seems to have a problem >>>> with Qemu/KVM. >>>> >>>> With 191f08997577 I could boot without issues with and without the >>>> no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen. >>>> >>>> With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I >>>> jumped to 128 vCPUs it failed again. When I moved the series to >>>> df9726cb7178, then 64 vCPUs also failed pretty consistently. >>>> >>>> Strange thing is it is random. Sometimes (rarely) it works on the first >>>> boot and then sometimes it doesn't, at which point it will reset and >>>> reboot 3 or 4 times and then make it past the failure and fully boot. >>> >>> Hm, some of that is just artifacts of timing, I'm sure. But now I'm >> >> that's most likely the case (there is a race somewhere left). >> To trigger CPU bringup (hotplug) races, I used to run QEMU guest with >> heavy vCPU overcommit. It helps to induce unexpected delays at CPU bringup >> time. > > That last commit which actually enables parallel bringup does *two* > things. It makes the generic cpuhp code bring all the CPUs through all > the CPUHP_*_PREPARE stages and then actually brings them up. With that > test patch I sent, the bringup basically *wasn't* parallel any more; > they were using the trampoline lock all the way to the point where they > start waiting on cpu_callin_mask. > > So maybe it's the 'prepare' ordering, like the x2apic one I already > fixed... but some weirdness that only triggers on some CPUs. Can we > back out the actual pseudo-parallel bringup and do *only* the prepare > part, by doing something like this on top... > > --- a/arch/x86/kernel/smpboot.c > +++ b/arch/x86/kernel/smpboot.c > @@ -1337,7 +1337,7 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) > int ret; > > /* If parallel AP bringup isn't enabled, perform the first steps now. */ > - if (!do_parallel_bringup) { > + if (1 || !do_parallel_bringup) { > ret = do_cpu_up(cpu, tidle); > if (ret) > return ret; > @@ -1366,7 +1366,8 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) > /* Bringup step one: Send INIT/SIPI to the target AP */ > static int native_cpu_kick(unsigned int cpu) > { > - return do_cpu_up(cpu, idle_thread_get(cpu)); > + return 0; > + // return do_cpu_up(cpu, idle_thread_get(cpu)); > } Took the tree back to commit df9726cb7178 and then applied this change. I'm unable to trigger any kind of failure with this change. Thanks, Tom > > /** > >
On Mon, 2021-12-20 at 12:54 -0600, Tom Lendacky wrote: > Took the tree back to commit df9726cb7178 and then applied this change. > I'm unable to trigger any kind of failure with this change. Hm... I fired up an EC2 m6a.48xlarge instance (192 CPUs) to play with. I can reproduce your triple-fault on SMP bringup, but only with kexec. And I basically can't get *anything* to kexec without that triple- fault. Not a clean 5.16-rc2, not the Fedora stock 5.14.10 kernel. If I *boot* instead of kexec, I have not yet seen the problem at all. This is using Legacy BIOS not UEFI.
On 12/20/21 3:29 PM, David Woodhouse wrote: > On Mon, 2021-12-20 at 12:54 -0600, Tom Lendacky wrote: >> Took the tree back to commit df9726cb7178 and then applied this change. >> I'm unable to trigger any kind of failure with this change. > > Hm... I fired up an EC2 m6a.48xlarge instance (192 CPUs) to play with. > > I can reproduce your triple-fault on SMP bringup, but only with kexec. > And I basically can't get *anything* to kexec without that triple- > fault. Not a clean 5.16-rc2, not the Fedora stock 5.14.10 kernel. > > If I *boot* instead of kexec, I have not yet seen the problem at all. > This is using Legacy BIOS not UEFI. Let me try with a legacy BIOS and see if I can repro. Might not be until tomorrow, though, since I had to let someone borrow the machine. Thanks, Tom > >
On 12/20/21 3:47 PM, Tom Lendacky wrote: > On 12/20/21 3:29 PM, David Woodhouse wrote: >> On Mon, 2021-12-20 at 12:54 -0600, Tom Lendacky wrote: >>> Took the tree back to commit df9726cb7178 and then applied this change. >>> I'm unable to trigger any kind of failure with this change. >> >> Hm... I fired up an EC2 m6a.48xlarge instance (192 CPUs) to play with. >> >> I can reproduce your triple-fault on SMP bringup, but only with kexec. >> And I basically can't get *anything* to kexec without that triple- >> fault. Not a clean 5.16-rc2, not the Fedora stock 5.14.10 kernel. >> >> If I *boot* instead of kexec, I have not yet seen the problem at all. >> This is using Legacy BIOS not UEFI. > > Let me try with a legacy BIOS and see if I can repro. Might not be until > tomorrow, though, since I had to let someone borrow the machine. I still encounter the issue using a legacy BIOS (SeaBIOS). Thanks, Tom > > Thanks, > Tom > >> >>
On 21 December 2021 22:25:35 GMT, Tom Lendacky <thomas.lendacky@amd.com> wrote: >On 12/20/21 3:47 PM, Tom Lendacky wrote: >> On 12/20/21 3:29 PM, David Woodhouse wrote: >>> On Mon, 2021-12-20 at 12:54 -0600, Tom Lendacky wrote: >>>> Took the tree back to commit df9726cb7178 and then applied this change. >>>> I'm unable to trigger any kind of failure with this change. >>> >>> Hm... I fired up an EC2 m6a.48xlarge instance (192 CPUs) to play with. >>> >>> I can reproduce your triple-fault on SMP bringup, but only with kexec. >>> And I basically can't get *anything* to kexec without that triple- >>> fault. Not a clean 5.16-rc2, not the Fedora stock 5.14.10 kernel. >>> >>> If I *boot* instead of kexec, I have not yet seen the problem at all. >>> This is using Legacy BIOS not UEFI. >> >> Let me try with a legacy BIOS and see if I can repro. Might not be until >> tomorrow, though, since I had to let someone borrow the machine. > >I still encounter the issue using a legacy BIOS (SeaBIOS). I haven't had much time to play but have seen it with a stock kernel at least as far back as v5.0. They all triple-fault on bringing up secondary CPUs, on kexec.
Dear David, Am 15.12.21 um 15:56 schrieb David Woodhouse: > Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for > them shaves about 80% off the AP bringup time on a 96-thread socket > Skylake box (EC2 c5.metal) — from about 500ms to 100ms. > > There are more wins to be had with further parallelisation, but this is > the simple part. > > v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more > v3: Clean up x2apic patch, add MTRR optimisation, lock topology update > in preparation for more parallelisation. > > > David Woodhouse (8): > x86/apic/x2apic: Fix parallel handling of cluster_mask > cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> > cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU > x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() > x86/smpboot: Split up native_cpu_up into separate phases and document them > x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel > x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup > x86/smpboot: Serialize topology updates for secondary bringup > > Thomas Gleixner (1): > x86/smpboot: Support parallel startup of secondary CPUs > > arch/x86/include/asm/realmode.h | 3 + > arch/x86/include/asm/smp.h | 13 +- > arch/x86/include/asm/topology.h | 2 - > arch/x86/kernel/acpi/sleep.c | 1 + > arch/x86/kernel/apic/apic.c | 2 +- > arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++++----- > arch/x86/kernel/cpu/common.c | 6 +- > arch/x86/kernel/cpu/mtrr/mtrr.c | 9 + > arch/x86/kernel/head_64.S | 71 ++++++++ > arch/x86/kernel/smpboot.c | 324 ++++++++++++++++++++++++---------- > arch/x86/realmode/init.c | 3 + > arch/x86/realmode/rm/trampoline_64.S | 14 ++ > arch/x86/xen/smp_pv.c | 4 +- > include/linux/cpuhotplug.h | 2 + > include/linux/smpboot.h | 7 + > kernel/cpu.c | 27 ++- > kernel/smpboot.c | 2 +- > kernel/smpboot.h | 2 - > 18 files changed, 441 insertions(+), 159 deletions(-) Thank you for working on this. I tested this on a MSI MS-7A37/B350M MORTAR (BIOS 1.MW 11/01/2021) with a Ryzen 3 2200G, but nothing was printed to the screen after the GRUB loading messages, so it crashed or hung somewhere. Unfortunately, this device is used by others, and no serial console is connected and I do not know how to capture the Linux log with other means. Kind regards, Paul
Dear David, Am 27.12.21 um 17:57 schrieb Paul Menzel: > Am 15.12.21 um 15:56 schrieb David Woodhouse: >> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for >> them shaves about 80% off the AP bringup time on a 96-thread socket >> Skylake box (EC2 c5.metal) — from about 500ms to 100ms. >> >> There are more wins to be had with further parallelisation, but this is >> the simple part. >> >> v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more >> v3: Clean up x2apic patch, add MTRR optimisation, lock topology update >> in preparation for more parallelisation. >> >> >> David Woodhouse (8): >> x86/apic/x2apic: Fix parallel handling of cluster_mask >> cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> >> cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU >> x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() >> x86/smpboot: Split up native_cpu_up into separate phases and document them >> x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel >> x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup >> x86/smpboot: Serialize topology updates for secondary bringup >> >> Thomas Gleixner (1): >> x86/smpboot: Support parallel startup of secondary CPUs >> >> arch/x86/include/asm/realmode.h | 3 + >> arch/x86/include/asm/smp.h | 13 +- >> arch/x86/include/asm/topology.h | 2 - >> arch/x86/kernel/acpi/sleep.c | 1 + >> arch/x86/kernel/apic/apic.c | 2 +- >> arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++++----- >> arch/x86/kernel/cpu/common.c | 6 +- >> arch/x86/kernel/cpu/mtrr/mtrr.c | 9 + >> arch/x86/kernel/head_64.S | 71 ++++++++ >> arch/x86/kernel/smpboot.c | 324 ++++++++++++++++++++++++---------- >> arch/x86/realmode/init.c | 3 + >> arch/x86/realmode/rm/trampoline_64.S | 14 ++ >> arch/x86/xen/smp_pv.c | 4 +- >> include/linux/cpuhotplug.h | 2 + >> include/linux/smpboot.h | 7 + >> kernel/cpu.c | 27 ++- >> kernel/smpboot.c | 2 +- >> kernel/smpboot.h | 2 - >> 18 files changed, 441 insertions(+), 159 deletions(-) > > Thank you for working on this. I tested this on a MSI MS-7A37/B350M > MORTAR (BIOS 1.MW 11/01/2021) with a Ryzen 3 2200G, but nothing was > printed to the screen after the GRUB loading messages, so it crashed or > hung somewhere. Unfortunately, this device is used by others, and no > serial console is connected and I do not know how to capture the Linux > log with other means. Same on the ASUS F2A85-M PRO with AMD A6-6400K. Without serial console, the messages below are printed below to the monitor after nine seconds. [ 1.078879] smp: Bringing up secondary CPUs ... [ 1.080950] x86: Booting SMP configuration: Please find the serial log attached. Kind regards, Paul [ 0.000000] Linux version 5.16.0-rc7-00106-gcc498e0c43be (root@45e877da5b3e) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.37) #245 SMP PREEMPT Tue Dec 28 10:00:33 UTC 2021 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc7-00106-gcc498e0c43be root=/dev/sda3 rw debug noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 console=ttyS0,115200 console=tty1 earlyprintk=serial,ttyS0,115200,keep [ 0.000000] random: get_random_u32 called from bsp_init_amd+0x142/0x210 with crng_init=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005fe45fff] usable [ 0.000000] BIOS-e820: [mem 0x000000005fe46000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000017effffff] usable [ 0.000000] printk: console [earlyser0] enabled [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.15-676-g90cfb8f5ef 12/28/2021 [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Initial usec timer 20439600 [ 0.000000] tsc: Detected 3900.178 MHz processor [ 0.000588] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.007106] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.012655] last_pfn = 0x17f000 max_arch_pfn = 0x400000000 [ 0.018249] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT Memory KASLR using RDTSC... [ 0.027700] last_pfn = 0x5fe46 max_arch_pfn = 0x400000000 [ 0.036861] Using GB pages for direct mapping [ 0.041210] ACPI: Early table checksum verification disabled [ 0.046691] ACPI: RSDP 0x00000000000F6250 000024 (v02 COREv4) [ 0.052409] ACPI: XSDT 0x000000005FE4C0E0 000074 (v01 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.060905] ACPI: FACP 0x000000005FE4DBC0 000114 (v06 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.069398] ACPI: DSDT 0x000000005FE4C280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925) [ 0.077890] ACPI: FACS 0x000000005FE4C240 000040 [ 0.082483] ACPI: FACS 0x000000005FE4C240 000040 [ 0.087077] ACPI: SSDT 0x000000005FE4DCE0 00008A (v02 COREv4 COREBOOT 0000002A CORE 20200925) [ 0.095570] ACPI: MCFG 0x000000005FE4DD70 00003C (v01 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.104064] ACPI: APIC 0x000000005FE4DDB0 000062 (v03 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.112557] ACPI: HPET 0x000000005FE4DE20 000038 (v01 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.121051] ACPI: HEST 0x000000005FE4DE60 0001D0 (v01 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.129544] ACPI: IVRS 0x000000005FE4E030 000070 (v02 AMD AMDIOMMU 00000001 AMD 00000000) [ 0.138037] ACPI: SSDT 0x000000005FE4E0A0 00051F (v02 AMD ALIB 00000001 MSFT 04000000) [ 0.146531] ACPI: SSDT 0x000000005FE4E5C0 0006B2 (v01 AMD POWERNOW 00000001 AMD 00000001) [ 0.155025] ACPI: VFCT 0x000000005FE4EC80 00F269 (v01 COREv4 COREBOOT 00000000 CORE 20200925) [ 0.163517] ACPI: Reserving FACP table memory at [mem 0x5fe4dbc0-0x5fe4dcd3] [ 0.170537] ACPI: Reserving DSDT table memory at [mem 0x5fe4c280-0x5fe4dbb9] [ 0.177558] ACPI: Reserving FACS table memory at [mem 0x5fe4c240-0x5fe4c27f] [ 0.184578] ACPI: Reserving FACS table memory at [mem 0x5fe4c240-0x5fe4c27f] [ 0.191598] ACPI: Reserving SSDT table memory at [mem 0x5fe4dce0-0x5fe4dd69] [ 0.198619] ACPI: Reserving MCFG table memory at [mem 0x5fe4dd70-0x5fe4ddab] [ 0.205638] ACPI: Reserving APIC table memory at [mem 0x5fe4ddb0-0x5fe4de11] [ 0.212659] ACPI: Reserving HPET table memory at [mem 0x5fe4de20-0x5fe4de57] [ 0.219679] ACPI: Reserving HEST table memory at [mem 0x5fe4de60-0x5fe4e02f] [ 0.226699] ACPI: Reserving IVRS table memory at [mem 0x5fe4e030-0x5fe4e09f] [ 0.233719] ACPI: Reserving SSDT table memory at [mem 0x5fe4e0a0-0x5fe4e5be] [ 0.240740] ACPI: Reserving SSDT table memory at [mem 0x5fe4e5c0-0x5fe4ec71] [ 0.247760] ACPI: Reserving VFCT table memory at [mem 0x5fe4ec80-0x5fe5dee8] [ 0.254835] No NUMA configuration found [ 0.258593] Faking a node at [mem 0x0000000000000000-0x000000017effffff] [ 0.265273] NODE_DATA(0) allocated [mem 0x17efe7000-0x17effdfff] [ 0.283316] Zone ranges: [ 0.285678] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.291830] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.297984] Normal [mem 0x0000000100000000-0x000000017effffff] [ 0.304138] Device empty [ 0.306998] Movable zone start for each node [ 0.311245] Early memory node ranges [ 0.314798] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.321039] node 0: [mem 0x0000000000100000-0x000000005fe45fff] [ 0.327278] node 0: [mem 0x0000000100000000-0x000000017effffff] [ 0.333520] Initmem setup node 0 [mem 0x0000000000001000-0x000000017effffff] [ 0.340544] On node 0, zone DMA: 1 pages in unavailable ranges [ 0.340602] On node 0, zone DMA: 97 pages in unavailable ranges [ 0.359000] On node 0, zone Normal: 442 pages in unavailable ranges [ 0.364808] On node 0, zone Normal: 4096 pages in unavailable ranges [ 0.371106] ACPI: PM-Timer IO Port: 0x818 [ 0.381304] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.387198] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23 [ 0.394039] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.400367] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.406868] ACPI: Using ACPI (MADT) for SMP configuration information [ 0.413280] ACPI: HPET id: 0x10228210 base: 0xfed00000 [ 0.418398] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.423333] smpboot: smpboot: XXX end of prefill_possible_map [ 0.429053] After prefill_possible_map [ 0.432781] After init_cpu_to_node [ 0.436160] After init_gi_nodes [ 0.439281] After io_apic_init_mappings [ 0.443094] After x86_init.hyper.guest_late_init [ 0.447696] [mem 0x80000000-0xf7ffffff] available for PCI devices [ 0.453754] After e820 [ 0.456096] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.470502] After unwind_init [ 0.473298] After setup_arch [ 0.476169] After setup_command_line [ 0.479711] After setup_nr_cpu_ids [ 0.483091] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:2 nr_node_ids:1 [ 0.491127] percpu: Embedded 54 pages/cpu s182040 r8192 d30952 u1048576 [ 0.497579] pcpu-alloc: s182040 r8192 d30952 u1048576 alloc=1*2097152 [ 0.503978] pcpu-alloc: [0] 0 1 [ 0.507209] After setup_per_cpu_areas [ 0.510826] After smp_perpare_boot_cpu [ 0.514553] After boot_cpu_hotplug_init [ 0.518368] Fallback order for Node 0: 0 [ 0.522352] Built 1 zonelists, mobility grouping on. Total pages: 898444 [ 0.529113] Policy zone: Normal [ 0.532233] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc7-00106-gcc498e0c43be root=/dev/sda3 rw debug noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 console=ttyS0,115200 console=tty1 earlyprintk=serial,ttyS0,115200,keep [ 0.553561] Unknown kernel command line parameters "noisapnp BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc7-00106-gcc498e0c43be", will be passed to user space. [ 0.567513] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear) [ 0.575640] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 0.583229] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.629839] Memory: 3483292K/3651472K available (14344K kernel code, 2321K rwdata, 4212K rodata, 1692K init, 6332K bss, 167920K reserved, 0K cma-reserved) [ 0.643901] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.650282] After mm_init [ 0.652850] ftrace: allocating 35324 entries in 138 pages [ 0.670177] ftrace: allocated 138 pages with 3 groups [ 0.675169] Dynamic Preempt: full [ 0.678348] After sched_init [ 0.681282] rcu: Preemptible hierarchical RCU implementation. [ 0.686928] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2. [ 0.693515] Trampoline variant of Tasks RCU enabled. [ 0.698541] Rude variant of Tasks RCU enabled. [ 0.703048] Tracing variant of Tasks RCU enabled. [ 0.707815] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.715441] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 [ 0.722125] After rcu_init [ 0.734046] NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16 [ 0.740046] rcu: Offload RCU callbacks from CPUs: (none). [ 0.745386] random: crng_init_try_arch_early failed with i = 4, X86_FEATURE_RDRAND = no [ 0.745388] random: crng_init_try_arch_early failed with i = 5, X86_FEATURE_RDRAND = no [ 0.753324] random: crng_init_try_arch_early failed with i = 6, X86_FEATURE_RDRAND = no [ 0.761299] random: crng_init_try_arch_early failed with i = 7, X86_FEATURE_RDRAND = no [ 0.769272] random: crng_init_try_arch_early failed with i = 8, X86_FEATURE_RDRAND = no [ 0.777245] random: crng_init_try_arch_early failed with i = 9, X86_FEATURE_RDRAND = no [ 0.785218] random: crng_init_try_arch_early failed with i = 10, X86_FEATURE_RDRAND = no [ 0.793192] random: crng_init_try_arch_early failed with i = 11, X86_FEATURE_RDRAND = no [ 0.801252] random: crng_init_try_arch_early failed with i = 12, X86_FEATURE_RDRAND = no [ 0.809313] random: crng_init_try_arch_early failed with i = 13, X86_FEATURE_RDRAND = no [ 0.817372] random: crng_init_try_arch_early failed with i = 14, X86_FEATURE_RDRAND = no [ 0.825432] random: crng_init_try_arch_early failed with i = 15, X86_FEATURE_RDRAND = no [ 0.833494] After add_latent_entropy [ 0.845109] After add_device_randomness [ 0.848921] After boot_init_stack_canary [ 0.852875] spurious 8259A interrupt: IRQ7. [ 0.854860] Console: colour VGA+ 80x25 [ 0.866354] printk: console [tty1] enabled [ 0.870342] ACPI: Core revision 20210930 [ 0.874423] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns [ 0.883411] APIC: Switch to symmetric I/O mode setup [ 0.923446] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.933411] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x7070070e77e, max_idle_ns: 881591209168 ns [ 0.943779] Calibrating delay loop (skipped), value calculated using timer frequency.. 7800.35 BogoMIPS (lpj=3900178) [ 0.944776] pid_max: default: 32768 minimum: 301 [ 0.945884] LSM: Security Framework initializing [ 0.946890] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 0.947791] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) Poking KASLR using RDTSC... [ 0.952654] Bit 30 in CPUID ECX not set. [ 0.952681] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512 [ 0.953775] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512, 1GB 0 [ 0.954780] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.955776] Spectre V2 : Mitigation: Full AMD retpoline [ 0.956775] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.957776] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.958776] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl [ 0.963298] Freeing SMP alternatives memory: 40K [ 0.963777] After check_bugs [ 0.964776] After acpi_subsystem_init [ 0.965776] After arch_post_acpi_subsys_init [ 0.966776] After rcu_scheduler_starting [ 0.967851] After find_task_by_pid_ns and PF_NO_SETAFFINITY [ 0.968781] After numa_default_policy [ 0.969801] After rcu_read_lock [ 0.970775] After rcu_read_unlock [ 0.971776] After kthreadd_done [ 0.972786] smpboot: Start of smp_prepare_cpus_common [ 0.973777] smpboot: smpboot: zalloc 0 [ 0.974776] smpboot: smpboot: zalloc 1 [ 0.975775] smpboot: smpboot: After set_sched_topology() [ 0.976777] smpboot: smpboot: After smp_sanity_check() [ 0.977775] smpboot: smpboot: Before x86_init.timers.setup_percpu_clockev() [ 0.997775] random: random: 1 [ 0.998775] random: random: 2 [ 0.998775] random: random: 3 [ 0.998775] random: random: 4 [ 1.061775] random: random: 1 [ 1.062775] random: random: 2 [ 1.062775] random: random: 3 [ 1.062775] random: random: 4 [ 1.062808] APIC calibration not consistent with PM-Timer: 102ms instead of 100ms [ 1.063775] APIC delta adjusted to PM-Timer: 625036 (640760) [ 1.063780] smpboot: smpboot: After x86_init.timers.setup_percpu_clockev() [ 1.064775] smpboot: smp_get_logical_apicid() [ 1.065775] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1) [ 1.067103] Performance Events: Fam15h core perfctr, AMD PMU driver. [ 1.067777] ... version: 0 [ 1.068775] ... bit width: 48 [ 1.069775] ... generic registers: 6 [ 1.070777] ... value mask: 0000ffffffffffff [ 1.071775] ... max period: 00007fffffffffff [ 1.072775] ... fixed-purpose events: 0 [ 1.073775] ... event mask: 000000000000003f [ 1.075812] rcu: Hierarchical SRCU implementation. [ 1.078397] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter. [ 1.078879] smp: Bringing up secondary CPUs ... [ 1.080950] x86: Booting SMP configuration:
On Tue, 2021-12-28 at 12:34 +0100, Paul Menzel wrote: > Same on the ASUS F2A85-M PRO with AMD A6-6400K. Without serial console, > the messages below are printed below to the monitor after nine seconds. > > [ 1.078879] smp: Bringing up secondary CPUs ... > [ 1.080950] x86: Booting SMP configuration: > > Please find the serial log attached. > Thanks for testing. That looks like the same triple-fault on bringup that we have been seeing, and that I reproduced without my patches using kexec all the way back to a 5.0 kernel. Out of interest, are you also able to reproduce it with kexec and without the parallel bringup? And with that patch I sent Tom in https://lore.kernel.org/lkml/721484e0fa719e99f9b8f13e67de05033dd7cc86.camel@infradead.org/ to expand the bitlock exclusion and stop the bringup being truly in parallel at all? Or tbe one in https://lore.kernel.org/lkml/d4cde50b4aab24612823714dfcbe69bc4bb63b60.camel@infradead.org which makes it do nothing except prepare all the CPUs before bringing them up one at a time? My current theory (not that I've spent that much time thinking about it in the last week) is that there's something about the existing CPU bringup, possibly a CPU bug or something special about the AMD CPUs, which is triggered by just making it a little bit *faster*, which is why bringing them up from kexec (especially in qemu) can cause it too? Tom seemed to find that it was in load_TR_desc(), so if you could try this hack on a machine that doesn't magically wink out of existence on a triplefault before even flushing its serial output, that would be much appreciated... diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h index ab97b22ac04a..cc6590712ff4 100644 --- a/arch/x86/include/asm/desc.h +++ b/arch/x86/include/asm/desc.h @@ -8,7 +8,7 @@ #include <asm/fixmap.h> #include <asm/irq_vectors.h> #include <asm/cpu_entry_area.h> - +#include <asm/io.h> #include <linux/debug_locks.h> #include <linux/smp.h> #include <linux/percpu.h> @@ -265,11 +265,16 @@ static inline void native_load_tr_desc(void) * If the current GDT is the read-only fixmap, swap to the original * writeable version. Swap back at the end. */ + outb('d', 0x3f8); if (gdt.address == (unsigned long)fixmap_gdt) { + outb('e', 0x3f8); load_direct_gdt(cpu); restore = 1; + outb('f', 0x3f8); } + outb('g', 0x3f8); asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); + outb('h', 0x3f8); if (restore) load_fixmap_gdt(cpu); } diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 0083464de5e3..5bc8f30c3283 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1716,7 +1716,9 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) enable_sep_cpu(); #endif mtrr_ap_init(); +outb('A', 0x3f8); validate_apic_and_package_id(c); +outb('B', 0x3f8); x86_spec_ctrl_setup_ap(); update_srbds_msr(); } @@ -1957,6 +1959,7 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss) tss->io_bitmap.mapall[IO_BITMAP_LONGS] = ~0UL; #endif } +#include <asm/realmode.h> /* * Setup everything needed to handle exceptions from the IDT, including the IST @@ -1969,16 +1972,24 @@ void cpu_init_exception_handling(void) /* paranoid_entry() gets the CPU number from the GDT */ setup_getcpu(cpu); - + outb('\n', 0x3f8); + outb('0' + cpu / 100, 0x3f8); + outb('0' + (cpu % 100) / 10, 0x3f8); + outb('0' + (cpu % 10), 0x3f8); + /* IST vectors need TSS to be set up. */ tss_setup_ist(tss); + outb('a', 0x3f8); tss_setup_io_bitmap(tss); set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss); - + outb('b', 0x3f8); load_TR_desc(); + outb('c', 0x3f8); /* Finally load the IDT */ load_current_idt(); + outb('z', 0x3f8); + } /*
Dear David, Am 28.12.21 um 15:18 schrieb David Woodhouse: > On Tue, 2021-12-28 at 12:34 +0100, Paul Menzel wrote: >> Same on the ASUS F2A85-M PRO with AMD A6-6400K. Without serial console, >> the messages below are printed below to the monitor after nine seconds. >> >> [ 1.078879] smp: Bringing up secondary CPUs ... >> [ 1.080950] x86: Booting SMP configuration: >> >> Please find the serial log attached. > > Thanks for testing. That looks like the same triple-fault on bringup > that we have been seeing, and that I reproduced without my patches > using kexec all the way back to a 5.0 kernel. > > Out of interest, are you also able to reproduce it with kexec and > without the parallel bringup? No, I am not able to reproduce that with Debian’s *linux-image-5.15.0-2-686*, and kexec. With this board, `module_blacklist=radeon` is needed, as the driver *radeon* is not able to deal with kexec – and amdgpu neither [1]. ``` [ 3.349911] [drm] Found VCE firmware/feedback version 50.0.1 / 17! [ 3.365259] clocksource: Switched to clocksource tsc [ 3.365284] [drm] GART: num cpu pages 262144, num gpu pages 262144 [ 3.405159] random: fast init done [ 3.420492] [drm] PCIE GART of 1024M enabled (table at 0x00000000001D6000). [ 3.427634] radeon 0000:00:01.0: WB enabled [ 3.431828] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000020000c00 [ 3.440100] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr 0x0000000000075a18 [ 3.458182] radeon 0000:00:01.0: failed VCE resume (-22). [ 3.463591] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr 0x0000000020000c04 [ 3.471615] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr 0x0000000020000c08 [ 3.479636] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr 0x0000000020000c0c [ 3.487650] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr 0x0000000020000c10 [ 3.495990] radeon 0000:00:01.0: radeon: MSI limited to 32-bit [ 3.502008] radeon 0000:00:01.0: radeon: using MSI. [ 3.506906] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 3.506918] [drm] radeon: irq initialized. [ ``` > And with that patch I sent Tom in > https://lore.kernel.org/lkml/721484e0fa719e99f9b8f13e67de05033dd7cc86.camel@infradead.org/ > to expand the bitlock exclusion and stop the bringup being truly in > parallel at all? No, this does not help, and the Linux kernel resets at the same spot. ``` [ 1.036036] smpboot: smpboot: After x86_init.timers.setup_percpu_clockev() [ 1.037031] smpboot: smp_get_logical_apicid() [ 1.038031] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1) [ 1.039366] Performance Events: Fam15h core perfctr, AMD PMU driver. [ 1.040033] ... version: 0 [ 1.041031] ... bit width: 48 [ 1.042031] ... generic registers: 6 [ 1.043033] ... value mask: 0000ffffffffffff [ 1.044031] ... max period: 00007fffffffffff [ 1.045031] ... fixed-purpose events: 0 [ 1.046031] ... event mask: 000000000000003f [ 1.048065] rcu: Hierarchical SRCU implementation. [ 1.050642] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter. [ 1.051133] smp: Bringing up secondary CPUs ... [ 1.053202] x86: Booting SMP configuration: ``` > Or the one in > https://lore.kernel.org/lkml/d4cde50b4aab24612823714dfcbe69bc4bb63b60.camel@infradead.org > which makes it do nothing except prepare all the CPUs before bringing > them up one at a time? I applied it on top the other one, and it made no difference either. > My current theory (not that I've spent that much time thinking about it > in the last week) is that there's something about the existing CPU > bringup, possibly a CPU bug or something special about the AMD CPUs, > which is triggered by just making it a little bit *faster*, which is > why bringing them up from kexec (especially in qemu) can cause it too? Would having the serial console enabled make a difference? > Tom seemed to find that it was in load_TR_desc(), so if you could try > this hack on a machine that doesn't magically wink out of existence on > a triplefault before even flushing its serial output, that would be > much appreciated... > > diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h > index ab97b22ac04a..cc6590712ff4 100644 > --- a/arch/x86/include/asm/desc.h > +++ b/arch/x86/include/asm/desc.h > @@ -8,7 +8,7 @@ > #include <asm/fixmap.h> > #include <asm/irq_vectors.h> > #include <asm/cpu_entry_area.h> > - > +#include <asm/io.h> > #include <linux/debug_locks.h> > #include <linux/smp.h> > #include <linux/percpu.h> > @@ -265,11 +265,16 @@ static inline void native_load_tr_desc(void) > * If the current GDT is the read-only fixmap, swap to the original > * writeable version. Swap back at the end. > */ > + outb('d', 0x3f8); > if (gdt.address == (unsigned long)fixmap_gdt) { > + outb('e', 0x3f8); > load_direct_gdt(cpu); > restore = 1; > + outb('f', 0x3f8); > } > + outb('g', 0x3f8); > asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); > + outb('h', 0x3f8); > if (restore) > load_fixmap_gdt(cpu); > } > diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c > index 0083464de5e3..5bc8f30c3283 100644 > --- a/arch/x86/kernel/cpu/common.c > +++ b/arch/x86/kernel/cpu/common.c > @@ -1716,7 +1716,9 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) > enable_sep_cpu(); > #endif > mtrr_ap_init(); > +outb('A', 0x3f8); > validate_apic_and_package_id(c); > +outb('B', 0x3f8); > x86_spec_ctrl_setup_ap(); > update_srbds_msr(); > } > @@ -1957,6 +1959,7 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss) > tss->io_bitmap.mapall[IO_BITMAP_LONGS] = ~0UL; > #endif > } > +#include <asm/realmode.h> > > /* > * Setup everything needed to handle exceptions from the IDT, including the IST > @@ -1969,16 +1972,24 @@ void cpu_init_exception_handling(void) > > /* paranoid_entry() gets the CPU number from the GDT */ > setup_getcpu(cpu); > - > + outb('\n', 0x3f8); > + outb('0' + cpu / 100, 0x3f8); > + outb('0' + (cpu % 100) / 10, 0x3f8); > + outb('0' + (cpu % 10), 0x3f8); > + > /* IST vectors need TSS to be set up. */ > tss_setup_ist(tss); > + outb('a', 0x3f8); > tss_setup_io_bitmap(tss); > set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss); > - > + outb('b', 0x3f8); > load_TR_desc(); > + outb('c', 0x3f8); > > /* Finally load the IDT */ > load_current_idt(); > + outb('z', 0x3f8); > + > } > > /* Unfortunately, no more messages were printed on the serial console. Kind regards, Paul [1]: https://gitlab.freedesktop.org/drm/amd/-/issues/1597
On Wed, 2021-12-29 at 14:18 +0100, Paul Menzel wrote: > > Or the one in > > https://lore.kernel.org/lkml/d4cde50b4aab24612823714dfcbe69bc4bb63b60.camel@infradead.org > > > > which makes it do nothing except prepare all the CPUs before bringing > > them up one at a time? > > I applied it on top the other one, and it made no difference either. It's possible I missed something else in the prepare stage that doesn't cope with all CPUs being prepared first. My next attempt might be to change the loop in bringup_nonboot_cpus() to bring all the CPUs not to the CPUHP_BP_PARALLEL_DYN state(s) but instead just bring them to somewhere like CPUHP_RCUTREE_PREP, which is somewhere in the middle between CPUHP_OFFLINE and CPUHP_BRINGUP_CPU. Then a binary chop search — if that one boots, try maybe CPUHP_TOPOLOGY_PREPARE. And if not, try CPUHP_PROFILE_PREPARE. Etc. > > My current theory (not that I've spent that much time thinking about it > > in the last week) is that there's something about the existing CPU > > bringup, possibly a CPU bug or something special about the AMD CPUs, > > which is triggered by just making it a little bit *faster*, which is > > why bringing them up from kexec (especially in qemu) can cause it too? > > Would having the serial console enabled make a difference? > Yes. I couldn't make this fail in my EC2 m6a instance (for clean boots; I have never managed to kexec it) until I turned off the serial console to make things go faster. > > Tom seemed to find that it was in load_TR_desc(), so if you could try > > this hack on a machine that doesn't magically wink out of existence on > > a triplefault before even flushing its serial output, that would be > > much appreciated... > Unfortunately, no more messages were printed on the serial console. I suppose we need to litter those outputs somewhere earlier in the trampoline then, perhaps it *isn't* getting to load_TR_desc() in your case? Will be back online properly next week and can actually provide some of the above suggestions in patch form if you're willing to keep testing. Thanks!
On Fri, 2021-12-17 at 14:55 -0600, Tom Lendacky wrote: > On 12/17/21 2:13 PM, David Woodhouse wrote: > > On Fri, 2021-12-17 at 13:46 -0600, Tom Lendacky wrote: > > > There's no WARN or PANIC, just a reset. I can look to try and capture some > > > KVM trace data if that would help. If so, let me know what events you'd > > > like captured. > > > > > > Could start with just kvm_run_exit? > > > > Reason 8 would be KVM_EXIT_SHUTDOWN and would potentially indicate a > > triple fault. > > qemu-system-x86-24093 [005] ..... 1601.759486: kvm_exit: vcpu 112 reason shutdown rip 0xffffffff81070574 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x80000b08 error_code 0x00000000 > > # addr2line -e woodhouse-build-x86_64/vmlinux 0xffffffff81070574 > /root/kernels/woodhouse-build-x86_64/./arch/x86/include/asm/desc.h:272 > > Which is: asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); So, I remain utterly bemused by this, and the Milan *guests* I have access to can't even kexec with a stock kernel; that is also "too fast" and they take a triple fault during the bringup in much the same way — even without my parallel patches, and even going back to fairly old kernels. I wasn't able to follow up with raw serial output during the bringup to pinpoint precisely where it happens, because the VM would tear itself down in response to the triple fault without actually flushing the last virtual serial output :) It would be really useful to get access to a suitable host where I can spawn this in qemu and watch it fail. I am suspecting a chip-specific quirk or bug at this point. I might suggest in the short term that we could unblock the parallel bringup work by just not doing it for affected chips... but that won't make existing kexec work.
On Fri, Jan 28, 2022, David Woodhouse wrote: > On Fri, 2021-12-17 at 14:55 -0600, Tom Lendacky wrote: > > On 12/17/21 2:13 PM, David Woodhouse wrote: > > > On Fri, 2021-12-17 at 13:46 -0600, Tom Lendacky wrote: > > > > There's no WARN or PANIC, just a reset. I can look to try and capture some > > > > KVM trace data if that would help. If so, let me know what events you'd > > > > like captured. > > > > > > > > > Could start with just kvm_run_exit? > > > > > > Reason 8 would be KVM_EXIT_SHUTDOWN and would potentially indicate a > > > triple fault. > > > > qemu-system-x86-24093 [005] ..... 1601.759486: kvm_exit: vcpu 112 reason shutdown rip 0xffffffff81070574 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x80000b08 error_code 0x00000000 > > > > # addr2line -e woodhouse-build-x86_64/vmlinux 0xffffffff81070574 > > /root/kernels/woodhouse-build-x86_64/./arch/x86/include/asm/desc.h:272 > > > > Which is: asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); > > So, I remain utterly bemused by this, and the Milan *guests* I have > access to can't even kexec with a stock kernel; that is also "too fast" > and they take a triple fault during the bringup in much the same way — > even without my parallel patches, and even going back to fairly old > kernels. > > I wasn't able to follow up with raw serial output during the bringup to > pinpoint precisely where it happens, because the VM would tear itself > down in response to the triple fault without actually flushing the last > virtual serial output :) > > It would be really useful to get access to a suitable host where I can > spawn this in qemu and watch it fail. I am suspecting a chip-specific > quirk or bug at this point. Nope. You missed a spot. This also reproduces on a sufficiently large Intel system (and Milan). initial_gs gets overwritten by common_cpu_up(), which leads to a CPU getting the wrong MSR_GS_BASE and then the wrong raw_smp_processor_id(), resulting in cpu_init_exception_handling() stuffing the wrong GDT and leaving a NULL TR descriptor for itself. You also have a lurking bug in the x2APIC ID handling. Stripping the boot flags from the prescribed APICID needs to happen before retrieving the x2APIC ID from CPUID, otherwise bits 31:16 of the ID will be lost. You owe me two beers ;-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index dcdf49a137d6..23df88c86a0e 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -208,11 +208,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * in smpboot_control: * Bit 0-15 APICID if STARTUP_USE_CPUID_0B is not set * Bit 16 Secondary boot flag - * Bit 17 Parallel boot flag + * Bit 17 Parallel boot flag (STARTUP_USE_CPUID_0B) */ testl $STARTUP_USE_CPUID_0B, %eax - jz .Lsetup_AP + jnz .Luse_cpuid_0b + andl $0xFFFF, %eax + jmp .Lsetup_AP +.Luse_cpuid_0b: mov $0x0B, %eax xorl %ecx, %ecx cpuid @@ -220,7 +223,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) .Lsetup_AP: /* EAX contains the APICID of the current CPU */ - andl $0xFFFF, %eax xorl %ecx, %ecx leaq cpuid_to_apicid(%rip), %rbx diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 04f5c8de5606..e7fda406f39a 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1093,6 +1093,17 @@ wakeup_cpu_via_init_nmi(int cpu, unsigned long start_ip, int apicid, return boot_error; } +static bool do_parallel_bringup = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + + int common_cpu_up(unsigned int cpu, struct task_struct *idle) { int ret; @@ -1112,7 +1123,8 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) /* Stack for startup_32 can be just as for start_secondary onwards */ per_cpu(cpu_current_top_of_stack, cpu) = task_top_of_stack(idle); #else - initial_gs = per_cpu_offset(cpu); + if (!do_parallel_bringup) + initial_gs = per_cpu_offset(cpu); #endif return 0; } @@ -1336,16 +1348,6 @@ int do_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } -static bool do_parallel_bringup = true; - -static int __init no_parallel_bringup(char *str) -{ - do_parallel_bringup = false; - - return 0; -} -early_param("no_parallel_bringup", no_parallel_bringup); - int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret;
On 28 January 2022 21:40:42 GMT, Sean Christopherson <seanjc@google.com> wrote: >On Fri, Jan 28, 2022, David Woodhouse wrote: >> On Fri, 2021-12-17 at 14:55 -0600, Tom Lendacky wrote: >> > On 12/17/21 2:13 PM, David Woodhouse wrote: >> > > On Fri, 2021-12-17 at 13:46 -0600, Tom Lendacky wrote: >> > > > There's no WARN or PANIC, just a reset. I can look to try and capture some >> > > > KVM trace data if that would help. If so, let me know what events you'd >> > > > like captured. >> > > >> > > >> > > Could start with just kvm_run_exit? >> > > >> > > Reason 8 would be KVM_EXIT_SHUTDOWN and would potentially indicate a >> > > triple fault. >> > >> > qemu-system-x86-24093 [005] ..... 1601.759486: kvm_exit: vcpu 112 reason shutdown rip 0xffffffff81070574 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x80000b08 error_code 0x00000000 >> > >> > # addr2line -e woodhouse-build-x86_64/vmlinux 0xffffffff81070574 >> > /root/kernels/woodhouse-build-x86_64/./arch/x86/include/asm/desc.h:272 >> > >> > Which is: asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8)); >> >> So, I remain utterly bemused by this, and the Milan *guests* I have >> access to can't even kexec with a stock kernel; that is also "too fast" >> and they take a triple fault during the bringup in much the same way — >> even without my parallel patches, and even going back to fairly old >> kernels. >> >> I wasn't able to follow up with raw serial output during the bringup to >> pinpoint precisely where it happens, because the VM would tear itself >> down in response to the triple fault without actually flushing the last >> virtual serial output :) >> >> It would be really useful to get access to a suitable host where I can >> spawn this in qemu and watch it fail. I am suspecting a chip-specific >> quirk or bug at this point. > >Nope. You missed a spot. This also reproduces on a sufficiently large Intel >system (and Milan). initial_gs gets overwritten by common_cpu_up(), which leads >to a CPU getting the wrong MSR_GS_BASE and then the wrong raw_smp_processor_id(), >resulting in cpu_init_exception_handling() stuffing the wrong GDT and leaving a >NULL TR descriptor for itself. > >You also have a lurking bug in the x2APIC ID handling. Stripping the boot flags >from the prescribed APICID needs to happen before retrieving the x2APIC ID from >CPUID, otherwise bits 31:16 of the ID will be lost. > >You owe me two beers ;-) Oh Sean, I love you. Thanks. Will update and retest and resend.
On Fri, 2022-01-28 at 21:40 +0000, Sean Christopherson wrote: > Nope. You missed a spot. This also reproduces on a sufficiently large Intel > system (and Milan). initial_gs gets overwritten by common_cpu_up(), which leads > to a CPU getting the wrong MSR_GS_BASE and then the wrong raw_smp_processor_id(), > resulting in cpu_init_exception_handling() stuffing the wrong GDT and leaving a > NULL TR descriptor for itself. > > You also have a lurking bug in the x2APIC ID handling. Stripping the boot flags > from the prescribed APICID needs to happen before retrieving the x2APIC ID from > CPUID, otherwise bits 31:16 of the ID will be lost. > > You owe me two beers ;-) > > diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S > index dcdf49a137d6..23df88c86a0e 100644 > --- a/arch/x86/kernel/head_64.S > +++ b/arch/x86/kernel/head_64.S > @@ -208,11 +208,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) > * in smpboot_control: > * Bit 0-15 APICID if STARTUP_USE_CPUID_0B is not set > * Bit 16 Secondary boot flag > - * Bit 17 Parallel boot flag > + * Bit 17 Parallel boot flag (STARTUP_USE_CPUID_0B) > */ > testl $STARTUP_USE_CPUID_0B, %eax > - jz .Lsetup_AP > + jnz .Luse_cpuid_0b > + andl $0xFFFF, %eax > + jmp .Lsetup_AP > > +.Luse_cpuid_0b: > mov $0x0B, %eax > xorl %ecx, %ecx > cpuid Looks like I had already fixed that one in a cleanup at https://git.infradead.org/users/dwmw2/linux.git/commitdiff/191f08997577 I removed the mask entirely. We now use the APIC ID from the low 31 bits if bit 31 isn't set... and there's no need to mask it out because by definition it isn't set. + /* + * Secondary CPUs find out the offsets via the APIC ID. For parallel + * boot the APIC ID is retrieved from CPUID, otherwise it's encoded + * in smpboot_control: + * Bit 0-30 APIC ID if STARTUP_PARALLEL is not set + * Bit 31 Parallel boot flag (use CPUID leaf 0x0b for APIC ID). + */ + testl $STARTUP_PARALLEL, %eax + jz .Lsetup_AP + + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + mov %edx, %eax + +.Lsetup_AP: I am, of course, still prepared to buy you as many beers as you desire. Perhaps in Dublin in September, where we're (hopefully) going to be doing Linux Plumbers Conference in person again at last! (I actually think I'm going to rework that cleanup because it's given us a hard-coded assumption that no AP has APIC ID 0. I'll put back the explicit STARTUP_SECONDARY flag that Thomas had, and work your fix in too to avoid re-introducing the bug.) > int common_cpu_up(unsigned int cpu, struct task_struct *idle) > { > int ret; > @@ -1112,7 +1123,8 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) > /* Stack for startup_32 can be just as for start_secondary onwards */ > per_cpu(cpu_current_top_of_stack, cpu) = task_top_of_stack(idle); > #else > - initial_gs = per_cpu_offset(cpu); > + if (!do_parallel_bringup) > + initial_gs = per_cpu_offset(cpu); > #endif > return 0; > } Hm, I think that can be removed completely, can't it? We don't need to make it conditional, because even the non-parallel 64-bit bringup will still take the same path in head_64.S to *find* the stack and other per-CPU information; it just gets its APIC ID from the global variable in order to do so.
Dear David, Am 29.12.21 um 14:54 schrieb David Woodhouse: > On Wed, 2021-12-29 at 14:18 +0100, Paul Menzel wrote: >>> Or the one in >>> https://lore.kernel.org/lkml/d4cde50b4aab24612823714dfcbe69bc4bb63b60.camel@infradead.org >>> >>> which makes it do nothing except prepare all the CPUs before bringing >>> them up one at a time? >> >> I applied it on top the other one, and it made no difference either. > > It's possible I missed something else in the prepare stage that doesn't > cope with all CPUs being prepared first. > > My next attempt might be to change the loop in bringup_nonboot_cpus() > to bring all the CPUs not to the CPUHP_BP_PARALLEL_DYN state(s) but > instead just bring them to somewhere like CPUHP_RCUTREE_PREP, which is > somewhere in the middle between CPUHP_OFFLINE and CPUHP_BRINGUP_CPU. > > Then a binary chop search — if that one boots, try maybe > CPUHP_TOPOLOGY_PREPARE. And if not, try CPUHP_PROFILE_PREPARE. Etc. > >>> My current theory (not that I've spent that much time thinking about it >>> in the last week) is that there's something about the existing CPU >>> bringup, possibly a CPU bug or something special about the AMD CPUs, >>> which is triggered by just making it a little bit *faster*, which is >>> why bringing them up from kexec (especially in qemu) can cause it too? >> >> Would having the serial console enabled make a difference? > > Yes. I couldn't make this fail in my EC2 m6a instance (for clean boots; > I have never managed to kexec it) until I turned off the serial console > to make things go faster. > >>> Tom seemed to find that it was in load_TR_desc(), so if you could try >>> this hack on a machine that doesn't magically wink out of existence on >>> a triplefault before even flushing its serial output, that would be >>> much appreciated... > >> Unfortunately, no more messages were printed on the serial console. > > I suppose we need to litter those outputs somewhere earlier in the > trampoline then, perhaps it *isn't* getting to load_TR_desc() in your > case? > > Will be back online properly next week and can actually provide some of > the above suggestions in patch form if you're willing to keep testing. Sorry for replying so late. I saw your v4 patches, and tried commit 5e3524d21d2a () from your branch `parallel-5.17-part1`. Unfortunately, the boot problem still persists on an AMD Ryzen 3 2200 g system, I tested with. Please tell, where I should report these results too (here or posted v4 patches). Also, do you have (physical) access to a system with an AMD CPU? If not, maybe we can get you one, so it’s more convenient for you to test. Kind regards, Paul
Dear Paul, > Sorry for replying so late. I saw your v4 patches, and tried commit > 5e3524d21d2a () from your branch `parallel-5.17-part1`. Unfortunately, > the boot problem still persists on an AMD Ryzen 3 2200 g system, I > tested with. Please tell, where I should report these results too > (here or posted v4 patches). We have confirmed the issue on multiple AMD CPUs from multiple generations, leading to the guess that only Zen and Zen+ CPU seem affected with Zen3 and Zen2 (only tested ulv) working fine. Tho we struggled to get any output as the failing machines just go silent. Not working: Ryzen 5 Pro 2500u and 7 2700U Ryzen 3 2300G while e.g. Ryzen 7 Pro 4750U Ryzen 9 5950X both work fine. We will continue to investigate the issue but are currently a bit pulled into other topics. Thomas, could please maybe help us identify which CPUs and MC-Versions are worth looking at? David suggested you might have a good overview here. Best regards Johanna "Mimoja"
On 4/21/22 05:00, Mimoja wrote: > Dear Paul, > >> Sorry for replying so late. I saw your v4 patches, and tried commit >> 5e3524d21d2a () from your branch `parallel-5.17-part1`. Unfortunately, >> the boot problem still persists on an AMD Ryzen 3 2200 g system, I >> tested with. Please tell, where I should report these results too (here >> or posted v4 patches). > > We have confirmed the issue on multiple AMD CPUs from multiple > generations, leading to the guess that only Zen and Zen+ CPU seem affected > with Zen3 and Zen2 (only tested ulv) working fine. Tho we struggled to get > any output as the failing machines just go silent. > > Not working: > > Ryzen 5 Pro 2500u and 7 2700U > Ryzen 3 2300G > > while e.g. > > Ryzen 7 Pro 4750U > Ryzen 9 5950X > > both work fine. We will continue to investigate the issue but are > currently a bit pulled into other topics. > > Thomas, could please maybe help us identify which CPUs and MC-Versions are > worth looking at? David suggested you might have a good overview here. Sorry, but not knowing what the actual reason for the boot problem, I really couldn't give you an idea as to which CPUs and/or MC versions are appropriate to look at. Thanks, Tom > > > Best regards > > Johanna "Mimoja" >
On Fri, 2022-04-22 at 16:19 -0500, Tom Lendacky wrote: > On 4/21/22 05:00, Mimoja wrote: > > Dear Paul, > > > > > Sorry for replying so late. I saw your v4 patches, and tried commit > > > 5e3524d21d2a () from your branch `parallel-5.17-part1`. Unfortunately, > > > the boot problem still persists on an AMD Ryzen 3 2200 g system, I > > > tested with. Please tell, where I should report these results too (here > > > or posted v4 patches). > > > > We have confirmed the issue on multiple AMD CPUs from multiple > > generations, leading to the guess that only Zen and Zen+ CPU seem affected > > with Zen3 and Zen2 (only tested ulv) working fine. Tho we struggled to get > > any output as the failing machines just go silent. > > > > Not working: > > > > Ryzen 5 Pro 2500u and 7 2700U > > Ryzen 3 2300G > > > > while e.g. > > > > Ryzen 7 Pro 4750U > > Ryzen 9 5950X > > > > both work fine. We will continue to investigate the issue but are > > currently a bit pulled into other topics. > > > > Thomas, could please maybe help us identify which CPUs and MC-Versions are > > worth looking at? David suggested you might have a good overview here. > > Sorry, but not knowing what the actual reason for the boot problem, I > really couldn't give you an idea as to which CPUs and/or MC versions are > appropriate to look at. Well, that's kind of the point... we don't *know* what the problem is. We think we've eliminated the software concurrency issues, and it seems like the hardware just dies if you happen to bring up the CPUs 'too fast'. If we could get you to reproduce it in a lab and help work out what's going on, it would be much appreciated!
Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for them shaves about 80% off the AP bringup time on a 96-thread socket Skylake box (EC2 c5.metal) — from about 500ms to 100ms. There are more wins to be had with further parallelisation, but this is the simple part. v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more v3: Clean up x2apic patch, add MTRR optimisation, lock topology update in preparation for more parallelisation. David Woodhouse (8): x86/apic/x2apic: Fix parallel handling of cluster_mask cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() x86/smpboot: Split up native_cpu_up into separate phases and document them x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup x86/smpboot: Serialize topology updates for secondary bringup Thomas Gleixner (1): x86/smpboot: Support parallel startup of secondary CPUs arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/smp.h | 13 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/acpi/sleep.c | 1 + arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++++----- arch/x86/kernel/cpu/common.c | 6 +- arch/x86/kernel/cpu/mtrr/mtrr.c | 9 + arch/x86/kernel/head_64.S | 71 ++++++++ arch/x86/kernel/smpboot.c | 324 ++++++++++++++++++++++++---------- arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 14 ++ arch/x86/xen/smp_pv.c | 4 +- include/linux/cpuhotplug.h | 2 + include/linux/smpboot.h | 7 + kernel/cpu.c | 27 ++- kernel/smpboot.c | 2 +- kernel/smpboot.h | 2 - 18 files changed, 441 insertions(+), 159 deletions(-)