| Message ID | 20191126214443.99189-1-aaronlewis@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [kvm-unit-tests,v2] x86: Add RDTSC test | expand |
> On 26 Nov 2019, at 23:44, Aaron Lewis <aaronlewis@google.com> wrote: > > Verify that the difference between a guest RDTSC instruction and the > IA32_TIME_STAMP_COUNTER MSR value stored in the VMCS12's VM-exit > MSR-store list is less than 750 cycles, 99.9% of the time. It will help if commit message would also reference the KVM commit which fixes the issue tested by this test. i.e. 662f1d1d1931 ("KVM: nVMX: Add support for capturing highest observable L2 TSC”) > > Signed-off-by: Aaron Lewis <aaronlewis@google.com> > Reviewed-by: Jim Mattson <jmattson@google.com> > --- > x86/unittests.cfg | 6 ++++ > x86/vmx.h | 1 + > x86/vmx_tests.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 98 insertions(+) > > diff --git a/x86/unittests.cfg b/x86/unittests.cfg > index b4865ac..5291d96 100644 > --- a/x86/unittests.cfg > +++ b/x86/unittests.cfg > @@ -284,6 +284,12 @@ extra_params = -cpu host,+vmx -append vmx_vmcs_shadow_test > arch = x86_64 > groups = vmx > > +[vmx_rdtsc_vmexit_diff_test] > +file = vmx.flat > +extra_params = -cpu host,+vmx -append rdtsc_vmexit_diff_test > +arch = x86_64 > +groups = vmx > + I think we are missing some clear guidance on when a VMX unit-test should have it’s own test-section in x86/unittests.cfg. I believe the guidance should be that all VMX tests are suppose to be run by [vmx] except those that have special requirements on execution environment (e.g. vmx_smp*) or destroy execution environment after they run (e.g. vmx_init_signal_test) or require special timeout if they fail (e.g. vmx_apic_passthrough_tpr_threshold_test). These tests should both be removed from [vmx] by "-append -test_name” and have their own section which runs them. Being concrete to this patch, I think it shouldn’t have it’s own section. For example, it will cause the test to run twice: Both as part of [vmx] and as part of [vmx_rdtsc_vmexit_diff_test]. And I can submit a separate patches to: 1) Rename vmx_eoi_bitmap_ioapic_scan & vmx_apic_passthrough_thread to prefix with vmx_smp* (It actually seems to me that currently there are no vmx_smp* tests at all…) and create a [vmx_smp] section for running them. 2) Remove vmx_hlt_with_rvi_test and vmx_apicv_test sections. Does anyone think differently? > [debug] > file = debug.flat > arch = x86_64 > diff --git a/x86/vmx.h b/x86/vmx.h > index 8496be7..21ba953 100644 > --- a/x86/vmx.h > +++ b/x86/vmx.h > @@ -420,6 +420,7 @@ enum Ctrl1 { > CPU_SHADOW_VMCS = 1ul << 14, > CPU_RDSEED = 1ul << 16, > CPU_PML = 1ul << 17, > + CPU_USE_TSC_SCALING = 1ul << 25, > }; > > enum Intr_type { > diff --git a/x86/vmx_tests.c b/x86/vmx_tests.c > index 1d8932f..fcf71e7 100644 > --- a/x86/vmx_tests.c > +++ b/x86/vmx_tests.c > @@ -8790,7 +8790,97 @@ static void vmx_vmcs_shadow_test(void) > enter_guest(); > } > > +/* > + * This test monitors the difference between a guest RDTSC instruction > + * and the IA32_TIME_STAMP_COUNTER MSR value stored in the VMCS12 > + * VM-exit MSR-store list when taking a VM-exit on the instruction > + * following RDTSC. > + */ > +#define RDTSC_DIFF_ITERS 100000 > +#define RDTSC_DIFF_FAILS 100 > +#define HOST_RDTSC_LIMIT 750 Nit: I suggest to rename HOST_RDTSC_LIMIT to HOST_CAPTURED_GUEST_TSC_DIFF_THRESHOLD. > + > +/* > + * Set 'use TSC offsetting' and set the guest offset to the > + * inverse of the host's current TSC value, so that the guest starts running > + * with an effective TSC value of 0. > + */ > +static void reset_guest_tsc_to_zero(void) > +{ > + TEST_ASSERT_MSG(ctrl_cpu_rev[0].clr & CPU_USE_TSC_OFFSET, > + "Expected support for 'use TSC offsetting'"); > + > + vmcs_set_bits(CPU_EXEC_CTRL0, CPU_USE_TSC_OFFSET); > + vmcs_write(TSC_OFFSET, -rdtsc()); > +} > + > +static void rdtsc_vmexit_diff_test_guest(void) > +{ > + int i; > + > + for (i = 0; i < RDTSC_DIFF_ITERS; i++) > + /* Ensure rdtsc is the last instruction before the vmcall. */ > + asm volatile("rdtsc; vmcall" : : : "eax", "edx"); > +} > > +/* > + * This function only considers the "use TSC offsetting" VM-execution > + * control. It does not handle "use TSC scaling" (because the latter > + * isn't available to the host today.) > + */ > +static unsigned long long host_time_to_guest_time(unsigned long long t) > +{ > + TEST_ASSERT((vmcs_read(CPU_EXEC_CTRL1) & CPU_USE_TSC_SCALING) == 0); It’s problematic to vmcs_read(CPU_EXEC_CTRL1) when test runs on CPU that doesn’t support secondary VM-execution control. As this will cause VMfailInvalid (i.e. Clear CF,PF,AF,SF,OF and set ZF). What’s worse is that vmcs_read() today doesn’t assert that RFLAGS.ZF==0 after executing VMREAD. Maybe we should submit a separate patch for that as-well… Anyway, you can just change your assert condition to: TEST_ASSERT(!(ctrl_cpu_rev[0].clr & CPU_SECONDARY) || !(vmcs_read(CPU_EXEC_CTRL1) & CPU_USE_TSC_SCALING)); > + > + if (vmcs_read(CPU_EXEC_CTRL0) & CPU_USE_TSC_OFFSET) > + t += vmcs_read(TSC_OFFSET); > + > + return t; > +} > + > +static unsigned long long rdtsc_vmexit_diff_test_iteration(void) > +{ > + unsigned long long guest_tsc, host_to_guest_tsc; > + > + enter_guest(); > + skip_exit_vmcall(); > + guest_tsc = (u32) regs.rax + (regs.rdx << 32); > + host_to_guest_tsc = host_time_to_guest_time(exit_msr_store[0].value); > + > + return host_to_guest_tsc - guest_tsc; > +} > + > +static void rdtsc_vmexit_diff_test(void) > +{ > + int fail = 0; > + int i; > + > + test_set_guest(rdtsc_vmexit_diff_test_guest); > + > + reset_guest_tsc_to_zero(); > + > + /* > + * Set up the VMCS12 VM-exit MSR-store list to store just one > + * MSR: IA32_TIME_STAMP_COUNTER. Note that the value stored is > + * in the host time domain (i.e., it is not adjusted according > + * to the TSC multiplier and TSC offset fields in the VMCS12, > + * as a guest RDTSC would be.) > + */ > + exit_msr_store = alloc_page(); > + exit_msr_store[0].index = MSR_IA32_TSC; > + vmcs_write(EXI_MSR_ST_CNT, 1); > + vmcs_write(EXIT_MSR_ST_ADDR, virt_to_phys(exit_msr_store)); > + > + for (i = 0; i < RDTSC_DIFF_ITERS; i++) { > + if (rdtsc_vmexit_diff_test_iteration() >= HOST_RDTSC_LIMIT) > + fail++; > + } > + > + enter_guest(); > + > + report("RDTSC to VM-exit delta too high in %d of %d iterations", > + fail < RDTSC_DIFF_FAILS, fail, RDTSC_DIFF_ITERS); > +} > > static int invalid_msr_init(struct vmcs *vmcs) > { > @@ -9056,5 +9146,6 @@ struct vmx_test vmx_tests[] = { > /* Atomic MSR switch tests. */ > TEST(atomic_switch_max_msrs_test), > TEST(atomic_switch_overflow_msrs_test), > + TEST(rdtsc_vmexit_diff_test), > { NULL, NULL, NULL, NULL, NULL, {0} }, > }; > -- > 2.24.0.432.g9d3f5f5b63-goog >
diff --git a/x86/unittests.cfg b/x86/unittests.cfg index b4865ac..5291d96 100644 --- a/x86/unittests.cfg +++ b/x86/unittests.cfg @@ -284,6 +284,12 @@ extra_params = -cpu host,+vmx -append vmx_vmcs_shadow_test arch = x86_64 groups = vmx +[vmx_rdtsc_vmexit_diff_test] +file = vmx.flat +extra_params = -cpu host,+vmx -append rdtsc_vmexit_diff_test +arch = x86_64 +groups = vmx + [debug] file = debug.flat arch = x86_64 diff --git a/x86/vmx.h b/x86/vmx.h index 8496be7..21ba953 100644 --- a/x86/vmx.h +++ b/x86/vmx.h @@ -420,6 +420,7 @@ enum Ctrl1 { CPU_SHADOW_VMCS = 1ul << 14, CPU_RDSEED = 1ul << 16, CPU_PML = 1ul << 17, + CPU_USE_TSC_SCALING = 1ul << 25, }; enum Intr_type { diff --git a/x86/vmx_tests.c b/x86/vmx_tests.c index 1d8932f..fcf71e7 100644 --- a/x86/vmx_tests.c +++ b/x86/vmx_tests.c @@ -8790,7 +8790,97 @@ static void vmx_vmcs_shadow_test(void) enter_guest(); } +/* + * This test monitors the difference between a guest RDTSC instruction + * and the IA32_TIME_STAMP_COUNTER MSR value stored in the VMCS12 + * VM-exit MSR-store list when taking a VM-exit on the instruction + * following RDTSC. + */ +#define RDTSC_DIFF_ITERS 100000 +#define RDTSC_DIFF_FAILS 100 +#define HOST_RDTSC_LIMIT 750 + +/* + * Set 'use TSC offsetting' and set the guest offset to the + * inverse of the host's current TSC value, so that the guest starts running + * with an effective TSC value of 0. + */ +static void reset_guest_tsc_to_zero(void) +{ + TEST_ASSERT_MSG(ctrl_cpu_rev[0].clr & CPU_USE_TSC_OFFSET, + "Expected support for 'use TSC offsetting'"); + + vmcs_set_bits(CPU_EXEC_CTRL0, CPU_USE_TSC_OFFSET); + vmcs_write(TSC_OFFSET, -rdtsc()); +} + +static void rdtsc_vmexit_diff_test_guest(void) +{ + int i; + + for (i = 0; i < RDTSC_DIFF_ITERS; i++) + /* Ensure rdtsc is the last instruction before the vmcall. */ + asm volatile("rdtsc; vmcall" : : : "eax", "edx"); +} +/* + * This function only considers the "use TSC offsetting" VM-execution + * control. It does not handle "use TSC scaling" (because the latter + * isn't available to the host today.) + */ +static unsigned long long host_time_to_guest_time(unsigned long long t) +{ + TEST_ASSERT((vmcs_read(CPU_EXEC_CTRL1) & CPU_USE_TSC_SCALING) == 0); + + if (vmcs_read(CPU_EXEC_CTRL0) & CPU_USE_TSC_OFFSET) + t += vmcs_read(TSC_OFFSET); + + return t; +} + +static unsigned long long rdtsc_vmexit_diff_test_iteration(void) +{ + unsigned long long guest_tsc, host_to_guest_tsc; + + enter_guest(); + skip_exit_vmcall(); + guest_tsc = (u32) regs.rax + (regs.rdx << 32); + host_to_guest_tsc = host_time_to_guest_time(exit_msr_store[0].value); + + return host_to_guest_tsc - guest_tsc; +} + +static void rdtsc_vmexit_diff_test(void) +{ + int fail = 0; + int i; + + test_set_guest(rdtsc_vmexit_diff_test_guest); + + reset_guest_tsc_to_zero(); + + /* + * Set up the VMCS12 VM-exit MSR-store list to store just one + * MSR: IA32_TIME_STAMP_COUNTER. Note that the value stored is + * in the host time domain (i.e., it is not adjusted according + * to the TSC multiplier and TSC offset fields in the VMCS12, + * as a guest RDTSC would be.) + */ + exit_msr_store = alloc_page(); + exit_msr_store[0].index = MSR_IA32_TSC; + vmcs_write(EXI_MSR_ST_CNT, 1); + vmcs_write(EXIT_MSR_ST_ADDR, virt_to_phys(exit_msr_store)); + + for (i = 0; i < RDTSC_DIFF_ITERS; i++) { + if (rdtsc_vmexit_diff_test_iteration() >= HOST_RDTSC_LIMIT) + fail++; + } + + enter_guest(); + + report("RDTSC to VM-exit delta too high in %d of %d iterations", + fail < RDTSC_DIFF_FAILS, fail, RDTSC_DIFF_ITERS); +} static int invalid_msr_init(struct vmcs *vmcs) { @@ -9056,5 +9146,6 @@ struct vmx_test vmx_tests[] = { /* Atomic MSR switch tests. */ TEST(atomic_switch_max_msrs_test), TEST(atomic_switch_overflow_msrs_test), + TEST(rdtsc_vmexit_diff_test), { NULL, NULL, NULL, NULL, NULL, {0} }, };