[v6,00/11] x86: Expose consistent topology to guests

Message

Alejandro Vallejo Oct. 1, 2024, 12:37 p.m. UTC

(original cover letter at the bottom)

The series wasn't taking proper precautions to not break PVH. In particular,
because new APIC IDs break with the old convention, PVH is not something that
can be left for later as it otherwise suffers a mismatch between APIC IDs in
the vLAPICs and the MADT table.

This version is a rebased v5 with the additional fixes of:
  1. PVH should now work, because libacpi was refactored to stop taking a
     function pointer and start taking a LUT for the cpu->apic_id mapping.
  2. Expose leaf 0xb to guests even on hosts that don't themselves do so.
     (e.g: AMD Lisbon). Otherwise all such hosts are unable to create
     guests with this patch series on, and there's no good reason not to do so.

Hypervisor prerequisites:

  patch 1: lib/x86: Relax checks about policy compatibility
    * new patch to properly operate (after this series) on older AMD hardware. 
  patch 2: x86/vlapic: Move lapic migration checks to the check hooks
    * Same as in v5
  patch 3: xen/x86: Add initial x2APIC ID to the per-vLAPIC save area
    * Same as in v5.
  patch 4: xen/x86: Add supporting code for uploading LAPIC contexts during
           domain create
    * Same as in v5.

hvmloader prerequisites

  patch 5: tools/hvmloader: Retrieve (x2)APIC IDs from the APs themselves
    * Same as in v5.

Toolstack prerequisites:

  patch 6: tools/libacpi: Use LUT of APIC IDs rather than function pointer
    * new patch to enable proper topology reporting to PVH guests.
  patch 7: tools/libguest: Always set vCPU context in vcpu_hvm()
    * Same as in v5.

No functional changes:

  patch 8: xen/lib: Add topology generator for x86
    * Same as in v5.
  patch 9: xen/x86: Derive topologically correct x2APIC IDs from the policy
    * Same as in v5.

Final toolstack/xen stitching:

  patch 10: tools/libguest: Set distinct x2APIC IDs for each vCPU
    * Unlikely in v5, this patch takes the APIC IDs from a LUT stored in
      the xc_dom_image structure.
  patch 11: xen/x86: Synthesise domain topologies
    * Same as v5.

====================================================================

v5: https://lore.kernel.org/xen-devel/[e-mail]/
v4 -> v5:

Largely unchanged and resent for review after the 4.20 dev cycle started.

  * Addressed Andrew's nits in v4/patch1
  * Addressed Jan's concern with MTRR overrides in v4/patch6 by keeping the
    same MTRR data in the vCPU contexts for HVM domain creations.

v4: https://lore.kernel.org/xen-devel/[e-mail]/
v3 -> v4:

  * Fixed cpuid() bug in hvmloader, causing UB in v3
  * Fixed a bogus assert in hvmloader, also causing a crash in v3
  * Used HVM contexts rather than sync'd algos between Xen and toolstack in
    order to initialise the per-vCPU LAPIC state.
  * Formatting asjustments.

v3: https://lore.kernel.org/xen-devel/[e-mail]/
v2 -> v3:

  (v2/patch2 and v2/patch4 are already committed)

  * Moved the vlapic check hook addition to v3/patch1
    * And created a check hook for the architectural state too for consistency.
  * Fixed migrations from Xen <= 4.13 by reconstructing the previous topology.
  * Correctly set the APIC ID after a policy change when vlapic is already in
    x2APIC mode.
  * Removed bogus assumption introduced in v1 and v2 on hvmloader about which
    8bit APIC IDs represent ids > 254. (it's "id % 0xff", not "min(id, 0xff)".
        * Used an x2apic flag check instead.
  * Various formatting adjustments.

v2: https://lore.kernel.org/xen-devel/[e-mail]/
v1 -> v2:

  * v1/patch 4 replaced by a different strategy (See patches 4 and 5 in v2):
      * Have hvmloader populate MADT with the real APIC IDs as read by the APs
        themselves rather than giving it knowledge on how to derive them.
  * Removed patches 2 and 3 in v1, as no longer relevant.
  * Split v1/patch6 in two parts ((a) creating the generator and (b) plugging it
    in) and use the generator in the unit tests of the vcpuid->apicid mapping
    function. Becomes patches 6 and 8 in v2.

  Patch 1: Same as v1/patch1.
  Patch 2: Header dependency cleanup in preparation for patch3.
  Patch 3: Adds vlapic_hidden check for the newly introduced reserved area.
  Patch 4: [hvmloader] Replaces INIT+SIPI+SIPI sequences with hypercalls.
  Patch 5: [hvmloader] Retrieve the per-CPU APIC IDs from the APs themselves.
  Patch 6: Split from v1/patch6.
  Patch 7: Logically matching v1/patch5, but using v2/patch6 for testing.
  Patch 8: Split from v1/patch6.

v1: https://lore.kernel.org/xen-devel/[e-mail]/
=== Original cover letter ===

Current topology handling is close to non-existent. As things stand, APIC
IDs are allocated through the apic_id=vcpu_id*2 relation without giving any
hints to the OS on how to parse the x2APIC ID of a given CPU and assuming
the guest will assume 2 threads per core.

This series involves bringing x2APIC IDs into the migration stream, so
older guests keep operating as they used to and enhancing Xen+toolstack so
new guests get topology information consistent with their x2APIC IDs. As a
side effect of this, x2APIC IDs are now packed and don't have (unless under
a pathological case) gaps.

Further work ought to allow combining this topology configurations with
gang-scheduling of guest hyperthreads into affine physical hyperthreads.
For the time being it purposefully keeps the configuration of "1 socket" +
"1 thread per core" + "1 core per vCPU".

Patch 1: Includes x2APIC IDs in the migration stream. This allows Xen to
         reconstruct the right x2APIC IDs on migrated-in guests, and
         future-proofs itself in the face of x2APIC ID derivation changes.
Patch 2: Minor refactor to expose xc_cpu_policy in libxl
Patch 3: Refactors xen/lib/x86 to work on non-Xen freestanding environments
         (e.g: hvmloader)
Patch 4: Remove old assumptions about vcpu_id<->apic_id relationship in hvmloader
Patch 5: Add logic to derive x2APIC IDs given a CPU policy and vCPU IDs
Patch 6: Includes a simple topology generator for toolstack so new guests
         have topologically consistent information in CPUID

===================================================================
v6:

Alejandro Vallejo (11):
  lib/x86: Relax checks about policy compatibility
  x86/vlapic: Move lapic migration checks to the check hooks
  xen/x86: Add initial x2APIC ID to the per-vLAPIC save area
  xen/x86: Add supporting code for uploading LAPIC contexts during
    domain create
  tools/hvmloader: Retrieve (x2)APIC IDs from the APs themselves
  tools/libacpi: Use LUT of APIC IDs rather than function pointer
  tools/libguest: Always set vCPU context in vcpu_hvm()
  xen/lib: Add topology generator for x86
  xen/x86: Derive topologically correct x2APIC IDs from the policy
  tools/libguest: Set distinct x2APIC IDs for each vCPU
  xen/x86: Synthesise domain topologies

 tools/firmware/hvmloader/config.h        |   5 +-
 tools/firmware/hvmloader/hvmloader.c     |   6 +-
 tools/firmware/hvmloader/mp_tables.c     |   4 +-
 tools/firmware/hvmloader/smp.c           |  54 ++++--
 tools/firmware/hvmloader/util.c          |   7 +-
 tools/include/xen-tools/common-macros.h  |   5 +
 tools/include/xenguest.h                 |   8 +
 tools/libacpi/build.c                    |   6 +-
 tools/libacpi/libacpi.h                  |   2 +-
 tools/libs/guest/xg_cpuid_x86.c          |  29 +++-
 tools/libs/guest/xg_dom_x86.c            |  93 ++++++----
 tools/libs/light/libxl_dom.c             |  25 +++
 tools/libs/light/libxl_x86_acpi.c        |   7 +-
 tools/tests/cpu-policy/test-cpu-policy.c | 207 ++++++++++++++++++++++-
 xen/arch/x86/cpu-policy.c                |   9 +-
 xen/arch/x86/cpuid.c                     |  14 +-
 xen/arch/x86/hvm/vlapic.c                | 126 ++++++++++----
 xen/arch/x86/include/asm/hvm/vlapic.h    |   1 +
 xen/include/public/arch-x86/hvm/save.h   |   2 +
 xen/include/xen/lib/x86/cpu-policy.h     |  27 +++
 xen/lib/x86/policy.c                     | 175 ++++++++++++++++++-
 xen/lib/x86/private.h                    |   4 +
 22 files changed, 704 insertions(+), 112 deletions(-)