[RFC,00/26] Confidential guest live migration

Message

Dov Murik March 2, 2021, 8:47 p.m. UTC

This is an RFC series for fast migration of confidential guests using an
in-guest migration helper that lives in OVMF.  QEMU VM live migration
needs to read source VM's RAM and write it in the target VM; this
mechanism doesn't work when the guest memory is encrypted or QEMU is
prevented from reading it in another way.  In order to support live
migration in such scenarios, we introduce an in-guest migration helper
which can securely extract RAM content from the guest in order to send
it to the target.  The migration helper is implemented as part of the
VM's firmware in OVMF.

We've implemented and tested this on AMD SEV, but expect most of the
processes can be used with other technologies that prevent direct access
of hypervisor to the guest's memory.  Specifically, we don't use SEV's
PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all.

Corresponding RFC patches for OVMF have been posted by my colleague
Tobin Feldman-Fitzthum on edk2-devel.  Those include the crux of the
migration helper: a mailbox protocol over a shared memory page which
allows communication between QEMU and the migration helper.  In the
source VM this is used to read a page and encrypt it for transport; in
the target it is used to decrypt the incoming page and storing the
content in the correct address in the guest memory.  All encryption and
decryption operations occur inside the trusted context in the VM, and
therefore the VM's memory plaintext content is never accessible to the
hosts participating in the migration.

In order to allow OVMF to run the migration helper in parallel to the
guest OS, we introduce the notion of auxiliary vcpus, which are usable
for OVMF but are hidden from the guest OS.  These might have other
future uses for in-guest operations/agents.

In the target VM we need the migration handler running to receive
incoming RAM pages; to achieve that, we boot the VM into OVMF with a
special fw_cfg value that causes OVMF to not boot the guest OS; we then
allow QEMU to receive an incoming migration by issuing a new
start-migrate-incoming QMP command.

The confidential RAM migration requires checking whether a given guest
RAM page is encrypted or not.  This is currently achieved using AMD's
patches which track the encryption status of guest pages in KVM, using
hypercalls from OVMF and guest Linux to report changes of such status.
The QEMU side of these patches is included as the first two patches in
this series.  The concrete implementation of this page encryption tracking
is currently in flux in the KVM mailing list, but the underlying
implementation doesn't affect our confidential RAM migration as long as
it can check whether a given guest address is encrypted.

List of patches in this series:

1-2: reposting AMD encrypted page bitmap support.
3-11: introduce the notion of auxiliary vcpus.
12-21: introduce the migration specifics.
22-23: fix devices issues when loading state into a live VM
24: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
25: remove SEV migration blocker
26: add documentation


Brijesh Singh (1):
  kvm: add support to sync the page encryption state bitmap

Dov Murik (21):
  linux-headers: Add definitions of KVM page encryption bitmap ioctls
  machine: Add auxcpus=N suboption to -smp
  hw/boards: Add aux flag to CPUArchId
  hw/i386: Mark auxiliary vcpus in possible_cpus
  cpu: Add boolean aux field to CPUState
  hw/i386: Set CPUState.aux=true for auxiliary vcpus
  softmmu: Don't sync aux vcpus in pre_loadvm
  softmmu: Add cpu_synchronize_without_aux_post_init
  migration: Add helpers to save confidential RAM
  migration: Add helpers to load confidential RAM
  migration: Introduce gpa_inside_migration_helper_shared_area
  migration: Save confidential guest RAM using migration helper
  migration: Load confidential guest RAM using migration helper
  migration: Stop VM after loading confidential RAM
  migration: Don't sync vcpus when migrating confidential guests
  migration: When starting target, don't sync auxiliary vcpus
  hw/isa/lpc_ich9: Allow updating an already-running VM
  target/i386: Re-sync kvm-clock after confidential guest migration
  migration: Add start-migrate-incoming QMP command
  target/i386: SEV: Allow migration unless there are no aux vcpus
  docs: Add confidential guest live migration documentation

Tobin Feldman-Fitzthum (4):
  hw/acpi: Don't include auxiliary vcpus in ACPI tables
  softmmu: Add pause_all_vcpus_except_aux
  migration: Stop non-aux vcpus before copying the last pages
  migration: Call migration handler cleanup routines

 docs/confidential-guest-live-migration.rst | 142 ++++++++++++
 docs/confidential-guest-support.txt        |   5 +
 docs/index.rst                             |   1 +
 qapi/migration.json                        |  26 +++
 include/exec/ram_addr.h                    | 197 ++++++++++++++++
 include/exec/ramblock.h                    |   3 +
 include/exec/ramlist.h                     |   3 +-
 include/hw/boards.h                        |   3 +
 include/hw/core/cpu.h                      |   2 +
 include/hw/i386/x86.h                      |   2 +-
 include/sysemu/cpus.h                      |   2 +
 linux-headers/linux/kvm.h                  |  13 ++
 migration/confidential-ram.h               |  23 ++
 accel/kvm/kvm-all.c                        |  43 ++++
 hw/acpi/cpu.c                              |  10 +
 hw/core/cpu.c                              |   1 +
 hw/core/machine.c                          |   7 +
 hw/i386/acpi-build.c                       |   5 +
 hw/i386/acpi-common.c                      |   5 +
 hw/i386/pc.c                               |   7 +
 hw/i386/x86.c                              |  10 +-
 hw/isa/lpc_ich9.c                          |   3 +-
 migration/confidential-ram.c               | 258 +++++++++++++++++++++
 migration/migration.c                      |  18 +-
 migration/ram.c                            | 135 ++++++++++-
 migration/savevm.c                         |  13 +-
 softmmu/cpus.c                             |  68 +++++-
 softmmu/runstate.c                         |   1 +
 softmmu/vl.c                               |   3 +
 target/i386/machine.c                      |   9 +
 target/i386/sev.c                          |  25 +-
 migration/meson.build                      |   6 +-
 migration/trace-events                     |   4 +
 33 files changed, 1027 insertions(+), 26 deletions(-)
 create mode 100644 docs/confidential-guest-live-migration.rst
 create mode 100644 migration/confidential-ram.h
 create mode 100644 migration/confidential-ram.c


base-commit: 00d8ba9e0d62ea1c7459c25aeabf9c8bb7659462

Comments

[e-mail] March 2, 2021, 9:24 p.m. UTC | #1

Patchew URL: https://patchew.org/QEMU/[e-mail]/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: [e-mail]
Subject: [RFC PATCH 00/26] Confidential guest live migration

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
Switched to a new branch 'test'
f283a35 docs: Add confidential guest live migration documentation
08f3c3b target/i386: SEV: Allow migration unless there are no aux vcpus
99d1efb migration: Add start-migrate-incoming QMP command
6ca0894 target/i386: Re-sync kvm-clock after confidential guest migration
47e02a0 hw/isa/lpc_ich9: Allow updating an already-running VM
af25d7b migration: Call migration handler cleanup routines
48db1fb migration: When starting target, don't sync auxiliary vcpus
5eb9336 migration: Don't sync vcpus when migrating confidential guests
af9b2fa migration: Stop non-aux vcpus before copying the last pages
2789368 migration: Stop VM after loading confidential RAM
4692252 migration: Load confidential guest RAM using migration helper
e041f58 migration: Save confidential guest RAM using migration helper
328a888 migration: Introduce gpa_inside_migration_helper_shared_area
b71be10 migration: Add helpers to load confidential RAM
32bdb2e migration: Add helpers to save confidential RAM
8d2fccb softmmu: Add pause_all_vcpus_except_aux
ed10484 softmmu: Add cpu_synchronize_without_aux_post_init
e68df04 softmmu: Don't sync aux vcpus in pre_loadvm
17bfe19 hw/i386: Set CPUState.aux=true for auxiliary vcpus
808eb76 cpu: Add boolean aux field to CPUState
34d94b2 hw/acpi: Don't include auxiliary vcpus in ACPI tables
2e9bc24 hw/i386: Mark auxiliary vcpus in possible_cpus
3d07b10 hw/boards: Add aux flag to CPUArchId
0e07c01 machine: Add auxcpus=N suboption to -smp
49124f3 kvm: add support to sync the page encryption state bitmap
47f202b linux-headers: Add definitions of KVM page encryption bitmap ioctls

=== OUTPUT BEGIN ===
1/26 Checking commit 47f202b0bc8c (linux-headers: Add definitions of KVM page encryption bitmap ioctls)
2/26 Checking commit 49124f3bf03f (kvm: add support to sync the page encryption state bitmap)
ERROR: use qemu_real_host_page_size instead of getpagesize()
#51: FILE: accel/kvm/kvm-all.c:615:
+    ram_addr_t pages = int128_get64(section->size) / getpagesize();

ERROR: use qemu_real_host_page_size instead of getpagesize()
#171: FILE: include/exec/ram_addr.h:398:
+    unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;

WARNING: line over 80 characters
#197: FILE: include/exec/ram_addr.h:424:
+                qatomic_xchg(&blocks[DIRTY_MEMORY_ENCRYPTED][idx][offset], temp);

total: 2 errors, 1 warnings, 340 lines checked

Patch 2/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

3/26 Checking commit 0e07c0168e00 (machine: Add auxcpus=N suboption to -smp)
4/26 Checking commit 3d07b103b36f (hw/boards: Add aux flag to CPUArchId)
5/26 Checking commit 2e9bc24b9064 (hw/i386: Mark auxiliary vcpus in possible_cpus)
6/26 Checking commit 34d94b2baa07 (hw/acpi: Don't include auxiliary vcpus in ACPI tables)
7/26 Checking commit 808eb7693543 (cpu: Add boolean aux field to CPUState)
8/26 Checking commit 17bfe1902081 (hw/i386: Set CPUState.aux=true for auxiliary vcpus)
WARNING: line over 80 characters
#26: FILE: hw/i386/x86.c:104:
+void x86_cpu_new(X86MachineState *x86ms, int64_t apic_id, bool aux, Error **errp)

WARNING: line over 80 characters
#58: FILE: include/hw/i386/x86.h:88:
+void x86_cpu_new(X86MachineState *pcms, int64_t apic_id, bool aux, Error **errp);

total: 0 errors, 2 warnings, 34 lines checked

Patch 8/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
9/26 Checking commit e68df046c12e (softmmu: Don't sync aux vcpus in pre_loadvm)
10/26 Checking commit ed104842e25d (softmmu: Add cpu_synchronize_without_aux_post_init)
11/26 Checking commit 8d2fccbd9cf9 (softmmu: Add pause_all_vcpus_except_aux)
WARNING: Block comments use a leading /* on a separate line
#88: FILE: softmmu/cpus.c:614:
+    /* We need to drop the replay_lock so any vCPU threads woken up

total: 0 errors, 1 warnings, 78 lines checked

Patch 11/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
12/26 Checking commit 32bdb2e1c795 (migration: Add helpers to save confidential RAM)
Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#28: 
new file mode 100644

total: 0 errors, 1 warnings, 216 lines checked

Patch 12/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
13/26 Checking commit b71be10a582a (migration: Add helpers to load confidential RAM)
14/26 Checking commit 328a888fe8fd (migration: Introduce gpa_inside_migration_helper_shared_area)
15/26 Checking commit e041f580a1da (migration: Save confidential guest RAM using migration helper)
16/26 Checking commit 4692252e1994 (migration: Load confidential guest RAM using migration helper)
WARNING: line over 80 characters
#59: FILE: migration/ram.c:3990:
+                error_report("%s: failed to get gpa for host %p", __func__, host);

total: 0 errors, 1 warnings, 43 lines checked

Patch 16/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
17/26 Checking commit 278936890c8d (migration: Stop VM after loading confidential RAM)
WARNING: line over 80 characters
#44: FILE: migration/confidential-ram.c:235:
+static EndOfConfidentialRAMState end_of_confidential_ram_state = { .dummy = false };

total: 0 errors, 1 warnings, 67 lines checked

Patch 17/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
18/26 Checking commit af9b2fa4e956 (migration: Stop non-aux vcpus before copying the last pages)
19/26 Checking commit 5eb933666eda (migration: Don't sync vcpus when migrating confidential guests)
20/26 Checking commit 48db1fbc2c51 (migration: When starting target, don't sync auxiliary vcpus)
21/26 Checking commit af25d7bee8ac (migration: Call migration handler cleanup routines)
22/26 Checking commit 47e02a0e2441 (hw/isa/lpc_ich9: Allow updating an already-running VM)
23/26 Checking commit 6ca089420af2 (target/i386: Re-sync kvm-clock after confidential guest migration)
24/26 Checking commit 99d1efb7f73c (migration: Add start-migrate-incoming QMP command)
25/26 Checking commit 08f3c3b68478 (target/i386: SEV: Allow migration unless there are no aux vcpus)
26/26 Checking commit f283a35951da (docs: Add confidential guest live migration documentation)
Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#20: 
new file mode 100644

total: 0 errors, 1 warnings, 154 lines checked

Patch 26/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/[e-mail]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [e-mail]

Dov Murik March 3, 2021, 8:08 a.m. UTC | #2

The errors reported below on patch 02/26 are due to rebasing an older patch
from AMD.  I didn't want to make any changes to the code (except make it compile
and run correctly) because this feature (encrypted pages bitmap) is still
work-in-progress (in KVM and QEMU).

-Dov

On 02/03/2021 23:24, [e-mail] wrote:
> Patchew URL: https://patchew.org/QEMU/[e-mail]/
> 
> 
> 
> Hi,
> 
> This series seems to have some coding style problems. See output below for
> more information:
> 
> Type: series
> Message-id: [e-mail]
> Subject: [RFC PATCH 00/26] Confidential guest live migration
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> git rev-parse base > /dev/null || exit 0
> git config --local diff.renamelimit 0
> git config --local diff.renames True
> git config --local diff.algorithm histogram
> ./scripts/checkpatch.pl --mailback base..
> === TEST SCRIPT END ===
> 
> Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> From https://github.com/patchew-project/qemu
>  - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
>  * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
> Switched to a new branch 'test'
> f283a35 docs: Add confidential guest live migration documentation
> 08f3c3b target/i386: SEV: Allow migration unless there are no aux vcpus
> 99d1efb migration: Add start-migrate-incoming QMP command
> 6ca0894 target/i386: Re-sync kvm-clock after confidential guest migration
> 47e02a0 hw/isa/lpc_ich9: Allow updating an already-running VM
> af25d7b migration: Call migration handler cleanup routines
> 48db1fb migration: When starting target, don't sync auxiliary vcpus
> 5eb9336 migration: Don't sync vcpus when migrating confidential guests
> af9b2fa migration: Stop non-aux vcpus before copying the last pages
> 2789368 migration: Stop VM after loading confidential RAM
> 4692252 migration: Load confidential guest RAM using migration helper
> e041f58 migration: Save confidential guest RAM using migration helper
> 328a888 migration: Introduce gpa_inside_migration_helper_shared_area
> b71be10 migration: Add helpers to load confidential RAM
> 32bdb2e migration: Add helpers to save confidential RAM
> 8d2fccb softmmu: Add pause_all_vcpus_except_aux
> ed10484 softmmu: Add cpu_synchronize_without_aux_post_init
> e68df04 softmmu: Don't sync aux vcpus in pre_loadvm
> 17bfe19 hw/i386: Set CPUState.aux=true for auxiliary vcpus
> 808eb76 cpu: Add boolean aux field to CPUState
> 34d94b2 hw/acpi: Don't include auxiliary vcpus in ACPI tables
> 2e9bc24 hw/i386: Mark auxiliary vcpus in possible_cpus
> 3d07b10 hw/boards: Add aux flag to CPUArchId
> 0e07c01 machine: Add auxcpus=N suboption to -smp
> 49124f3 kvm: add support to sync the page encryption state bitmap
> 47f202b linux-headers: Add definitions of KVM page encryption bitmap ioctls
> 
> === OUTPUT BEGIN ===
> 1/26 Checking commit 47f202b0bc8c (linux-headers: Add definitions of KVM page encryption bitmap ioctls)
> 2/26 Checking commit 49124f3bf03f (kvm: add support to sync the page encryption state bitmap)
> ERROR: use qemu_real_host_page_size instead of getpagesize()
> #51: FILE: accel/kvm/kvm-all.c:615:
> +    ram_addr_t pages = int128_get64(section->size) / getpagesize();
> 
> ERROR: use qemu_real_host_page_size instead of getpagesize()
> #171: FILE: include/exec/ram_addr.h:398:
> +    unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
> 
> WARNING: line over 80 characters
> #197: FILE: include/exec/ram_addr.h:424:
> +                qatomic_xchg(&blocks[DIRTY_MEMORY_ENCRYPTED][idx][offset], temp);
> 
> total: 2 errors, 1 warnings, 340 lines checked
> 
> Patch 2/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 
> 3/26 Checking commit 0e07c0168e00 (machine: Add auxcpus=N suboption to -smp)
> 4/26 Checking commit 3d07b103b36f (hw/boards: Add aux flag to CPUArchId)
> 5/26 Checking commit 2e9bc24b9064 (hw/i386: Mark auxiliary vcpus in possible_cpus)
> 6/26 Checking commit 34d94b2baa07 (hw/acpi: Don't include auxiliary vcpus in ACPI tables)
> 7/26 Checking commit 808eb7693543 (cpu: Add boolean aux field to CPUState)
> 8/26 Checking commit 17bfe1902081 (hw/i386: Set CPUState.aux=true for auxiliary vcpus)
> WARNING: line over 80 characters
> #26: FILE: hw/i386/x86.c:104:
> +void x86_cpu_new(X86MachineState *x86ms, int64_t apic_id, bool aux, Error **errp)
> 
> WARNING: line over 80 characters
> #58: FILE: include/hw/i386/x86.h:88:
> +void x86_cpu_new(X86MachineState *pcms, int64_t apic_id, bool aux, Error **errp);
> 
> total: 0 errors, 2 warnings, 34 lines checked
> 
> Patch 8/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 9/26 Checking commit e68df046c12e (softmmu: Don't sync aux vcpus in pre_loadvm)
> 10/26 Checking commit ed104842e25d (softmmu: Add cpu_synchronize_without_aux_post_init)
> 11/26 Checking commit 8d2fccbd9cf9 (softmmu: Add pause_all_vcpus_except_aux)
> WARNING: Block comments use a leading /* on a separate line
> #88: FILE: softmmu/cpus.c:614:
> +    /* We need to drop the replay_lock so any vCPU threads woken up
> 
> total: 0 errors, 1 warnings, 78 lines checked
> 
> Patch 11/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 12/26 Checking commit 32bdb2e1c795 (migration: Add helpers to save confidential RAM)
> Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #28: 
> new file mode 100644
> 
> total: 0 errors, 1 warnings, 216 lines checked
> 
> Patch 12/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 13/26 Checking commit b71be10a582a (migration: Add helpers to load confidential RAM)
> 14/26 Checking commit 328a888fe8fd (migration: Introduce gpa_inside_migration_helper_shared_area)
> 15/26 Checking commit e041f580a1da (migration: Save confidential guest RAM using migration helper)
> 16/26 Checking commit 4692252e1994 (migration: Load confidential guest RAM using migration helper)
> WARNING: line over 80 characters
> #59: FILE: migration/ram.c:3990:
> +                error_report("%s: failed to get gpa for host %p", __func__, host);
> 
> total: 0 errors, 1 warnings, 43 lines checked
> 
> Patch 16/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 17/26 Checking commit 278936890c8d (migration: Stop VM after loading confidential RAM)
> WARNING: line over 80 characters
> #44: FILE: migration/confidential-ram.c:235:
> +static EndOfConfidentialRAMState end_of_confidential_ram_state = { .dummy = false };
> 
> total: 0 errors, 1 warnings, 67 lines checked
> 
> Patch 17/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 18/26 Checking commit af9b2fa4e956 (migration: Stop non-aux vcpus before copying the last pages)
> 19/26 Checking commit 5eb933666eda (migration: Don't sync vcpus when migrating confidential guests)
> 20/26 Checking commit 48db1fbc2c51 (migration: When starting target, don't sync auxiliary vcpus)
> 21/26 Checking commit af25d7bee8ac (migration: Call migration handler cleanup routines)
> 22/26 Checking commit 47e02a0e2441 (hw/isa/lpc_ich9: Allow updating an already-running VM)
> 23/26 Checking commit 6ca089420af2 (target/i386: Re-sync kvm-clock after confidential guest migration)
> 24/26 Checking commit 99d1efb7f73c (migration: Add start-migrate-incoming QMP command)
> 25/26 Checking commit 08f3c3b68478 (target/i386: SEV: Allow migration unless there are no aux vcpus)
> 26/26 Checking commit f283a35951da (docs: Add confidential guest live migration documentation)
> Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #20: 
> new file mode 100644
> 
> total: 0 errors, 1 warnings, 154 lines checked
> 
> Patch 26/26 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> === OUTPUT END ===
> 
> Test command exited with code: 1
> 
> 
> The full log is available at
> http://patchew.org/logs/[e-mail]/testing.checkpatch/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to [e-mail]
>

Paolo Bonzini March 4, 2021, 9:10 a.m. UTC | #3

On 02/03/21 21:47, Dov Murik wrote:
> In order to allow OVMF to run the migration helper in parallel to the
> guest OS, we introduce the notion of auxiliary vcpus, which are usable
> for OVMF but are hidden from the guest OS.  These might have other
> future uses for in-guest operations/agents.

Hi Dov,

I think the helper approach to migration in general is great, but I'm 
not sure I agree with the concept of auxiliary vCPUs.  I would rather 
have a completely separate VM file descriptor that does not even go 
through the regular KVM run loop.  Patches were posted recently to the 
KVM mailing list to create secondary VMs sharing the encryption context 
(ASID) with a primary VM.

When starting the VM, the firmware would proceed with attestation as 
usual, detect it was running as a migration helper VM during the SEC 
phase, and divert execution to the migration helper instead of 
continuing with PEI.

The main advantage would be that the migration VM would not have to 
share the address space with the primary VM.  This would allow migrating 
encrypted RAM areas that are not visible to the primary VM, for example 
PCI BARs (those areas would be a problem for the kernel migration bitmap 
though; I'll remark on that separately on Ashish's KVM series).

The VM would not even have an interrupt controller, so that HLT exits to 
the host when it's done processing the mailbox.  This would make it much 
simpler to audit both the QEMU and the firmware sides.

Paolo

> In the target VM we need the migration handler running to receive
> incoming RAM pages; to achieve that, we boot the VM into OVMF with a
> special fw_cfg value that causes OVMF to not boot the guest OS; we then
> allow QEMU to receive an incoming migration by issuing a new
> start-migrate-incoming QMP command.
> 
> The confidential RAM migration requires checking whether a given guest
> RAM page is encrypted or not.  This is currently achieved using AMD's
> patches which track the encryption status of guest pages in KVM, using
> hypercalls from OVMF and guest Linux to report changes of such status.
> The QEMU side of these patches is included as the first two patches in
> this series.  The concrete implementation of this page encryption tracking
> is currently in flux in the KVM mailing list, but the underlying
> implementation doesn't affect our confidential RAM migration as long as
> it can check whether a given guest address is encrypted.
> 
> List of patches in this series:
> 
> 1-2: reposting AMD encrypted page bitmap support.
> 3-11: introduce the notion of auxiliary vcpus.
> 12-21: introduce the migration specifics.
> 22-23: fix devices issues when loading state into a live VM
> 24: introduce the start-migrate-incoming QMP command to switch the
> target into accepting the incoming migration.
> 25: remove SEV migration blocker
> 26: add documentation
> 
> 
> Brijesh Singh (1):
>    kvm: add support to sync the page encryption state bitmap
> 
> Dov Murik (21):
>    linux-headers: Add definitions of KVM page encryption bitmap ioctls
>    machine: Add auxcpus=N suboption to -smp
>    hw/boards: Add aux flag to CPUArchId
>    hw/i386: Mark auxiliary vcpus in possible_cpus
>    cpu: Add boolean aux field to CPUState
>    hw/i386: Set CPUState.aux=true for auxiliary vcpus
>    softmmu: Don't sync aux vcpus in pre_loadvm
>    softmmu: Add cpu_synchronize_without_aux_post_init
>    migration: Add helpers to save confidential RAM
>    migration: Add helpers to load confidential RAM
>    migration: Introduce gpa_inside_migration_helper_shared_area
>    migration: Save confidential guest RAM using migration helper
>    migration: Load confidential guest RAM using migration helper
>    migration: Stop VM after loading confidential RAM
>    migration: Don't sync vcpus when migrating confidential guests
>    migration: When starting target, don't sync auxiliary vcpus
>    hw/isa/lpc_ich9: Allow updating an already-running VM
>    target/i386: Re-sync kvm-clock after confidential guest migration
>    migration: Add start-migrate-incoming QMP command
>    target/i386: SEV: Allow migration unless there are no aux vcpus
>    docs: Add confidential guest live migration documentation
> 
> Tobin Feldman-Fitzthum (4):
>    hw/acpi: Don't include auxiliary vcpus in ACPI tables
>    softmmu: Add pause_all_vcpus_except_aux
>    migration: Stop non-aux vcpus before copying the last pages
>    migration: Call migration handler cleanup routines
> 
>   docs/confidential-guest-live-migration.rst | 142 ++++++++++++
>   docs/confidential-guest-support.txt        |   5 +
>   docs/index.rst                             |   1 +
>   qapi/migration.json                        |  26 +++
>   include/exec/ram_addr.h                    | 197 ++++++++++++++++
>   include/exec/ramblock.h                    |   3 +
>   include/exec/ramlist.h                     |   3 +-
>   include/hw/boards.h                        |   3 +
>   include/hw/core/cpu.h                      |   2 +
>   include/hw/i386/x86.h                      |   2 +-
>   include/sysemu/cpus.h                      |   2 +
>   linux-headers/linux/kvm.h                  |  13 ++
>   migration/confidential-ram.h               |  23 ++
>   accel/kvm/kvm-all.c                        |  43 ++++
>   hw/acpi/cpu.c                              |  10 +
>   hw/core/cpu.c                              |   1 +
>   hw/core/machine.c                          |   7 +
>   hw/i386/acpi-build.c                       |   5 +
>   hw/i386/acpi-common.c                      |   5 +
>   hw/i386/pc.c                               |   7 +
>   hw/i386/x86.c                              |  10 +-
>   hw/isa/lpc_ich9.c                          |   3 +-
>   migration/confidential-ram.c               | 258 +++++++++++++++++++++
>   migration/migration.c                      |  18 +-
>   migration/ram.c                            | 135 ++++++++++-
>   migration/savevm.c                         |  13 +-
>   softmmu/cpus.c                             |  68 +++++-
>   softmmu/runstate.c                         |   1 +
>   softmmu/vl.c                               |   3 +
>   target/i386/machine.c                      |   9 +
>   target/i386/sev.c                          |  25 +-
>   migration/meson.build                      |   6 +-
>   migration/trace-events                     |   4 +
>   33 files changed, 1027 insertions(+), 26 deletions(-)
>   create mode 100644 docs/confidential-guest-live-migration.rst
>   create mode 100644 migration/confidential-ram.h
>   create mode 100644 migration/confidential-ram.c
> 
> 
> base-commit: 00d8ba9e0d62ea1c7459c25aeabf9c8bb7659462
>