mbox

[PULL,00/17] Migration 20230209 patches

Message ID 20230209233426.37811-1-quintela@redhat.com (mailing list archive)
State New, archived
Headers show

Pull-request

https://gitlab.com/juan.quintela/qemu.git tags/migration-20230209-pull-request

Message

Juan Quintela Feb. 9, 2023, 11:34 p.m. UTC
The following changes since commit 417296c8d8588f782018d01a317f88957e9786d6:

  tests/qtest/netdev-socket: Raise connection timeout to 60 seconds (2023-02-09 11:23:53 +0000)

are available in the Git repository at:

  https://gitlab.com/juan.quintela/qemu.git tags/migration-20230209-pull-request

for you to fetch changes up to 858191aebda251a4d1e3bc77b238096673241cdd:

  migration: Postpone postcopy preempt channel to be after main (2023-02-09 21:26:02 +0100)

----------------------------------------------------------------
Migration Pull request

Hi

This are all the reviewed patches for migration:
- AVX512 support for xbzrle (Ling Xu)
- /dev/userfaultd support (Peter Xu)
- Improve ordering of channels (Peter Xu)
- multifd cleanups (Li Zhang)
- Remove spurious files from last merge (me)
  Rebase makes that to you
- Fix mixup between state_pending_{exact,estimate} (me)
- Cache RAM size during migration (me)
- cleanup several functions (me)

Please apply.

----------------------------------------------------------------

Juan Quintela (7):
  migration: Remove spurious files
  migration: Simplify ram_find_and_save_block()
  migration: Make find_dirty_block() return a single parameter
  migration: Split ram_bytes_total_common() in two functions
  migration: Calculate ram size once
  migration: Make ram_save_target_page() a pointer
  migration: I messed state_pending_exact/estimate

Li Zhang (2):
  multifd: cleanup the function multifd_channel_connect
  multifd: Remove some redundant code

Peter Xu (6):
  linux-headers: Update to v6.1
  util/userfaultfd: Support /dev/userfaultfd
  migration: Rework multi-channel checks on URI
  migration: Cleanup postcopy_preempt_setup()
  migration: Add a semaphore to count PONGs
  migration: Postpone postcopy preempt channel to be after main

ling xu (2):
  AVX512 support for xbzrle_encode_buffer
  Update bench-code for addressing CI problem

 MAINTAINERS                                   |    1 +
 .../x86_64-quintela-devices.mak               |    7 -
 .../x86_64-quintela2-devices.mak              |    6 -
 meson.build                                   |   17 +
 include/standard-headers/drm/drm_fourcc.h     |   34 +-
 include/standard-headers/linux/ethtool.h      |   63 +-
 include/standard-headers/linux/fuse.h         |    6 +-
 .../linux/input-event-codes.h                 |    1 +
 include/standard-headers/linux/virtio_blk.h   |   19 +
 linux-headers/asm-generic/hugetlb_encode.h    |   26 +-
 linux-headers/asm-generic/mman-common.h       |    2 +
 linux-headers/asm-mips/mman.h                 |    2 +
 linux-headers/asm-riscv/kvm.h                 |    4 +
 linux-headers/linux/kvm.h                     |    1 +
 linux-headers/linux/psci.h                    |   14 +
 linux-headers/linux/userfaultfd.h             |    4 +
 linux-headers/linux/vfio.h                    |  142 ++
 migration/migration.h                         |   15 +-
 migration/postcopy-ram.h                      |    4 +-
 migration/xbzrle.h                            |    4 +
 migration/migration.c                         |  122 +-
 migration/multifd.c                           |   70 +-
 migration/postcopy-ram.c                      |   31 +-
 migration/ram.c                               |  136 +-
 migration/savevm.c                            |   56 +-
 migration/xbzrle.c                            |  124 ++
 tests/bench/xbzrle-bench.c                    |  473 ++++++
 tests/unit/test-xbzrle.c                      |   42 +-
 util/userfaultfd.c                            |   32 +
 meson_options.txt                             |    2 +
 migration/multifd.c.orig                      | 1274 -----------------
 scripts/meson-buildoptions.sh                 |    3 +
 tests/bench/meson.build                       |    4 +
 util/trace-events                             |    1 +
 34 files changed, 1257 insertions(+), 1485 deletions(-)
 delete mode 100644 configs/devices/x86_64-softmmu/x86_64-quintela-devices.mak
 delete mode 100644 configs/devices/x86_64-softmmu/x86_64-quintela2-devices.mak
 create mode 100644 tests/bench/xbzrle-bench.c
 delete mode 100644 migration/multifd.c.orig

Comments

Peter Maydell Feb. 10, 2023, 1:36 p.m. UTC | #1
On Thu, 9 Feb 2023 at 23:35, Juan Quintela <quintela@redhat.com> wrote:
>
> The following changes since commit 417296c8d8588f782018d01a317f88957e9786d6:
>
>   tests/qtest/netdev-socket: Raise connection timeout to 60 seconds (2023-02-09 11:23:53 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/juan.quintela/qemu.git tags/migration-20230209-pull-request
>
> for you to fetch changes up to 858191aebda251a4d1e3bc77b238096673241cdd:
>
>   migration: Postpone postcopy preempt channel to be after main (2023-02-09 21:26:02 +0100)
>
> ----------------------------------------------------------------
> Migration Pull request
>
> Hi
>
> This are all the reviewed patches for migration:
> - AVX512 support for xbzrle (Ling Xu)
> - /dev/userfaultd support (Peter Xu)
> - Improve ordering of channels (Peter Xu)
> - multifd cleanups (Li Zhang)
> - Remove spurious files from last merge (me)
>   Rebase makes that to you
> - Fix mixup between state_pending_{exact,estimate} (me)
> - Cache RAM size during migration (me)
> - cleanup several functions (me)
>
> Please apply.
>
> ----------------------------------------------------------------

Fails to build the user-mode emulators:
https://gitlab.com/qemu-project/qemu/-/jobs/3749435025

In file included from ../authz/base.c:24:
../authz/trace.h:1:10: fatal error: trace/trace-authz.h: No such file
or directory
1 | #include "trace/trace-authz.h"

https://gitlab.com/qemu-project/qemu/-/jobs/3749435094
In file included from ../authz/simple.c:23:
../authz/trace.h:1:10: fatal error: trace/trace-authz.h: No such file
or directory
1 | #include "trace/trace-authz.h"


https://gitlab.com/qemu-project/qemu/-/jobs/3749434963
In file included from ../authz/listfile.c:23:
../authz/trace.h:1:10: fatal error: trace/trace-authz.h: No such file
or directory
1 | #include "trace/trace-authz.h"

etc

thanks
-- PMM
Peter Maydell Feb. 10, 2023, 2:33 p.m. UTC | #2
On Fri, 10 Feb 2023 at 14:21, Juan Quintela <quintela@redhat.com> wrote:
>
> Peter Maydell <peter.maydell@linaro.org> wrote:
> > Fails to build the user-mode emulators:
>
> This is weird.

> > https://gitlab.com/qemu-project/qemu/-/jobs/3749435025
> >
> > In file included from ../authz/base.c:24:
> > ../authz/trace.h:1:10: fatal error: trace/trace-authz.h: No such file
> > or directory
> > 1 | #include "trace/trace-authz.h"
>
> This series only have one change for traces:
>
> diff --git a/util/trace-events b/util/trace-events
> index c8f53d7d9f..16f78d8fe5 100644
> --- a/util/trace-events
> +++ b/util/trace-events
> @@ -93,6 +93,7 @@ qemu_vfio_region_info(const char *desc, uint64_t region_ofs, uint64_t region_siz
>  qemu_vfio_pci_map_bar(int index, uint64_t region_ofs, uint64_t region_size, int ofs, void *host) "map region bar#%d addr 0x%"PRIx64" size 0x%"PRIx64" ofs 0x%x host %p"
>
>  #userfaultfd.c
> +uffd_detect_open_mode(int mode) "%d"
>  uffd_query_features_nosys(int err) "errno: %i"
>  uffd_query_features_api_failed(int err) "errno: %i"
>  uffd_create_fd_nosys(int err) "errno: %i"
>
> Rest of trace mentions are for the removal of migration.multifd.c.orig
>
> And I don't play with authentication at all.
>
> This is Fedora 37.
>
> > https://gitlab.com/qemu-project/qemu/-/jobs/3749435094
> > In file included from ../authz/simple.c:23:
> > ../authz/trace.h:1:10: fatal error: trace/trace-authz.h: No such file
> > or directory
>
> Problem is that this trace file is not generated, but I can think how
> any change that I did can influence this.
>
> > 1 | #include "trace/trace-authz.h"
> >
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/3749434963
> > In file included from ../authz/listfile.c:23:
> > ../authz/trace.h:1:10: fatal error: trace/trace-authz.h: No such file
> > or directory
> > 1 | #include "trace/trace-authz.h"
>
> Looking at the ouptut of these, they are not informatives at all.
>
> I am going to try to compile linux-user without system, and see if that
> brings a clue.

Yes, I suspect this is a "user-mode only build" specific failure
(you may need --disable-system --disable-tools to see it).

meson.build only puts authz into trace_events_subdirs "if have_block"
(which is to say "if have_system or have_tools"). However the
bit of meson.build that says "subdir('authz') does not have
the same condition on it -- it's just been put in the list without
any condition on it. So I think that in a build-only-user-emulators
config meson will not generate trace events for the subdirectory
but will try to build it, which falls over.

Contrast 'block', 'nbd', 'scsi', which are all guarded by
'if have_block' for their subdir() lines, to match the guard on
the trace_events_subdirs. OTOH 'io' is also mismatched-guards...

Why this only shows up with your pullreq I have no idea.

thanks
-- PMM
Peter Maydell Feb. 10, 2023, 4:17 p.m. UTC | #3
On Fri, 10 Feb 2023 at 16:13, Juan Quintela <quintela@redhat.com> wrote:
> Again, I don't know why it fails.
>
> diff --git a/tests/bench/meson.build b/tests/bench/meson.build
> index daefead58d..7477a1f401 100644
> --- a/tests/bench/meson.build
> +++ b/tests/bench/meson.build
> @@ -3,9 +3,11 @@ qht_bench = executable('qht-bench',
>                         sources: 'qht-bench.c',
>                         dependencies: [qemuutil])
>
> +if have_system
>  xbzrle_bench = executable('xbzrle-bench',
>                         sources: 'xbzrle-bench.c',
>                         dependencies: [qemuutil,migration])
> +endif
>
>  executable('atomic_add-bench',
>             sources: files('atomic_add-bench.c'),
>
> This make it works.

Before you added your test, meson had no need to compile
any of the object files in 'migration', so it didn't. Now
you tell meson to build a new executable, and it says "OK,
I must build these object files". Only it turns out that
they won't actually compile in this config, so you get an
error.

The same issue can happen in good old Make :-)

thanks
-- PMM