[v5,0/6] MSG_ZEROCOPY + multifd

Message

Leonardo Bras Nov. 12, 2021, 5:10 a.m. UTC

This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make
use of it for multifd migration performance improvement. 

Patch #1 creates new callbacks for QIOChannel, allowing the implementation
of zerocopy writing.

Patch #2 reworks qio_channel_socket_writev() so it accepts flags for 
that are later passed to sendmsg().

Patch #3 implements writev_zerocopy and flush_zerocopy on QIOChannelSocket,
making use of MSG_ZEROCOPY on Linux.

Patch #4 adds a "zerocopy" migration property, only available with
CONFIG_LINUX, and compiled-out in any other architectures. 
This migration property has to be enabled before multifd migration starts.

Patch #5 adds a helper function that allows to see if TLS is going to be used.
This helper will be later used in patch #6.

Patch #6 Makes use of QIOChannelSocket zerocopy implementation on
nocomp multifd migration.

Results:
So far, the resource usage of __sys_sendmsg() reduced 15 times, and the
overall migration took 13-22% less time, based in synthetic cpu workload.

The objective is to reduce migration time in hosts with heavy cpu usage.

---
Changes since v4:
- 3 patches got splitted in 6
- Flush is used for syncing after each iteration, instead of only at the end
- If zerocopy is not available, fail in connect instead of failing on write
- 'multifd-zerocopy' property renamed to 'zerocopy'
- Fail migrations that don't support zerocopy, if it's enabled.
- Instead of checking for zerocopy at each write, save the flags in
  MultiFDSendParams->write_flags and use them on write
- Reorganized flag usage in QIOChannelSocket 
- A lot of typos fixed
- More doc on buffer restrictions

Changes since v3:
- QIOChannel interface names changed from io_async_{writev,flush} to
  io_{writev,flush}_zerocopy
- Instead of falling back in case zerocopy is not implemented, return
  error and abort operation.
- Flush now waits as long as needed, or return error in case anything
  goes wrong, aborting the operation.
- Zerocopy is now conditional in multifd, being set by parameter
  multifd-zerocopy
- Moves zerocopy_flush to multifd_send_sync_main() from multifd_save_cleanup
  so migration can abort if flush goes wrong.
- Several other small improvements

Changes since v2:
- Patch #1: One more fallback
- Patch #2: Fall back to sync if fails to lock buffer memory in MSG_ZEROCOPY send.

Changes since v1:
- Reimplemented the patchset using async_write + async_flush approach.
- Implemented a flush to be able to tell whenever all data was written.

Leonardo Bras (6):
  QIOChannel: Add io_writev_zerocopy & io_flush_zerocopy callbacks
  QIOChannelSocket: Add flags parameter for writing
  QIOChannelSocket: Implement io_writev_zerocopy & io_flush_zerocopy for
    CONFIG_LINUX
  migration: Add zerocopy parameter for QMP/HMP for Linux
  migration: Add migrate_use_tls() helper
  multifd: Implement zerocopy write in multifd migration
    (multifd-zerocopy)

 qapi/migration.json         |  18 ++++
 include/io/channel-socket.h |   2 +
 include/io/channel.h        |  94 ++++++++++++++++---
 migration/migration.h       |   6 ++
 migration/multifd.h         |   4 +-
 io/channel-socket.c         | 176 ++++++++++++++++++++++++++++++++++--
 io/channel.c                |  63 ++++++++++---
 migration/channel.c         |   6 +-
 migration/migration.c       |  41 +++++++++
 migration/multifd.c         |  64 ++++++++++---
 migration/ram.c             |  29 ++++--
 migration/socket.c          |   5 +
 monitor/hmp-cmds.c          |   6 ++
 13 files changed, 455 insertions(+), 59 deletions(-)