[0/2] virtio: failover: allow to keep the VFIO device rather than the virtio-net one

Message

Laurent Vivier July 29, 2021, 7:19 p.m. UTC

With failover, when the guest virtio-net driver doesn't support the
STANDBY feature, the primary device is not plugged and only the virtio-net
device is kept. Doing like that we can migrate the machine and
keep the network connection.

But in some cases, when performance is more important than availability
we would prefer to keep the VFIO device rather than the virtio-net one,
even if it means to lose the network connection during the migration of
the machine.

To do that we can't simply unplug the virtio-net device and plug the
VFIO one because for the migration the initial state must be kept
(virtio-net plugged, VFIO unplugged) but we can try to disable the
virtio-net driver and plug the VFIO card, so the initial state is
correct (the virtio-net card is plugged, but disabled in guest, and
the VFIO card is unplugged before migration).

A way to disable the virtio-net driver at startup is to trigger an
error in the kernel probing function. We can do that by disabling
the RX queues. I tried to add a function to disable the queue that
does that by setting the queue vring "num" value to 0 (and re-enable the
queue by setting back to the default value).

This change doesn't impact the case when guest and host support
the STANDBY feature.

I've introduced the "failover-default" property to virtio-net device
to set which device to keep (failover-default=true keeps the virtio-net
device, =off the other one).

For example, with a guest driver that doesn't support STANDBY:

  ...
  -device virtio-net-pci,id=virtio0,failover=on,failover-default=on \
  -device vfio-pci,host=$PCI,id=hostdev0,failover_pair_id=virtio0 \
  ...

  [root@localhost ~]# ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP q0
      link/ether 26:28:c5:7f:14:24 brd ff:ff:ff:ff:ff:ff
      inet 192.168.20.2/24 brd 192.168.20.255 scope global eth0
         valid_lft forever preferred_lft forever
      inet6 fe80::2428:c5ff:fe7f:1424/64 scope link
         valid_lft forever preferred_lft forever
  # ethtool -i eth0
  driver: virtio_net
  version: 1.0.0
  firmware-version:
  expansion-rom-version:
  bus-info: 0000:04:00.0
  supports-statistics: no
  supports-test: no
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: no

  ...
  -device virtio-net-pci,id=virtio0,failover=on,failover-default=off \
  -device vfio-pci,host=$PCI,id=hostdev0,failover_pair_id=virtio0 \
  ...

  [root@localhost ~]# ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 100
      link/ether 26:28:c5:7f:14:24 brd ff:ff:ff:ff:ff:ff
      inet 192.168.20.2/24 brd 192.168.20.255 scope global enp2s0
         valid_lft forever preferred_lft forever
      inet6 fe80::2428:c5ff:fe7f:1424/64 scope link
         valid_lft forever preferred_lft forever
  [root@localhost ~]# ethtool -i enp2s0
  driver: i40evf
  version: 1.6.27-k
  firmware-version: N/A
  expansion-rom-version:
  bus-info: 0000:02:00.0
  supports-statistics: yes
  supports-test: no
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: no

With guest driver that supports STANDBY, we would always have:

  [root@localhost ~]# ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defau0
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP gr0
      link/ether 26:28:c5:7f:14:24 brd ff:ff:ff:ff:ff:ff
      inet 192.168.20.2/24 brd 192.168.20.255 scope global noprefixroute enp4s0
         valid_lft forever preferred_lft forever
      inet6 fe80::2428:c5ff:fe7f:1424/64 scope link
         valid_lft forever preferred_lft forever
  3: enp4s0nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master0
      link/ether 26:28:c5:7f:14:24 brd ff:ff:ff:ff:ff:ff
  4: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master enp4s0 st0
      link/ether 26:28:c5:7f:14:24 brd ff:ff:ff:ff:ff:ff
  [root@localhost ~]# ethtool -i enp4s0
  driver: net_failover
  version: 0.1
  firmware-version:
  expansion-rom-version:
  bus-info:
  supports-statistics: no
  supports-test: no
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: no
  [root@localhost ~]# ethtool -i enp4s0nsby
  driver: virtio_net
  version: 1.0.0
  firmware-version:
  expansion-rom-version:
  bus-info: 0000:04:00.0
  supports-statistics: yes
  supports-test: no
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: no
  [root@localhost ~]# ethtool -i enp2s0
  driver: iavf
  version: 4.18.0-310.el8.x86_64
  firmware-version: N/A
  expansion-rom-version:
  bus-info: 0000:02:00.0
  supports-statistics: yes
  supports-test: no
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: yes

Laurent Vivier (2):
  virtio: add a way to disable a queue
  virtio: failover: define the default device to use in case of error

 include/hw/virtio/virtio-net.h |  1 +
 include/hw/virtio/virtio.h     |  2 ++
 hw/net/virtio-net.c            | 34 ++++++++++++++++++++++++++++++++++
 hw/virtio/virtio.c             | 10 ++++++++++
 4 files changed, 47 insertions(+)