From patchwork Mon Dec 10 16:15:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venu Busireddy X-Patchwork-Id: 10721505 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6AEEB1759 for ; Mon, 10 Dec 2018 16:17:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 599872A89B for ; Mon, 10 Dec 2018 16:17:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D8392AB9E; Mon, 10 Dec 2018 16:17:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D42662A89B for ; Mon, 10 Dec 2018 16:17:42 +0000 (UTC) Received: from localhost ([::1]:33553 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gWOFZ-00025u-L2 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 10 Dec 2018 11:17:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55442) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gWOEK-0001OL-74 for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gWOEB-0001on-Ap for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:21 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:46212) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gWOE9-0001mZ-8W for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:15 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wBAGD4DV005144; Mon, 10 Dec 2018 16:16:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=9yNkGO3ScnTJ9C7Z3b4BimoHgvKXa+MGstp/F/AQpvM=; b=xD2uAND1EonWw3Aj8rdKFisFPfnIdo0b5K83mVtFbK1CZFuWkYgOnB7ZHjs8HB37Z4RI ct/Q8jHgHH633oyhQIG7vvPpgEUNGsKpyzxvg/KRfHRfLBLf5xEAR0R+ViSzbfgO2Wt8 dR1EEZ1NdZbSrHH7SYhuhIYFv5VzXD2hq4DH2cEDL0wUJRnzrWyG+WQz92FUNrSJMeWl mgpCoBuuQa/KLu4Q17EFA12AFwLLkhG1+NXF1JzNdfD8n2eXA36JWgF3dMCQJTzuOdG6 C336TLPpuC4CJFc/GIOsVqB5e/uUmrDiQoVN/hZdPHiUnpkU8en8tr7T8/QWCvXFx4mM Sg== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2p85ctxwpw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Dec 2018 16:16:09 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wBAGG3Ne008096 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Dec 2018 16:16:03 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wBAGG29j019388; Mon, 10 Dec 2018 16:16:02 GMT Received: from ban25x6uut28.us.oracle.com (/10.153.73.28) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 10 Dec 2018 08:16:02 -0800 From: Venu Busireddy To: venu.busireddy@oracle.com, "Michael S. Tsirkin" , Marcel Apfelbaum Date: Mon, 10 Dec 2018 11:15:46 -0500 Message-Id: <1544458548-5986-2-git-send-email-venu.busireddy@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1544458548-5986-1-git-send-email-venu.busireddy@oracle.com> References: <1544458548-5986-1-git-send-email-venu.busireddy@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9103 signatures=668679 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812100146 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] [fuzzy] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH 1/3] virtio_net: Add VIRTIO_NET_F_STANDBY feature bit. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sridhar Samudrala , qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Sridhar Samudrala This feature bit can be used by a hypervisor to indicate to the virtio_net device that it can act as a standby for another device with the same MAC address. Signed-off-by: Sridhar Samudrala Signed-off-by: Venu Busireddy --- hw/net/virtio-net.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 385b1a0..411f8fb 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -2198,6 +2198,8 @@ static Property virtio_net_properties[] = { true), DEFINE_PROP_INT32("speed", VirtIONet, net_conf.speed, SPEED_UNKNOWN), DEFINE_PROP_STRING("duplex", VirtIONet, net_conf.duplex_str), + DEFINE_PROP_BIT64("standby", VirtIONet, host_features, VIRTIO_NET_F_STANDBY, + false), DEFINE_PROP_END_OF_LIST(), }; From patchwork Mon Dec 10 16:15:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venu Busireddy X-Patchwork-Id: 10721547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 057C63E9D for ; Mon, 10 Dec 2018 16:19:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E9E952A86D for ; Mon, 10 Dec 2018 16:19:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DE8192A95E; Mon, 10 Dec 2018 16:19:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 342EB2A952 for ; Mon, 10 Dec 2018 16:19:13 +0000 (UTC) Received: from localhost ([::1]:33558 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gWOH2-0003I5-EE for patchwork-qemu-devel@patchwork.kernel.org; Mon, 10 Dec 2018 11:19:12 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55424) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gWOED-0001KG-5o for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gWOE9-0001nG-CB for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:17 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:46172) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gWOE7-0001m4-Bh for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:13 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wBAGCu8Z005030; Mon, 10 Dec 2018 16:16:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=OaRDJtZKn9NN11FtcrNjxSEGyzqXSyu5KgHkBJAtLn8=; b=qe1VDO0iWcKaN+yaQ23/ef6xzH74UTMuaH6MtSoy1T/VSCyefMmKl/aUF5TV0INKvuoj LDNJZ9XKTrs/ZDdRSLCj7RwMAStpeFYOsfbbSSL4QlWFTHL2AwkWpGV+TFDlcVwthUTq SWW+17oa2OLLbMo0wCcPNO0XP4Ilk9Tpjl+yrtA96UmQ5CVy47A0/vbGQzVMHhLJv9m+ 8jyWUNwBhCvgmrwn+QW455+7aQmFJQ9j6EBKq1xkbtwj7xyclXLR5ysUeTWpbguoxK5/ AHomPeyh22Wkd+aZQjqSHxoEoCQfrw7OFkFT7EOfN31AqNQc6nOAlMn9XQ3f76xAbDA+ Zw== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2p85ctxwpr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Dec 2018 16:16:08 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wBAGG7Ov022801 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Dec 2018 16:16:07 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wBAGG75L022087; Mon, 10 Dec 2018 16:16:07 GMT Received: from ban25x6uut28.us.oracle.com (/10.153.73.28) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 10 Dec 2018 08:16:07 -0800 From: Venu Busireddy To: venu.busireddy@oracle.com, "Michael S. Tsirkin" , Marcel Apfelbaum Date: Mon, 10 Dec 2018 11:15:47 -0500 Message-Id: <1544458548-5986-3-git-send-email-venu.busireddy@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1544458548-5986-1-git-send-email-venu.busireddy@oracle.com> References: <1544458548-5986-1-git-send-email-venu.busireddy@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9103 signatures=668679 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812100146 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] [fuzzy] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH 2/3] virtio_net: Add support for "Data Path Switching" during Live Migration. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: virtio-dev@lists.oasis-open.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Added two new events, FAILOVER_PLUG_PRIMARY and FAILOVER_UNPLUG_PRIMARY. The first is emitted when the guest negotiates the F_STANDBY feature bit. The second is emitted when the virtio_net driver is removed, either manually or as a result of guest reboot. Management stack can use these events to determine when to plug/unplug the VF device to/from the guest. Also, the Virtual Functions will be automatically removed from the guest if the guest is rebooted. To properly identify the VFIO devices that must be removed, a new property named "x-failover-primary" is added to the vfio-pci devices. Only the vfio-pci devices that have this property enabled are removed from the guest upon reboot. Signed-off-by: Venu Busireddy --- hw/acpi/pcihp.c | 27 ++++++++++++++++++++++++++ hw/net/virtio-net.c | 23 ++++++++++++++++++++++ hw/vfio/pci.c | 3 +++ hw/vfio/pci.h | 1 + include/hw/pci/pci.h | 1 + include/hw/virtio/virtio-net.h | 4 ++++ qapi/net.json | 44 ++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 103 insertions(+) diff --git a/hw/acpi/pcihp.c b/hw/acpi/pcihp.c index 80d42e1..2a3ffd3 100644 --- a/hw/acpi/pcihp.c +++ b/hw/acpi/pcihp.c @@ -176,6 +176,25 @@ static void acpi_pcihp_eject_slot(AcpiPciHpState *s, unsigned bsel, unsigned slo } } +static void acpi_pcihp_cleanup_failover_primary(AcpiPciHpState *s, int bsel) +{ + BusChild *kid, *next; + PCIBus *bus = acpi_pcihp_find_hotplug_bus(s, bsel); + + if (!bus) { + return; + } + QTAILQ_FOREACH_SAFE(kid, &bus->qbus.children, sibling, next) { + DeviceState *qdev = kid->child; + PCIDevice *pdev = PCI_DEVICE(qdev); + int slot = PCI_SLOT(pdev->devfn); + + if (pdev->failover_primary) { + s->acpi_pcihp_pci_status[bsel].down |= (1U << slot); + } + } +} + static void acpi_pcihp_update_hotplug_bus(AcpiPciHpState *s, int bsel) { BusChild *kid, *next; @@ -207,6 +226,14 @@ static void acpi_pcihp_update(AcpiPciHpState *s) int i; for (i = 0; i < ACPI_PCIHP_MAX_HOTPLUG_BUS; ++i) { + /* + * Set the acpi_pcihp_pci_status[].down bits of all the + * failover_primary devices so that the devices are ejected + * from the guest. We can't use the qdev_unplug() as well as the + * hotplug_handler to unplug the devices, because the guest may + * not be in a state to cooperate. + */ + acpi_pcihp_cleanup_failover_primary(s, i); acpi_pcihp_update_hotplug_bus(s, i); } } diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 411f8fb..d37f33c 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -248,6 +248,28 @@ static void virtio_net_drop_tx_queue_data(VirtIODevice *vdev, VirtQueue *vq) } } +static void virtio_net_failover_notify_event(VirtIONet *n, uint8_t status) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(n); + + if (virtio_has_feature(vdev->guest_features, VIRTIO_NET_F_STANDBY)) { + gchar *path = object_get_canonical_path(OBJECT(n->qdev)); + /* + * Emit the FAILOVER_PLUG_PRIMARY event + * when the status transitions from 0 to VIRTIO_CONFIG_S_DRIVER_OK + * Emit the FAILOVER_UNPLUG_PRIMARY event + * when the status transitions from VIRTIO_CONFIG_S_DRIVER_OK to 0 + */ + if ((status & VIRTIO_CONFIG_S_DRIVER_OK) && + (!(vdev->status & VIRTIO_CONFIG_S_DRIVER_OK))) { + FAILOVER_NOTIFY_EVENT(plug, n, path); + } else if ((!(status & VIRTIO_CONFIG_S_DRIVER_OK)) && + (vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) { + FAILOVER_NOTIFY_EVENT(unplug, n, path); + } + } +} + static void virtio_net_set_status(struct VirtIODevice *vdev, uint8_t status) { VirtIONet *n = VIRTIO_NET(vdev); @@ -256,6 +278,7 @@ static void virtio_net_set_status(struct VirtIODevice *vdev, uint8_t status) uint8_t queue_status; virtio_net_vnet_endian_status(n, status); + virtio_net_failover_notify_event(n, status); virtio_net_vhost_status(n, status); for (i = 0; i < n->max_queues; i++) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5c7bd96..ce1f33c 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3077,6 +3077,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); + pdev->failover_primary = vdev->failover_primary; return; @@ -3219,6 +3220,8 @@ static Property vfio_pci_dev_properties[] = { qdev_prop_nv_gpudirect_clique, uint8_t), DEFINE_PROP_OFF_AUTO_PCIBAR("x-msix-relocation", VFIOPCIDevice, msix_relo, OFF_AUTOPCIBAR_OFF), + DEFINE_PROP_BOOL("x-failover-primary", VFIOPCIDevice, + failover_primary, false), /* * TODO - support passed fds... is this necessary? * DEFINE_PROP_STRING("vfiofd", VFIOPCIDevice, vfiofd_name), diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index b1ae4c0..06ca661 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -167,6 +167,7 @@ typedef struct VFIOPCIDevice { bool no_vfio_ioeventfd; bool enable_ramfb; VFIODisplay *dpy; + bool failover_primary; } VFIOPCIDevice; uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len); diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h index e6514bb..b0111d1 100644 --- a/include/hw/pci/pci.h +++ b/include/hw/pci/pci.h @@ -351,6 +351,7 @@ struct PCIDevice { MSIVectorUseNotifier msix_vector_use_notifier; MSIVectorReleaseNotifier msix_vector_release_notifier; MSIVectorPollNotifier msix_vector_poll_notifier; + bool failover_primary; }; void pci_register_bar(PCIDevice *pci_dev, int region_num, diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h index 4d7f3c8..a697903 100644 --- a/include/hw/virtio/virtio-net.h +++ b/include/hw/virtio/virtio-net.h @@ -22,6 +22,10 @@ #define VIRTIO_NET(obj) \ OBJECT_CHECK(VirtIONet, (obj), TYPE_VIRTIO_NET) +#define FAILOVER_NOTIFY_EVENT(type, n, path) \ + qapi_event_send_failover_##type##_primary \ + (!!n->netclient_name, n->netclient_name, path) + #define TX_TIMER_INTERVAL 150000 /* 150 us */ /* Limit the number of packets that can be sent via a single flush diff --git a/qapi/net.json b/qapi/net.json index 8f99fd9..04a9de9 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -683,3 +683,47 @@ ## { 'event': 'NIC_RX_FILTER_CHANGED', 'data': { '*name': 'str', 'path': 'str' } } + +## +# @FAILOVER_PLUG_PRIMARY: +# +# Emitted when the guest successfully loads the driver after the STANDBY +# feature bit is negotiated. +# +# @device: Indicates the virtio_net device. +# +# @path: Indicates the device path. +# +# Since: 3.0 +# +# Example: +# +# <- {"timestamp": {"seconds": 1432121972, "microseconds": 744001}, +# "event": "FAILOVER_PLUG_PRIMARY", +# "data": {"device": "net0", "path": "/machine/peripheral/net0/virtio-backend"} } +# +## +{ 'event': 'FAILOVER_PLUG_PRIMARY', + 'data': {'*device': 'str', 'path': 'str'} } + +## +# @FAILOVER_UNPLUG_PRIMARY: +# +# Emitted when the guest resets the virtio_net driver. +# The reset can be the result of either unloading the driver or a reboot. +# +# @device: Indicates the virtio_net device. +# +# @path: Indicates the device path. +# +# Since: 3.0 +# +# Example: +# +# <- {"timestamp": {"seconds": 1432121972, "microseconds": 744001}, +# "event": "FAILOVER_UNPLUG_PRIMARY", +# "data": {"device": "net0", "path": "/machine/peripheral/net0/virtio-backend"} } +# +## +{ 'event': 'FAILOVER_UNPLUG_PRIMARY', + 'data': {'*device': 'str', 'path': 'str'} } From patchwork Mon Dec 10 16:15:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venu Busireddy X-Patchwork-Id: 10721571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0306E18A7 for ; Mon, 10 Dec 2018 16:21:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0E492ADF7 for ; Mon, 10 Dec 2018 16:21:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DF2FA2AE31; Mon, 10 Dec 2018 16:21:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 36DBB2AE51 for ; Mon, 10 Dec 2018 16:21:42 +0000 (UTC) Received: from localhost ([::1]:33569 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gWOJR-0004Ll-Be for patchwork-qemu-devel@patchwork.kernel.org; Mon, 10 Dec 2018 11:21:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55448) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gWOEK-0001Og-UI for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gWOED-0001rH-8d for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:24 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:41024) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gWOEB-0001nb-9D for qemu-devel@nongnu.org; Mon, 10 Dec 2018 11:16:17 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wBAGDVV0164622; Mon, 10 Dec 2018 16:16:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=tWUuFNE9YrnSTU0LV3bn9KJaac0Mi//FXLe8n/HMDcg=; b=n0dECaT99nrBiiMUfvauIBnK02v/ZTzaEs2nJc3GEP6XhaLNxbo42d+LZXWfuEtJ+/lf AY6r2v7ThVkyddafyr9jOMqEm7nyS2o/B7UD88SLmKu1SzuNO4w02OOzEhWMBnf+DA1Q Bbjs5dF7z2e/3fykZZkD98ArROypLzN+QXuyyxrL1lEvG2awVuaA+FOileHjD+btz16q zP0xvoAo6FG0cfftV2YV01AOBzuAQs4EtFTCtkXwilIK63nd6btPaLTVNNarZERXkxAG ZV9srGlDiU5ZFgF/BXPQPpLmvG0WUdMHTDhDXRI+kcligV+Zg4a42y+S8K8Q2yf5Ecoz Ug== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2p86kqpu9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Dec 2018 16:16:13 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wBAGGCJb008584 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Dec 2018 16:16:12 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wBAGGCWt022174; Mon, 10 Dec 2018 16:16:12 GMT Received: from ban25x6uut28.us.oracle.com (/10.153.73.28) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 10 Dec 2018 08:16:12 -0800 From: Venu Busireddy To: venu.busireddy@oracle.com, "Michael S. Tsirkin" , Marcel Apfelbaum Date: Mon, 10 Dec 2018 11:15:48 -0500 Message-Id: <1544458548-5986-4-git-send-email-venu.busireddy@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1544458548-5986-1-git-send-email-venu.busireddy@oracle.com> References: <1544458548-5986-1-git-send-email-venu.busireddy@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9103 signatures=668679 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812100146 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] [fuzzy] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH 3/3] vfio-pci: Add FAILOVER_PRIMARY_CHANGED event to shorten downtime during failover X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Si-Wei Liu , virtio-dev@lists.oasis-open.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Si-Wei Liu When a VF is hotplugged into the guest, datapath switching will be performed immediately, which is sub-optimal in terms of timing, and could end up with substantial network downtime. One of ways to shorten this downtime is to switch the datapath only after the VF is seen to get enabled by guest, indicated by the bus master bit in VF's PCI config space getting enabled. The FAILOVER_PRIMARY_CHANGED event is emitted at that time to indicate this condition. Then management stack can kick off datapath switching upon receiving the event. Signed-off-by: Si-Wei Liu Signed-off-by: Venu Busireddy --- hw/vfio/pci.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ qapi/net.json | 26 ++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ce1f33c..ea24ca2 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -34,6 +34,7 @@ #include "pci.h" #include "trace.h" #include "qapi/error.h" +#include "qapi/qapi-events-net.h" #define MSIX_CAP_LENGTH 12 @@ -42,6 +43,7 @@ static void vfio_disable_interrupts(VFIOPCIDevice *vdev); static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled); +static void vfio_failover_notify(VFIOPCIDevice *vdev, bool status); /* * Disabling BAR mmaping can be slow, but toggling it around INTx can @@ -1170,6 +1172,8 @@ void vfio_pci_write_config(PCIDevice *pdev, { VFIOPCIDevice *vdev = PCI_VFIO(pdev); uint32_t val_le = cpu_to_le32(val); + bool may_notify = false; + bool master_was = false; trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); @@ -1180,6 +1184,14 @@ void vfio_pci_write_config(PCIDevice *pdev, __func__, vdev->vbasedev.name, addr, val, len); } + /* Bus Master Enabling/Disabling */ + if (pdev->failover_primary && current_cpu && + range_covers_byte(addr, len, PCI_COMMAND)) { + master_was = !!(pci_get_word(pdev->config + PCI_COMMAND) & + PCI_COMMAND_MASTER); + may_notify = true; + } + /* MSI/MSI-X Enabling/Disabling */ if (pdev->cap_present & QEMU_PCI_CAP_MSI && ranges_overlap(addr, len, pdev->msi_cap, vdev->msi_cap_size)) { @@ -1235,6 +1247,14 @@ void vfio_pci_write_config(PCIDevice *pdev, /* Write everything to QEMU to keep emulated bits correct */ pci_default_write_config(pdev, addr, val, len); } + + if (may_notify) { + bool master_now = !!(pci_get_word(pdev->config + PCI_COMMAND) & + PCI_COMMAND_MASTER); + if (master_was != master_now) { + vfio_failover_notify(vdev, master_now); + } + } } /* @@ -2801,6 +2821,17 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static void vfio_failover_notify(VFIOPCIDevice *vdev, bool status) +{ + PCIDevice *pdev = &vdev->pdev; + const char *n; + gchar *path; + + n = pdev->qdev.id ? pdev->qdev.id : vdev->vbasedev.name; + path = object_get_canonical_path(OBJECT(vdev)); + qapi_event_send_failover_primary_changed(!!n, n, path, status); +} + static void vfio_realize(PCIDevice *pdev, Error **errp) { VFIOPCIDevice *vdev = PCI_VFIO(pdev); @@ -3109,10 +3140,36 @@ static void vfio_instance_finalize(Object *obj) vfio_put_group(group); } +static void vfio_exit_failover_notify(VFIOPCIDevice *vdev) +{ + PCIDevice *pdev = &vdev->pdev; + + /* + * Guest driver may not get the chance to disable bus mastering + * before the device object gets to be unrealized. In that event, + * send out a "disabled" notification on behalf of guest driver. + */ + if (pdev->failover_primary && + pdev->bus_master_enable_region.enabled) { + vfio_failover_notify(vdev, false); + } +} + static void vfio_exitfn(PCIDevice *pdev) { VFIOPCIDevice *vdev = PCI_VFIO(pdev); + /* + * During the guest reboot sequence, it is sometimes possible that + * the guest may not get sufficient time to complete the entire driver + * removal sequence, near the end of which a PCI config space write to + * disable bus mastering can be intercepted by device. In such cases, + * the FAILOVER_PRIMARY_CHANGED "disable" event will not be emitted. It + * is imperative to generate the event on the guest's behalf if the + * guest fails to make it. + */ + vfio_exit_failover_notify(vdev); + vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); diff --git a/qapi/net.json b/qapi/net.json index 04a9de9..e5992c8 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -727,3 +727,29 @@ ## { 'event': 'FAILOVER_UNPLUG_PRIMARY', 'data': {'*device': 'str', 'path': 'str'} } + +## +# @FAILOVER_PRIMARY_CHANGED: +# +# Emitted whenever the driver of failover primary is loaded or unloaded +# by the guest. +# +# @device: device name +# +# @path: device path +# +# @enabled: true if driver is loaded thus device is enabled in guest +# +# Since: 3.0 +# +# Example: +# +# <- { "event": "FAILOVER_PRIMARY_CHANGED", +# "data": { "device": "vfio-0", +# "path": "/machine/peripheral/vfio-0" }, +# "enabled": "true" }, +# "timestamp": { "seconds": 1539935213, "microseconds": 753529 } } +# +## +{ 'event': 'FAILOVER_PRIMARY_CHANGED', + 'data': { '*device': 'str', 'path': 'str', 'enabled': 'bool' } }