From patchwork Mon Jun 26 08:23:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13292474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D918EB64D7 for ; Mon, 26 Jun 2023 08:29:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qDhbD-00018U-KA; Mon, 26 Jun 2023 04:29:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qDhbB-00015X-Nz for qemu-devel@nongnu.org; Mon, 26 Jun 2023 04:29:25 -0400 Received: from mail-sn1nam02on2050.outbound.protection.outlook.com ([40.107.96.50] helo=NAM02-SN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qDhb3-0004Dq-F8 for qemu-devel@nongnu.org; Mon, 26 Jun 2023 04:29:24 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lqMGIfZczlAElzYCfYzeTzJewaHQNXV5J5zNhyy7fkrkomIlLT2gEtISKXtEY/7BFPdlsqWZLF+d91vYCSrUWTZla+aGExtgiL8qPkgB49M4CAZt0rrPZul/C6xSvKs+XLhmRY2zsH0zDc1cRRqi4dySCn1YE8fN0ITxajHKvFabUNpp/v5w3o99gQmFS89IM9av7LnmBaTRYkYcRzAyKBC/6bSMd8u8d21PXaD5w4oOYWbI3vtIfTQdr5vlHSBVKRrG7ocPKF/65tnCxrT4LoIG4RCHs1iXbL0KcJ0HYCtFA/eu3BPH7i0/ZqdFLk1pBwE9wKI/xwxiy+ARann1Vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Myto345SPZplQL2pF3pF3n8IQnpGEGItr0GimxqrUfE=; b=mVMLjtxBhw+jb0wmd3wcwa34wmuXW9gK9AJLeae8yhWbGTpKVFThugY2znXOz8ZEAuZb56xJ87KNImUO/IefQ6UiDwhUzEuTC5oUmnBZpeujKhlbR2geEcX0AqmC98qqMiqQGYsHoeR7V39UdzwzJNO5sYU+XEIVDqLFP5BskzsQ1Drm8VlhpcBNaSJxwcMO3tBh09xgBJXBIG8/rN/x3l9Z5XsTLqYfjGvIrnTIELGF13MC6fQVSlK2QTcmzAICyQilfxPIQSYGgYUR7aM5IdmbZeKhwdSJkBKFmeZZgHB8E2OYqD635iuXlL+CEPI/py2yH3Iz5oYuV7mPUtFm3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Myto345SPZplQL2pF3pF3n8IQnpGEGItr0GimxqrUfE=; b=fUAleG1Xc28bGMvMv3af5BUJkACpnEa2D586VDRa7yvAs/nLIN+uDXKfVMnrWStZIfLPJH76UXVzzoXwbmr+1JcTALVsSm4kwRxntZWG9hMRwsJxo+MLs8oAb+U7cnckQyDHd66+IKQuSYFvmEhKaHJTzHNqYoLrwCFygA8n8j8N8ZXz+D2ATOF9mjLjPNkzA4gmp1r1ZRok0LLyj+eNDtpv1wtOunKOf+YaqYuxDksCRWfTWkwpBDPOKj6CE4gPgFmzaKi9YDXW2s5Th8aR70lb7MCLM8hGml5ExiWknI3Iq0PukaZgmPH7FA43woO/py2ZSqp9Qy6R6QXd+7qY9g== Received: from DM6PR06CA0013.namprd06.prod.outlook.com (2603:10b6:5:120::26) by DM4PR12MB5865.namprd12.prod.outlook.com (2603:10b6:8:64::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.23; Mon, 26 Jun 2023 08:24:12 +0000 Received: from DM6NAM11FT073.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::db) by DM6PR06CA0013.outlook.office365.com (2603:10b6:5:120::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.34 via Frontend Transport; Mon, 26 Jun 2023 08:24:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DM6NAM11FT073.mail.protection.outlook.com (10.13.173.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.47 via Frontend Transport; Mon, 26 Jun 2023 08:24:12 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 26 Jun 2023 01:24:02 -0700 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 26 Jun 2023 01:24:01 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 26 Jun 2023 01:23:58 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Zhenzhong Duan , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 1/3] vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup() Date: Mon, 26 Jun 2023 11:23:51 +0300 Message-ID: <20230626082353.18535-2-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230626082353.18535-1-avihaih@nvidia.com> References: <20230626082353.18535-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT073:EE_|DM4PR12MB5865:EE_ X-MS-Office365-Filtering-Correlation-Id: 35f4e4c1-bd4d-4a31-65a1-08db761eb8c4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gxumrOASINa41Y9FUXGYvrrTdInAjGDEadEyP9yGkR91yDBrqDtyuM3CviE7Ljfw4cp2Y1jQ3DhjfFn4YRn5Mzpb+Bq3ZxTVLdGR4VNNsaPjok2ee94nktSCvY3mm151Q9yU5JVlhKdxTnidkO40hlTWrSv40Ex3dGQI2sUiKBY0oXdmlXXsEtuwaFbPcBiAH5QatFoHqIvrdJe/qsiZbfEFKzJDPiomL7rMiOsPFFVETVhfOpcapo8UPIVj2h1JufO5v4F8hLWcdOUu43aExv6CscqMBnyPEB59TLXlRt/v9UFWZQxpKhqZkzKqGp5havW76CCD2rhbNgdpnPdMSj0JTDV1OctGTCOKPhSkelgYAo7EWJMPYvc4ATywKMD0qwG27J2QwaAyQtKuxcpdxOF1raN0EXJZfS/3gvnLN88gF97v398fhDP8Ig/WY06uFFrMfWZlq2hiI6I9Y2pIZQpCm7zqbp2LJqYYRLocirWczyesjNNyo4j8UAmZH2ctX5WIhuo3i4hTlDfxZNSk5xShi99CJtwGVhE+HnidINxqtFrs81KmA0CZE8ijLGeLAzXYeNL5JuiEJ1xZMoXEJ9dOGGWxxBZMp95kURtF223/jhkwYWNK2Uwka6ZKjJjdy/F2v5fWmIUJoQGCvqNWE7AsseK7BHFhbWrggZJoMhhYIr12kx6DKQTT/ZdPNr0ZrctM6mbsGbRzlp7fNkdTESNflj6BdAx745qM7T+JLgi6ODMtELllxcro9M9G7Uod X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(376002)(136003)(346002)(39860400002)(451199021)(46966006)(40470700004)(36840700001)(6666004)(7696005)(82310400005)(83380400001)(2616005)(186003)(7636003)(356005)(40480700001)(1076003)(426003)(47076005)(336012)(40460700003)(36860700001)(36756003)(316002)(6916009)(70586007)(4326008)(70206006)(8936002)(8676002)(26005)(86362001)(2906002)(41300700001)(5660300002)(54906003)(82740400003)(478600001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2023 08:24:12.2470 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 35f4e4c1-bd4d-4a31-65a1-08db761eb8c4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT073.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5865 Received-SPF: softfail client-ip=40.107.96.50; envelope-from=avihaih@nvidia.com; helo=NAM02-SN1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Changing the device state from STOP_COPY to STOP can take time as the device may need to free resources and do other operations as part of the transition. Currently, this is done in vfio_save_complete_precopy() and therefore it is counted in the migration downtime. To avoid this, change the device state from STOP_COPY to STOP in vfio_save_cleanup(), which is called after migration has completed and thus is not part of migration downtime. Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index acbf0bb7ab..a8bfbe4b89 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -383,6 +383,19 @@ static void vfio_save_cleanup(void *opaque) VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; + /* + * Changing device state from STOP_COPY to STOP can take time. Do it here, + * after migration has completed, so it won't increase downtime. + */ + if (migration->device_state == VFIO_DEVICE_STATE_STOP_COPY) { + /* + * If setting the device in STOP state fails, the device should be + * reset. To do so, use ERROR state as a recover state. + */ + vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, + VFIO_DEVICE_STATE_ERROR); + } + g_free(migration->data_buffer); migration->data_buffer = NULL; migration->precopy_init_size = 0; @@ -508,12 +521,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } - /* - * If setting the device in STOP state fails, the device should be reset. - * To do so, use ERROR state as a recover state. - */ - ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, - VFIO_DEVICE_STATE_ERROR); trace_vfio_save_complete_precopy(vbasedev->name, ret); return ret; From patchwork Mon Jun 26 08:23:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13292471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7AC7EB64DA for ; Mon, 26 Jun 2023 08:24:57 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qDhWM-0004BB-To; Mon, 26 Jun 2023 04:24:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qDhWL-0004B2-Jp for qemu-devel@nongnu.org; Mon, 26 Jun 2023 04:24:25 -0400 Received: from mail-dm6nam12on20602.outbound.protection.outlook.com ([2a01:111:f400:fe59::602] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qDhWI-00034M-CL for qemu-devel@nongnu.org; Mon, 26 Jun 2023 04:24:25 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dTnLtF3ZHXoGH+4VlylYzs1FPEIFl3mFU8AgDIfxPlzTvPC2jLQdR43PPURw/+X0ICt1qZecw24EFAjeXqW0U+eNBckfb0HuN3s+C6vO3zHsCLAJuUoA/gXztKsI4NQqFmGrvqx/oHeMjbZIZikTMKyfPMPqB7NKkBWRPPi0KEGdh0yNdxZzwX2mQnJ7pfYr3sBYhi6lwRCFCxk/QkePu2geJTLCL0m/G51M07MjtLHA/fQE7ztUy3fk4u/dy6P7JvJ/b3U4yYa6hMIEsZxddKDA3dULEd/SkBf6xviCi5iR3BwP1vPhiebmqIbdc9HEx3k4Bb4VMiFbMKtzp7SnVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ltq5nsS59lQoe80rfolg7qjptpZfDkJvzc0TJhRFeDI=; b=mhyWqyZ+9dUUS3Pzb6aK4Lcon1tZLjUdG/ucIkE6UXqGwnTHGmjhRDMwxpK7F1DeHHTVqwVymd4HaTfqS/b7QCBEEHf81fVJwZAfmDk3cpf6Krf8qdLwo+5yKk7IT3C5phU4p6X8tHRK7a6Vdtirg1CVoIIcYDNnStZmmRvCaynSf036i6CUJw3vn2De8dvzhviAybFhDidQkzCNfyqPZzwSbBgaSY2SGO/IIGfEp8E1ia2fFJWfLMvCZ5DjZfIi775VDKbovjcnvNHVwDduKJYyABMGKHAqr11mLaO1CfneUWN+levix54zJF4+SGkxVWLjWL/FIazYh+jbaDeWYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ltq5nsS59lQoe80rfolg7qjptpZfDkJvzc0TJhRFeDI=; b=jcd6LqijIE2HQ3xD4prb2ZTv2GIHRjVExw6J4lgbgpzZ8MAXQUS7/woFrc43P71aHBeYswYQXUaZ1v71dma7qtzqfXHJn2rnNLN/yIHQqWODRk9I5svy1ar3kft/ftIPXixa1rhea8ybDz0PyQluKXxiRuMNNzZAYweR8L+tIoa4nporwmBZuP+tNQE6FPXrazpnm33P10+5xzIUaFZsUs+zydRmnje4P8lI1M6giudyRwkZodub6pKnLQ9NwFHvZfiaMRYcr6AoWGvwQXcgXoTIAvXXSpSSnSTqy6CeNuJh/yDBRlSYTM0vGV1a1IZoSh/ilNCzgZGCBMZlLmQ3Fg== Received: from BN9P222CA0008.NAMP222.PROD.OUTLOOK.COM (2603:10b6:408:10c::13) by DS7PR12MB5958.namprd12.prod.outlook.com (2603:10b6:8:7d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.23; Mon, 26 Jun 2023 08:24:18 +0000 Received: from BN8NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10c:cafe::b2) by BN9P222CA0008.outlook.office365.com (2603:10b6:408:10c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.33 via Frontend Transport; Mon, 26 Jun 2023 08:24:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT065.mail.protection.outlook.com (10.13.177.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.47 via Frontend Transport; Mon, 26 Jun 2023 08:24:18 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 26 Jun 2023 01:24:06 -0700 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 26 Jun 2023 01:24:05 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 26 Jun 2023 01:24:02 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Zhenzhong Duan , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 2/3] vfio/migration: Reset bytes_transferred properly Date: Mon, 26 Jun 2023 11:23:52 +0300 Message-ID: <20230626082353.18535-3-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230626082353.18535-1-avihaih@nvidia.com> References: <20230626082353.18535-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT065:EE_|DS7PR12MB5958:EE_ X-MS-Office365-Filtering-Correlation-Id: 5b4f8f4c-8eb4-4b1b-e8d6-08db761ebc75 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wA4N2chcX0iX1DtfKDhQmxJ1xZu/wtv0kL4n/O+tt+4X9eKWs6Fp+WjGPqB6GrFqRiqKMxM+Ou25KTfXkzQReHwv1l6BWMGNCFhUy2uj2YlTLPJm8H77ZdcRMYfczVrNIo0BnbMRPU9lnYR2ggwnDPRreKj1r7TsWzR6LRxXqxb2wzu42/sAEZaE3FCNzXYYOtYWFHbrasPZn5Iia2XiOCqgqC07a2s1CPI+OotrCCrc9T9NZBSGb7hQOsfDq/88NH43slhVveWNobpca+OnaBrqMq4iCa3H12z/zzVuqiXyc4AqJ5kWIRRQUf/t5siXqMiphVJt+KDQxiEX7DHACZ7AN/sHA318hEVYU8s2p/nX2ttTF8PqnCUjP527TWb3sh5ey5nkPfCAHe9RbXzEL2joNrY1BPnhuzCVXm5M5q6QFgX4n0qM83AN0k9NnQSPlUkxclQ6NH2R6/E81SfkVHWwgRxAk24uAQwxeyg2cXxwO6WL0bKVIQkxzLNtklWf+oy1xCAqsBa4G7HOzAwRV6XnTkQMSnMeCNJsI4smihtLgLCrj1WrUOW+wxHGdZvykFozBlAXrnALVPnV8DF3ZYuAnz/gT5aIRuQa2QdUuATBTKvAzYGwxuw1llaWAqSRgokLXvrwSN/HTadwpt3+U1CWg3xGDsRhbP43nWRavZ+xBeZxUZ/SUtcFZxOLtpAHyYmeVIVb68Btac5O5XnrDD7a6k5ZTvVnqzI8CLJEKZYwnwU2K52AVQnuSinw6FL3 X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(376002)(346002)(136003)(396003)(39860400002)(451199021)(36840700001)(46966006)(40470700004)(82310400005)(36860700001)(47076005)(478600001)(426003)(54906003)(2616005)(6666004)(7696005)(336012)(83380400001)(1076003)(26005)(186003)(2906002)(5660300002)(356005)(36756003)(70206006)(40460700003)(70586007)(82740400003)(40480700001)(7636003)(8936002)(8676002)(316002)(86362001)(6916009)(4326008)(41300700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2023 08:24:18.3966 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5b4f8f4c-8eb4-4b1b-e8d6-08db761ebc75 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT065.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5958 Received-SPF: softfail client-ip=2a01:111:f400:fe59::602; envelope-from=avihaih@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently, VFIO bytes_transferred is not reset properly: 1. bytes_transferred is not reset after a VM snapshot (so a migration following a snapshot will report incorrect value). 2. bytes_transferred is a single counter for all VFIO devices, however upon migration failure it is reset multiple times, by each VFIO device. Fix it by introducing a new function vfio_reset_bytes_transferred() and calling it during migration and snapshot start. Remove existing bytes_transferred reset in VFIO migration state notifier, which is not needed anymore. Signed-off-by: Avihai Horon --- include/hw/vfio/vfio-common.h | 1 + migration/migration.h | 1 + hw/vfio/migration.c | 6 +++++- migration/migration.c | 1 + migration/savevm.c | 1 + migration/target.c | 17 +++++++++++++++-- 6 files changed, 24 insertions(+), 3 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 3dc5f2104c..b4c28f318f 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -228,6 +228,7 @@ int vfio_block_multiple_devices_migration(Error **errp); void vfio_unblock_multiple_devices_migration(void); int vfio_block_giommu_migration(Error **errp); int64_t vfio_mig_bytes_transferred(void); +void vfio_reset_bytes_transferred(void); #ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index, diff --git a/migration/migration.h b/migration/migration.h index c859a0d35e..a80b22b703 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -514,6 +514,7 @@ bool migration_rate_limit(void); void migration_cancel(const Error *error); void populate_vfio_info(MigrationInfo *info); +void reset_vfio_bytes_transferred(void); void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page); #endif diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index a8bfbe4b89..79eb81dfd7 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -704,7 +704,6 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) case MIGRATION_STATUS_CANCELLING: case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: - bytes_transferred = 0; /* * If setting the device in RUNNING state fails, the device should * be reset. To do so, use ERROR state as a recover state. @@ -825,6 +824,11 @@ int64_t vfio_mig_bytes_transferred(void) return bytes_transferred; } +void vfio_reset_bytes_transferred(void) +{ + bytes_transferred = 0; +} + int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) { int ret = -ENOTSUP; diff --git a/migration/migration.c b/migration/migration.c index 7653787f74..096e8191d1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1628,6 +1628,7 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, */ memset(&mig_stats, 0, sizeof(mig_stats)); memset(&compression_counters, 0, sizeof(compression_counters)); + reset_vfio_bytes_transferred(); return true; } diff --git a/migration/savevm.c b/migration/savevm.c index cdf4793924..95c2abf47c 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1622,6 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) migrate_init(ms); memset(&mig_stats, 0, sizeof(mig_stats)); memset(&compression_counters, 0, sizeof(compression_counters)); + reset_vfio_bytes_transferred(); ms->to_dst_file = f; qemu_mutex_unlock_iothread(); diff --git a/migration/target.c b/migration/target.c index 00ca007f97..f39c9a8d88 100644 --- a/migration/target.c +++ b/migration/target.c @@ -14,12 +14,25 @@ #include "hw/vfio/vfio-common.h" #endif +#ifdef CONFIG_VFIO void populate_vfio_info(MigrationInfo *info) { -#ifdef CONFIG_VFIO if (vfio_mig_active()) { info->vfio = g_malloc0(sizeof(*info->vfio)); info->vfio->transferred = vfio_mig_bytes_transferred(); } -#endif } + +void reset_vfio_bytes_transferred(void) +{ + vfio_reset_bytes_transferred(); +} +#else +void populate_vfio_info(MigrationInfo *info) +{ +} + +void reset_vfio_bytes_transferred(void) +{ +} +#endif From patchwork Mon Jun 26 08:23:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13292472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FE04EB64DA for ; Mon, 26 Jun 2023 08:25:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qDhWT-0004Bs-BF; Mon, 26 Jun 2023 04:24:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qDhWR-0004Bh-Rx for qemu-devel@nongnu.org; Mon, 26 Jun 2023 04:24:31 -0400 Received: from mail-dm6nam11on20612.outbound.protection.outlook.com ([2a01:111:f400:7eaa::612] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qDhWO-00034g-PS for qemu-devel@nongnu.org; Mon, 26 Jun 2023 04:24:31 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YbFp2QWqabhE/eSweGt4Koq4WWZE/81rNBghRupsrYs9+kjckgIeoikxCyu9k0WGozf/bSzB09mVvIGd4A7IA4Swm0OQzDiJhucMnWD9hL6eJOpGRs7il9d7drPeZoKE5xdhresUS0xmWPuzicGW1wHh93XaXho9RwGYcnZ0is1MUfQRo0BH+VRKk5dpnA3Cw8w9d4LS7pGHpnYrQQDXmROnr3ZQM1LDCDUmEKpVefqtW13hkgu/Iwd0kfosRlvLzviESOkFhodig3A5hmgFpJ6dDTZXQqophltKbI+HnqPjVjX2D1MewmZs26039e9t4flRz+IMpxdGykFfu7LBgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lb8EHhPAsEDfLr8hzvZqlDif7Z9MMT/ZERa3VhHC250=; b=dJMwkjCGsS8f4lacwi3114qM3X+my9ojL8IqPKm9C2e14XF9qZwpWGKNvQhxcroHLF5hQmDoVM6r8DtahJ32QBR+g3j9LkMZ1BikP8p9Ofet5F/doAfWrk9x/uKkcJySXraOca9qCT/fw79Xiy+v9pigkR9iRYIRS3LVZ/YIZYCp3LqcvvbXPqWqwhn2mr42tJREY79WHY9fYTayo4+oluAURtx6pqGp6fBUPjuEVaCvJ2vYPB9ZjkEwGllZonWTiBP1HvACMab673obWL1SSyPCpH9LlG7XdtA1+evEUXSAgJ3htgDD5RVjVj0svn7/IzCPMHBsYZ7iB8OSKzVMjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lb8EHhPAsEDfLr8hzvZqlDif7Z9MMT/ZERa3VhHC250=; b=oj/lNUnYk+9/CByTJK2JpZFTIRJN76cjfpAdvw9ztNsjnnpTV+HYQHdTiJUCUDpqEyEvmA3IUPeAURMz2QeZEHgIi51cwOMsjpzfFNJP5BcCIVSvhZvlkByBnoBH8OHT3kbIYQMHIxcbi6fhVfMMQMEC0ZSq+TsKgig+AKBbpC+mnskJIm3u/X4PH3X32nUfwb0Sz6eD/oMqXEZ5YmLtT6CdIouVzaNnq/3Wpz+5KCKgT20HkD21+SB/Xk1xVuzSsRBgFeyePnt+PZHKqTCLfOeznhaiw+rAM9VCvpDUSNAFXJfBIpogSySb4YcCLD2/NKHa0nQuPXkXeDQB2/Ax0g== Received: from DS7PR03CA0254.namprd03.prod.outlook.com (2603:10b6:5:3b3::19) by SA1PR12MB8723.namprd12.prod.outlook.com (2603:10b6:806:385::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.24; Mon, 26 Jun 2023 08:24:24 +0000 Received: from DM6NAM11FT066.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b3:cafe::97) by DS7PR03CA0254.outlook.office365.com (2603:10b6:5:3b3::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.33 via Frontend Transport; Mon, 26 Jun 2023 08:24:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DM6NAM11FT066.mail.protection.outlook.com (10.13.173.179) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.47 via Frontend Transport; Mon, 26 Jun 2023 08:24:23 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 26 Jun 2023 01:24:10 -0700 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 26 Jun 2023 01:24:09 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 26 Jun 2023 01:24:06 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Zhenzhong Duan , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 3/3] vfio/migration: Make VFIO migration non-experimental Date: Mon, 26 Jun 2023 11:23:53 +0300 Message-ID: <20230626082353.18535-4-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230626082353.18535-1-avihaih@nvidia.com> References: <20230626082353.18535-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT066:EE_|SA1PR12MB8723:EE_ X-MS-Office365-Filtering-Correlation-Id: 44bebae8-8ec1-46c9-67bb-08db761ebf9f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QfriOs1UYJQzr21BjHBtExrffNFNiJkzSK0gn8mS87tgnbU99kOwLMdeucugfuDHMWYBsQsg/FsVjeBml3ajDs6D1+P0yhQO+RHf6TRclvC+hdMHvdR0DsNOjUX9U98/KyROaRYwzX8lotvFKSpaQ7bjLD1DM8bGeJTtCtFzU/XGP8iGPkq6+WYgKW7ye9FUTIL2PckU47pju1V18DtTITfBuKfn1jq65RKFrS6cdaL1E9QOOmzMtFjwnguKYVsX05StXt2vG/qX6BGTB+8M1oHBQECaEZD71kWiicEVP2J5OwZAXR/Z6qpGTsiFiSkibYu9Dhm2RkMcw+Ky353RolJMrFI/5ES74NufJ0iUEQ1LACyp8jEONNI4jQDZ1eH9GFb8x0P2PRCC5wysQDWXEtECMYLJXqxvG2l4CkjM7vBVG1eL0qapz5h73BVmM1Kt3QgplpNwf3zBktzhvsaf8wX0h570LYUfw/kc1FmRZOvYu/kWZhuJ5+tJOUptsUcJHPT3ff4245VKxD6+alGE7tDspggTF/L0QO/DC6g1EUjh3AG8/gg8T/a1RPN3YFy3LzbUYITRogTAsbv7ZiNju0Jr2sqxiZmMNpVX+ZOWFspO24Fk+VHNlAG+3qjBpRkIyHmSkTML42mzYdtH5QoqLNWK6nQ6d4DNTHj27cduyc2f4F4qi0iSruJ1fsguLzh0WXYkA6xDRDkpKRvy1KCbHLJoOFVtYLLRTy23kKvz++DLktMnUtfgF5ktj8f8Kost X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(346002)(376002)(136003)(396003)(451199021)(36840700001)(40470700004)(46966006)(82310400005)(36860700001)(2906002)(47076005)(478600001)(426003)(7696005)(54906003)(2616005)(6666004)(83380400001)(336012)(26005)(186003)(1076003)(5660300002)(356005)(40460700003)(36756003)(70206006)(70586007)(82740400003)(7636003)(40480700001)(316002)(4326008)(8936002)(8676002)(41300700001)(86362001)(6916009); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2023 08:24:23.7486 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 44bebae8-8ec1-46c9-67bb-08db761ebf9f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT066.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8723 Received-SPF: softfail client-ip=2a01:111:f400:7eaa::612; envelope-from=avihaih@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The major parts of VFIO migration are supported today in QEMU. This includes basic VFIO migration, device dirty page tracking and precopy support. Thus, at this point in time, it seems appropriate to make VFIO migration non-experimental: remove the x prefix from enable_migration property, change it to ON_OFF_AUTO and let the default value be AUTO. In addition, make the following adjustments: 1. Require device dirty tracking support when enable_migration is AUTO (i.e., not explicitly enabled). This is because device dirty tracking is currently the only method to do dirty page tracking, which is essential for migrating in a reasonable downtime. Setting enable_migration to ON will not require device dirty tracking. 2. Make migration blocker messages more elaborate. 3. Remove error prints in vfio_migration_query_flags(). 4. Remove a redundant assignment in vfio_migration_realize(). Signed-off-by: Avihai Horon --- include/hw/vfio/vfio-common.h | 2 +- hw/vfio/migration.c | 29 ++++++++++++++++------------- hw/vfio/pci.c | 4 ++-- 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index b4c28f318f..387eabde60 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -139,7 +139,7 @@ typedef struct VFIODevice { bool needs_reset; bool no_mmap; bool ram_block_discard_allowed; - bool enable_migration; + OnOffAuto enable_migration; VFIODeviceOps *ops; unsigned int num_irqs; unsigned int num_regions; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 79eb81dfd7..d8e0848635 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -731,14 +731,6 @@ static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) feature->argsz = sizeof(buf); feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION; if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { - if (errno == ENOTTY) { - error_report("%s: VFIO migration is not supported in kernel", - vbasedev->name); - } else { - error_report("%s: Failed to query VFIO migration support, err: %s", - vbasedev->name, strerror(errno)); - } - return -errno; } @@ -831,14 +823,28 @@ void vfio_reset_bytes_transferred(void) int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) { - int ret = -ENOTSUP; + int ret; - if (!vbasedev->enable_migration) { + if (vbasedev->enable_migration == ON_OFF_AUTO_OFF) { + error_setg(&vbasedev->migration_blocker, + "%s: Migration is disabled for VFIO device", vbasedev->name); goto add_blocker; } ret = vfio_migration_init(vbasedev); if (ret) { + error_setg(&vbasedev->migration_blocker, + "%s: Migration couldn't be initialized for VFIO device, " + "err: %d (%s)", + vbasedev->name, ret, strerror(-ret)); + goto add_blocker; + } + + if (vbasedev->enable_migration == ON_OFF_AUTO_AUTO && + !vbasedev->dirty_pages_supported) { + error_setg(&vbasedev->migration_blocker, + "%s: VFIO device doesn't support device dirty tracking", + vbasedev->name); goto add_blocker; } @@ -856,9 +862,6 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return 0; add_blocker: - error_setg(&vbasedev->migration_blocker, - "VFIO device doesn't support migration"); - ret = migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) { error_free(vbasedev->migration_blocker); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 73874a94de..48584e3b01 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3347,8 +3347,8 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_REQ_BIT, true), DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), - DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, - vbasedev.enable_migration, false), + DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, + vbasedev.enable_migration, ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice, vbasedev.ram_block_discard_allowed, false),