From patchwork Wed Aug 2 08:14:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13337835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89154C04A94 for ; Wed, 2 Aug 2023 08:16:01 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qR70r-0006oJ-Dy; Wed, 02 Aug 2023 04:15:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70q-0006im-3s for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:20 -0400 Received: from mail-co1nam11on2060c.outbound.protection.outlook.com ([2a01:111:f400:7eab::60c] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70o-0006Oa-8U for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:19 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F6Rng97+cjvCYvTnnJYHJa5IA3nT8x/pHK+tnyBqUUsAe7kQKtD1ttLtf3eQw8HKxgR5w/VTKE7Jxvn0sLJIl9VccNmDRqpU+orjLbbb2sAfIigDdY6EfyUiDAQE4U12GX8JKvisIP2z9mEoqhWgaDIpSu8OxVL6wwz1zV2KlXaJilk5LPpHWHPmCFCPmhvJ7PnvnzgUja+jtaiAnnBz+LO56otBOvMdYfmuiYnIZVn1gg4KJWb5M552MWTZvO+i5B4YIBGKS2przLXMlvqxPjPLt9er92BP0cQvBZxvauzSYe23bMqs5Uc3eAg7OLiXBvUKtWc2F9jBBOF/MhQbQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8JJJi1BljvRnqc53fMAMnBZ/7qU6EKYrUo6YPtPQy70=; b=RLGrdeqeXLim677zX/houHqOwVrD3iDGlBDVFban2vRk+ofdyJnExMGHQbWRKmgjNdBnK7jDcqMaelPE7H61KSOKeASwSfJAKMelkv4r64DQfcx6p/laEYdOTjMi8pfnFiPgoMio0yUwUHsyJl2Sb/8moVC4yERglIC1oCdL3rEV1FmSVMszwupiLBES9YKvvKi2Vp7o5W6UouQNzMMOpm3jEy7fBaGN+rjX9iV1ZLERw/DUBT782O0jRJC9FUytvAN2ANIvrZxq0O+dySg7goip2KQEbH6k9ikzdn+XhwLoXkm4iu9qvu0BL3YPAoG4xzbw8VwklJfOUQM/zf/B/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8JJJi1BljvRnqc53fMAMnBZ/7qU6EKYrUo6YPtPQy70=; b=AV+ZAcWfa/yH0f/j+47N2tKRHI7PX4h7ToMLQXtxC3IBw2RKHJNaSN09Z43nMOdklEN9TbUNoa1cGJw4nesFktTuqqXZojgheyCGydVWnF3XriZa30K3zvcsrM/xYZIR9i2LxwPhCvSeUBSbCvOc6/bDkzPShSMyJhb7dR2XlS6oabpcz2UTehDQEjxFrurxw+PNspZhH+sksxniAASynn4sEf8uviMHx1qTMw8YKJIq69kOwM+2uEM5hd7FVWTV1UwG+bgeFo3bou3JOrQly1+3bYGk/iikFQ5qAF9QucCwJ5TzHWrTfq8jfgUyVcO47C9xCRxhtHzGu8MtT7NpPg== Received: from BYAPR05CA0030.namprd05.prod.outlook.com (2603:10b6:a03:c0::43) by DM6PR12MB4105.namprd12.prod.outlook.com (2603:10b6:5:217::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 08:15:12 +0000 Received: from CO1PEPF000044EF.namprd05.prod.outlook.com (2603:10b6:a03:c0:cafe::6c) by BYAPR05CA0030.outlook.office365.com (2603:10b6:a03:c0::43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.18 via Frontend Transport; Wed, 2 Aug 2023 08:15:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CO1PEPF000044EF.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:12 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 01:14:58 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 01:14:57 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 01:14:54 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Paolo Bonzini , Joao Martins , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH for-8.2 v3 1/6] vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup() Date: Wed, 2 Aug 2023 11:14:44 +0300 Message-ID: <20230802081449.2528-2-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230802081449.2528-1-avihaih@nvidia.com> References: <20230802081449.2528-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044EF:EE_|DM6PR12MB4105:EE_ X-MS-Office365-Filtering-Correlation-Id: 3bbe9e71-7a6e-455e-6093-08db93309827 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qdzTIg1md3hf8LBbhkamzqlh3Q+MxBMs0/K9gc9+4dD5zv7+8RRbWjI/ZE7iKKGWOSUA3hyyN4GGrrjQ2V7XFIuwlpKPgaR/v3U6ZLxet4HfDxyiSmTgHXudsTrLcDzIBNziMbklC2UvvIk/DbtS1l40vTE2MzsOv0mfg9dYe/bmgYsqkqtt5dXfFw+fkSzbcvwNxEMl5L7PvCIHDhyG5vByD0FpjgNm7/qGUrOw7UFJyU9JMztCdKMIw1qRFok+Fw1z8NdHvuiPgV+l56+JLiXp4o4TenBS9LFDruz3FAi4H4CjG4ckweb3M+HzJhrdwBjcZelfoFC3+Ih/jg5RNSvCyHc9+stmDeGN3ezVX5ZxCK38GmiJVZZzn9I2NVZdkEz4pIb6xnWj1h5sqzNQYoBrf6wURY5jFKKPPnGt5A+oWywlGtlbOLMYxtTn+r74VG9p2xINBmehWe8Dyogscnt9zHLs6Zr0df7AgEE7Dqsx7QR+h7h2uVStqcxB8bkGRR0bTW8RFpS/1oUxIIVYWbjQZ+J1VRSZA3taqXGYTwmkI8HJfAZgKJR/PsDVg+mNRXZ8TYl87F8i31t9UYlJjlpk59cUcJ1MUDT407KPqoNzNCWuBrGa5s9pA6OTDk0ElMZwsvdAR60WdWftuJYi6xnfIULxka7csGymd9SwLyBqTxdMeQC3xzFNRAeFjPXKWMGj9/7ekldVFlXdalxubWUGD5zVPpAXGqy0uN/qZ0faddAGxAuFXr0OatYizxgQ X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(396003)(136003)(376002)(39860400002)(82310400008)(451199021)(36840700001)(46966006)(40470700004)(8936002)(8676002)(5660300002)(426003)(26005)(41300700001)(107886003)(36860700001)(2906002)(83380400001)(1076003)(47076005)(36756003)(40460700003)(336012)(2616005)(40480700001)(478600001)(54906003)(7636003)(316002)(86362001)(82740400003)(7696005)(356005)(70586007)(70206006)(6666004)(4326008)(6916009)(186003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 08:15:12.1925 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3bbe9e71-7a6e-455e-6093-08db93309827 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044EF.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4105 Received-SPF: softfail client-ip=2a01:111:f400:7eab::60c; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Changing the device state from STOP_COPY to STOP can take time as the device may need to free resources and do other operations as part of the transition. Currently, this is done in vfio_save_complete_precopy() and therefore it is counted in the migration downtime. To avoid this, change the device state from STOP_COPY to STOP in vfio_save_cleanup(), which is called after migration has completed and thus is not part of migration downtime. Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 2674f4bc47..8acd182a8b 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -383,6 +383,19 @@ static void vfio_save_cleanup(void *opaque) VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; + /* + * Changing device state from STOP_COPY to STOP can take time. Do it here, + * after migration has completed, so it won't increase downtime. + */ + if (migration->device_state == VFIO_DEVICE_STATE_STOP_COPY) { + /* + * If setting the device in STOP state fails, the device should be + * reset. To do so, use ERROR state as a recover state. + */ + vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, + VFIO_DEVICE_STATE_ERROR); + } + g_free(migration->data_buffer); migration->data_buffer = NULL; migration->precopy_init_size = 0; @@ -508,12 +521,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } - /* - * If setting the device in STOP state fails, the device should be reset. - * To do so, use ERROR state as a recover state. - */ - ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, - VFIO_DEVICE_STATE_ERROR); trace_vfio_save_complete_precopy(vbasedev->name, ret); return ret; From patchwork Wed Aug 2 08:14:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13337836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8CF8AC41513 for ; Wed, 2 Aug 2023 08:16:01 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qR70n-0006Tm-I9; Wed, 02 Aug 2023 04:15:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70m-0006QK-0q for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:16 -0400 Received: from mail-mw2nam12on20607.outbound.protection.outlook.com ([2a01:111:f400:fe5a::607] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70j-0006OD-NW for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:15 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IaJkNIllZTAHRdFy28ZEGI9+v3KOTpSdog2QcobRvG6sHRjBNcjvDCzyCF/aidMqge39Q8GmyGuOlRbSc5YRrT7sj4B8XlKBXFwjT9B0ERvU3qk5My8l8ZFL0HuLWatexcULxGzvoSEsZIlNfHCvutyYlDiocmSpnlnysgp6cNxULbl2H7DVMYvS91RZVJKGc1ArDh5z81rvjJk+23GRdwx7j5z0CMAFvGSgwkVTRdlDQAIVAfuTqTtItXRleMidKOvxAREqdHYYgvYOzYDZfe2JvUJGUMSYvCVZUr/PwFTEo2AZk+Xydawc5CceCaB2zOQYwSFblD4rKXLy7SsdwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+Z8KK1OuLXNINh93/PeHpPx0VHhqjtEVVX6bt0/1/gA=; b=iL9r5lvTTj+2G5U6VDwAy568Z0ZU2AiBYWcdJ+X871xrmpyN/fWXJwbDgnhUVlDNRZtEhOrSaiTuYSDdHNlSyIZGAZ8fz9MsiBLC5l+2ZOmW4hHj00lPy/AGRxXLuGdB9r2LTWtXStuQZyAIp1wfX+uQNaJB8v+Q37fgorTg+QU2TODHk2LTv2Xb5MbRO+ORUOajsVaT+wW07p/wzXAxnctQGz5mVtNpk6/8VV5wzgizhMbZ0WStTqCjZPPkPbh/s/p6DZ2dc3R45WcAyV/2rIynbHz4JGGWjTa63Nouur3R0m1dKGyJJf47jbvfy0BYYV29sYRjy+pC2wKUeCzu9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Z8KK1OuLXNINh93/PeHpPx0VHhqjtEVVX6bt0/1/gA=; b=Fi53GgsqcmSJmQYS8+QSwtokyg9S1sN7q+zT8cV2xZ0/7RHz4BQFa0/D7aNl4UT34upZSJqCJzQKcyvTPUS9rbG7tRzFSxgfhdPDY1N1i6Rt1/Djeu0s+zWhoXbSyf1qfwY5dvzIbp2Kdm5+pOQVm4QYUig1bVL9PkRc5sUG5o2kKjGH2ePP+r3d15mnMQJiQ4EqiuLIUHT34VSbGgHooDQ5Ei0eLEqwGYAHpauvAaixnbVrsL0Ch5Z/+NIXMrBgmrF9MbsncFVpy7UNKu3bOuXmW7Ihv6QEFso7fLKxdI6sLkctbuBsW7DqCDanrMFSGAL4TWN6s/wTFsl/WRq6Fw== Received: from CY5PR22CA0071.namprd22.prod.outlook.com (2603:10b6:930:80::16) by SA1PR12MB6893.namprd12.prod.outlook.com (2603:10b6:806:24c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 08:15:09 +0000 Received: from CY4PEPF0000E9D4.namprd03.prod.outlook.com (2603:10b6:930:80:cafe::fb) by CY5PR22CA0071.outlook.office365.com (2603:10b6:930:80::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45 via Frontend Transport; Wed, 2 Aug 2023 08:15:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000E9D4.mail.protection.outlook.com (10.167.241.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:09 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 01:15:01 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 01:15:01 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 01:14:58 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Paolo Bonzini , Joao Martins , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH for-8.2 v3 2/6] sysemu: Add prepare callback to struct VMChangeStateEntry Date: Wed, 2 Aug 2023 11:14:45 +0300 Message-ID: <20230802081449.2528-3-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230802081449.2528-1-avihaih@nvidia.com> References: <20230802081449.2528-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D4:EE_|SA1PR12MB6893:EE_ X-MS-Office365-Filtering-Correlation-Id: 53bbc9cd-89b8-4e88-ec9a-08db9330966a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 076ZI2GL2F4C44C5KBZUnTletJ2mYzw1pzniJokX5il5QCCtd0t1ePW5ZgmixouPwKH1Chl076h+6FQmcC2osDCYGgeMvVbd3kw9R85ahqxE8CbiDWuIeSURv/6aJKLk88JdqnyqYqHdNUl1x81bSzkxf2DgJtkyOSiT4V4u4vJ8BkdDrVMN+xiyymXbJMjXlaMTNCuxhxyRrtzsmj1NvnbhYAaz8weKKXTvGG5//w2zmH3Mqc5XZWiyAWWBY+/qZpmIHw/8k7/yVv+e2bA8893I+8U15974kYW87eLDSMdKiY7UyXVrM45dFigoZW2W+Mh8/iVgSalac4xpKoTlW1Z/EikmppIR/brdNxKxvt941mCy7nT3N5RI8EUfQh/QxRILl2+1jbomY+YOtRU7ztf6x4PvsDhCgvLYFARaSLJzxyQOmhrENM2NsWIf/pt48t1SvuH/AkvGc/clplTn5P+09+e4n9xObfBRv0OxxisSyihiBBijFddccfSU3bUJfDzK9V5IZEQSC8r8IKJ/Wxp1A5BZKpKwuK9cEoR2IFlnCRuGuWjTZ+cAj2I3MLxAgNGgpQ0zKv3vIyUED9uftDELLzyoOzAE26xh/TT5GNhkj7MIQzg5OuvYO9Sy4RZtUD8iqfvK2L18tosqF3CLiIaybKtAKmyLEAOcaULLfPgt/erB8H9dOIb/LwN4l6E3vSpCpxNfL4zGEWdpnfPQhUvl5wBlOY3098HKF7w16WVLO4NLk2m+YGBGC9J5YwQqBTAUMuUPfEf8LaSJ+KjckA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(136003)(346002)(376002)(39860400002)(82310400008)(451199021)(36840700001)(46966006)(40470700004)(70206006)(70586007)(86362001)(6916009)(6666004)(7696005)(2906002)(107886003)(4326008)(54906003)(40480700001)(36756003)(40460700003)(478600001)(83380400001)(82740400003)(66574015)(426003)(41300700001)(47076005)(2616005)(7636003)(356005)(5660300002)(36860700001)(8936002)(8676002)(26005)(1076003)(316002)(186003)(336012)(145603002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 08:15:09.3422 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 53bbc9cd-89b8-4e88-ec9a-08db9330966a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6893 Received-SPF: softfail client-ip=2a01:111:f400:fe5a::607; envelope-from=avihaih@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add prepare callback to struct VMChangeStateEntry. The prepare callback is optional and can be set by the new function qemu_add_vm_change_state_handler_prio_full() that allows setting this callback in addition to the main callback. The prepare callbacks and main callbacks are called in two separate phases: First all prepare callbacks are called and only then all main callbacks are called. The purpose of the new prepare callback is to allow all devices to run a preliminary task before calling the devices' main callbacks. This will facilitate adding P2P support for VFIO migration where all VFIO devices need to be put in an intermediate P2P quiescent state before being stopped or started by the main callback. Signed-off-by: Avihai Horon Reviewed-by: Cédric Le Goater --- include/sysemu/runstate.h | 4 ++++ softmmu/runstate.c | 40 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/include/sysemu/runstate.h b/include/sysemu/runstate.h index 7beb29c2e2..764a0fc6a4 100644 --- a/include/sysemu/runstate.h +++ b/include/sysemu/runstate.h @@ -16,6 +16,10 @@ VMChangeStateEntry *qemu_add_vm_change_state_handler(VMChangeStateHandler *cb, void *opaque); VMChangeStateEntry *qemu_add_vm_change_state_handler_prio( VMChangeStateHandler *cb, void *opaque, int priority); +VMChangeStateEntry * +qemu_add_vm_change_state_handler_prio_full(VMChangeStateHandler *cb, + VMChangeStateHandler *prepare_cb, + void *opaque, int priority); VMChangeStateEntry *qdev_add_vm_change_state_handler(DeviceState *dev, VMChangeStateHandler *cb, void *opaque); diff --git a/softmmu/runstate.c b/softmmu/runstate.c index f3bd862818..1652ed0439 100644 --- a/softmmu/runstate.c +++ b/softmmu/runstate.c @@ -271,6 +271,7 @@ void qemu_system_vmstop_request(RunState state) } struct VMChangeStateEntry { VMChangeStateHandler *cb; + VMChangeStateHandler *prepare_cb; void *opaque; QTAILQ_ENTRY(VMChangeStateEntry) entries; int priority; @@ -293,12 +294,39 @@ static QTAILQ_HEAD(, VMChangeStateEntry) vm_change_state_head = */ VMChangeStateEntry *qemu_add_vm_change_state_handler_prio( VMChangeStateHandler *cb, void *opaque, int priority) +{ + return qemu_add_vm_change_state_handler_prio_full(cb, NULL, opaque, + priority); +} + +/** + * qemu_add_vm_change_state_handler_prio_full: + * @cb: the main callback to invoke + * @prepare_cb: a callback to invoke before the main callback + * @opaque: user data passed to the callbacks + * @priority: low priorities execute first when the vm runs and the reverse is + * true when the vm stops + * + * Register a main callback function and an optional prepare callback function + * that are invoked when the vm starts or stops running. The main callback and + * the prepare callback are called in two separate phases: First all prepare + * callbacks are called and only then all main callbacks are called. As its + * name suggests, the prepare callback can be used to do some preparatory work + * before invoking the main callback. + * + * Returns: an entry to be freed using qemu_del_vm_change_state_handler() + */ +VMChangeStateEntry * +qemu_add_vm_change_state_handler_prio_full(VMChangeStateHandler *cb, + VMChangeStateHandler *prepare_cb, + void *opaque, int priority) { VMChangeStateEntry *e; VMChangeStateEntry *other; e = g_malloc0(sizeof(*e)); e->cb = cb; + e->prepare_cb = prepare_cb; e->opaque = opaque; e->priority = priority; @@ -333,10 +361,22 @@ void vm_state_notify(bool running, RunState state) trace_vm_state_notify(running, state, RunState_str(state)); if (running) { + QTAILQ_FOREACH_SAFE(e, &vm_change_state_head, entries, next) { + if (e->prepare_cb) { + e->prepare_cb(e->opaque, running, state); + } + } + QTAILQ_FOREACH_SAFE(e, &vm_change_state_head, entries, next) { e->cb(e->opaque, running, state); } } else { + QTAILQ_FOREACH_REVERSE_SAFE(e, &vm_change_state_head, entries, next) { + if (e->prepare_cb) { + e->prepare_cb(e->opaque, running, state); + } + } + QTAILQ_FOREACH_REVERSE_SAFE(e, &vm_change_state_head, entries, next) { e->cb(e->opaque, running, state); } From patchwork Wed Aug 2 08:14:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13337838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47F85C00528 for ; Wed, 2 Aug 2023 08:16:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qR70y-0006xR-2j; Wed, 02 Aug 2023 04:15:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70w-0006sr-1M for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:26 -0400 Received: from mail-bn8nam11on20601.outbound.protection.outlook.com ([2a01:111:f400:7eae::601] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70u-0006QD-7y for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:25 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FIDM4vCf4eypeubZ0pCemDC384CbBQr4ZT8jSUOCeaORN69sX7/eR28oemYcafEq/1dz9tHkTEbaakmjlDs7k3rxpo2tchOUch2SfpRrPSD4YCZFKIPYqczwmLkBsSziwSqby/sJu0DAAzEFuITbzMFnJXFpa9DaKy/6iRFHsWKiE9DVPM2u6e2iu8qEJYQaV5/IxaX05BfVA3jKTtyHWGhRSd9piNwoGYoHfvqlyg328G4uqffJKHU18Kbvj+6eAnt3wZMth8E5dJsumZ7fzQSZ5o72vk6tpfcX7zpe8/mUnXzlah5EF09RrFEuQA/J2SesA6UJ5uWGeEpoi07HZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WoQMIf0erWd9Xi61xbdWyd5xQ2K+Na/DWwzAYf3pLJA=; b=jjEblKXZRAoetn5wlEiRzhuDvi351m12kHfKBRmHlflqKIhc/1wnznJenazjqnG1a5DY9uR4R69cul3Sa6YwzbJ+gnCFCXaA3t53IqNKGO+66KrknVcMX5dByTUb6qNGiiEynOf/IN0BnD18I1P0hfaOMdUKzVR4330c2KEHg6hAe5Ml8W1wlqhcsY/APIA9kFD0CqjpEx+fGDi4xK5K0uNj+M1L4x/I8sKnYsbT46+BTSRuPlRrezR5XwFYi4v3yKJBSIrcVFVRFzX5ycuVTVJbt/3z6X9ncb/UOTZJi+GnzIZqEYyzgXRJOcpEIl3gIBS2l4krfpHyxTHx1nfjow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WoQMIf0erWd9Xi61xbdWyd5xQ2K+Na/DWwzAYf3pLJA=; b=ZSLBsVY0WrKS3fAPbxO+4NlIKRj6uBR0fYIE3B7q6LCd+O5SHyD26T/2n7O/u13HODKeTcFEVG+etO26LAn/9x4UPfkz1qFfj4hUsc0T+kdrC31JaRa17E47QN1XqcaEyx6gfQCOPyhiKcgA/IgQvjX0q4k+dZyfanYKu49LoUgbwpYnFgqvX5n5ohTr/TAP9m2LMaaQwcOQbKG2+qmd/v1f66bifF4iVqsq2+hbXbr0qBCbfG7TCWD18XHkPoSWHBI0lElxSoWHCAO+RqsHul7Q/n+ME4qdxrPDm+IOQDgQ4wyEkIP+Gdlo/73I0h2RZ2+8+p1GXgRoO4zmgS1sXg== Received: from MW4PR03CA0306.namprd03.prod.outlook.com (2603:10b6:303:dd::11) by MW5PR12MB5683.namprd12.prod.outlook.com (2603:10b6:303:1a0::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 08:15:19 +0000 Received: from CO1PEPF000044F1.namprd05.prod.outlook.com (2603:10b6:303:dd:cafe::3b) by MW4PR03CA0306.outlook.office365.com (2603:10b6:303:dd::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45 via Frontend Transport; Wed, 2 Aug 2023 08:15:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CO1PEPF000044F1.mail.protection.outlook.com (10.167.241.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:18 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 01:15:04 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 01:15:04 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 01:15:01 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Paolo Bonzini , Joao Martins , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH for-8.2 v3 3/6] qdev: Add qdev_add_vm_change_state_handler_full() Date: Wed, 2 Aug 2023 11:14:46 +0300 Message-ID: <20230802081449.2528-4-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230802081449.2528-1-avihaih@nvidia.com> References: <20230802081449.2528-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F1:EE_|MW5PR12MB5683:EE_ X-MS-Office365-Filtering-Correlation-Id: b35e5df8-d470-44d1-a10f-08db93309c15 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZBOWZxy1uQDeA0cDKCD9tv9dmdArB+F9gDWOA/USXCOk0W2xliaBXOajzPZ1PXQGgybKtUS+9+8877lhm+RcsKU7jCO+kkncVNDQ+Yk3v3Bm+YHNgbagJ27mKf9DJm5ZX++SaJ8RRYl/5/I8gMLCe0T2xju+kqrVPsmBsrZQOwh53n6jMsPbPzEVS2GkpIQq5OmI0rOqzdPNJdaua8W3/bhdMNDp7Max107QXMQ/BdItmB4AlAtcYMoVR7l3ts1E4Lc6aL+1195bqZiIpP5WTSdLC2RRbgq+OK/4DNVfcFhlC5HYHiR/2woRrpLkzTzhhnYkobQQFcbvqHsXFhAHyWPgexWUjw/tqrkukofJAZTYHTojCyTru35mxTviFRf4MfErT0Vvuk4gAQ0CiJiAUWwXoP2w9qiXezxiHCWUHMW/RBKK0rrxgacpIZwY0mFyMukoQwCmVT8Ec8uqmHqQUUURH69IlkvJAYLE/Zi4sSYjeiHMiNEcigp5T3hG9vtAW/Xi75hTBPrbrCQyr3/imwAKgrGB9iQre9L/ioYh51yEtmdU9qy1aLgsP5W2+MHPayT0vn74dorA54Oe1/uCekGSRDCSplUn9g2Y+NEA5DdOxfIgdqUQ1yBpAwu8GIe2C4YQbiq1HdmOHB8iaJXiqhm92/1yb0MZABiIn5T3hmckTbr2147iqDcxWe//4Idy6RTF3Kd6XcRG2Z3Q8vz5DoJIJvhe2q2qNUps2KDx4QmIFf1mNy/ocI4HeJwcyucw X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(396003)(136003)(376002)(39860400002)(82310400008)(451199021)(40470700004)(46966006)(36840700001)(66574015)(8936002)(8676002)(426003)(47076005)(41300700001)(107886003)(2906002)(5660300002)(83380400001)(36860700001)(40460700003)(2616005)(40480700001)(336012)(186003)(86362001)(7636003)(478600001)(54906003)(316002)(356005)(26005)(1076003)(7696005)(70586007)(70206006)(6666004)(82740400003)(4326008)(6916009)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 08:15:18.7714 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b35e5df8-d470-44d1-a10f-08db93309c15 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F1.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR12MB5683 Received-SPF: softfail client-ip=2a01:111:f400:7eae::601; envelope-from=avihaih@nvidia.com; helo=NAM11-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add qdev_add_vm_change_state_handler_full() variant that allows setting a prepare callback in addition to the main callback. This will facilitate adding P2P support for VFIO migration in the following patches. Signed-off-by: Avihai Horon Signed-off-by: Joao Martins Reviewed-by: Cédric Le Goater --- include/sysemu/runstate.h | 3 +++ hw/core/vm-change-state-handler.c | 14 +++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/include/sysemu/runstate.h b/include/sysemu/runstate.h index 764a0fc6a4..08afb97695 100644 --- a/include/sysemu/runstate.h +++ b/include/sysemu/runstate.h @@ -23,6 +23,9 @@ qemu_add_vm_change_state_handler_prio_full(VMChangeStateHandler *cb, VMChangeStateEntry *qdev_add_vm_change_state_handler(DeviceState *dev, VMChangeStateHandler *cb, void *opaque); +VMChangeStateEntry *qdev_add_vm_change_state_handler_full( + DeviceState *dev, VMChangeStateHandler *cb, + VMChangeStateHandler *prepare_cb, void *opaque); void qemu_del_vm_change_state_handler(VMChangeStateEntry *e); /** * vm_state_notify: Notify the state of the VM diff --git a/hw/core/vm-change-state-handler.c b/hw/core/vm-change-state-handler.c index 1f3630986d..8e2639224e 100644 --- a/hw/core/vm-change-state-handler.c +++ b/hw/core/vm-change-state-handler.c @@ -55,8 +55,20 @@ static int qdev_get_dev_tree_depth(DeviceState *dev) VMChangeStateEntry *qdev_add_vm_change_state_handler(DeviceState *dev, VMChangeStateHandler *cb, void *opaque) +{ + return qdev_add_vm_change_state_handler_full(dev, cb, NULL, opaque); +} + +/* + * Exactly like qdev_add_vm_change_state_handler() but passes a prepare_cb + * argument too. + */ +VMChangeStateEntry *qdev_add_vm_change_state_handler_full( + DeviceState *dev, VMChangeStateHandler *cb, + VMChangeStateHandler *prepare_cb, void *opaque) { int depth = qdev_get_dev_tree_depth(dev); - return qemu_add_vm_change_state_handler_prio(cb, opaque, depth); + return qemu_add_vm_change_state_handler_prio_full(cb, prepare_cb, opaque, + depth); } From patchwork Wed Aug 2 08:14:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13337845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7DB1C001DF for ; Wed, 2 Aug 2023 08:16:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qR711-0006y1-Ab; Wed, 02 Aug 2023 04:15:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70z-0006xd-Og for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:29 -0400 Received: from mail-sn1nam02on2060d.outbound.protection.outlook.com ([2a01:111:f400:7ea9::60d] helo=NAM02-SN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70x-0006QX-Sa for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:29 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EdPsH6NXGId4ePXvpVDaMdzqNxcS3HRZznqR8i/rrv2bxHJOdNoDu+5+jD54P1OxTCO5kTNr3KUD4EKdo2roLHuj0zphy74IDnmhYtCAFOUeP0TdzlZQhO4wsDCKaqrgc5LeB4G9aGSkeVX5sJB8cvUjZl+85u5u1dE2EiVN3nKscyG5uf2pfiAIcmRSD/ZlHRjs88oqhrYNnT6fsPNdjvPeQNIAOPzkfrWDDeYT6bOjQw9gEbw6/VZ/nEENhNrHrJTSwxnb5i+YO1Hp0nE+SGFBcYWZIu+3bJfKPcFffLc/MkAuqjK6CUqZoPu7Kf2cjUC1CuDWNR8ehtCutJwCvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WKp/pmecsHMhRlRbJZqM6zbZwX079spotXBKYXQmioA=; b=GN2JVHasSuTJVhTwuIIF4MbX1W3ubLC7xUuiw+qS3rqmiV1STpuKEvqWddnHZktEDk2G5/AEtSXOT+5mxEuLcVbtWYQQBu8qcvo/rM0Rb8DBRXZ1R0PX/ICRd6gJbpppI5M/YSy14eHX6jp4eCqYlvokRWhGGSeQHUoagbsRkXqKeXmAi3gYaQmH+LFeUEhCwUxfn03/MyZ6G4zfU0glIwqWkteuv4D5ZZKyAXdnOKgVQ/sVvh/KG1THnK/87bZkvtqMiyNvuon9i5VQ5lCNk4gO0bNgmf4cZ/CjGuFWg5cMGE0hBO8r6FDtTteR4SYROAjN9fYPeHWDKsVTRzHU/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WKp/pmecsHMhRlRbJZqM6zbZwX079spotXBKYXQmioA=; b=eBAnBoLFl8K2eWQYAyWovlYYKV0jAKSMbGhRd9nQchDHO0MDmzH+SRTDXSdTAj4r5rhBQleV8RQUXGTE634nOIZsw0Y4EwZfSGIwBduUnHWEHH0qcFweKJy+LkXrMGLgckAo1r/6g0Dr8hv32Za7UIvb/P7d2xzwL7x9f2wNNix4bCBKTC+rX7ddcDnbgAGvXEyOj70NzNQeW+Ae/0z2Y09WyGwasambslRsDOi3xg1jpnC6l1n8rN4lIIaDCinpI4WCO0YbMuEzr57KyLIgtr1C0M4/1EYxE0GaJf0kHqbP1iWvPpctk7RC0PbWe00ylP2T2A0N7NOUw3wXnq4UWA== Received: from MW4PR04CA0074.namprd04.prod.outlook.com (2603:10b6:303:6b::19) by CY8PR12MB7292.namprd12.prod.outlook.com (2603:10b6:930:53::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 08:15:24 +0000 Received: from CO1PEPF000044F5.namprd05.prod.outlook.com (2603:10b6:303:6b:cafe::7e) by MW4PR04CA0074.outlook.office365.com (2603:10b6:303:6b::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.44 via Frontend Transport; Wed, 2 Aug 2023 08:15:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CO1PEPF000044F5.mail.protection.outlook.com (10.167.241.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:23 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 01:15:08 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 01:15:08 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 01:15:05 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Paolo Bonzini , Joao Martins , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH for-8.2 v3 4/6] vfio/migration: Refactor PRE_COPY and RUNNING state checks Date: Wed, 2 Aug 2023 11:14:47 +0300 Message-ID: <20230802081449.2528-5-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230802081449.2528-1-avihaih@nvidia.com> References: <20230802081449.2528-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F5:EE_|CY8PR12MB7292:EE_ X-MS-Office365-Filtering-Correlation-Id: e442ff78-958f-43b6-5a3c-08db93309eee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PLCi2JpuNPx4lhlxYoRbdAf/Wto+GBj2KIxUqDmqbiKYMFIIcpKLkprNDB8DzvQw16M8LFAttCJnWIqnsSl7imyiqAGg3yQbNas4wiHDFKRt0fHBDcyF6MjNmehYZWIFoTRdRwR+qNk+HyVOMU44aTdUdcvJ49x27y8sL4x88eW+xG7dEK1l7QUHV4DmwELjfjsa8JvBk6Uwe77W1pKPovA8Q7BAnYSiNUFQDi7DDS+mcBGdehRRntSPV8U8lFa1+XC8WrsTH/J9a6AltHN/ebyMEsukzbLlt3LbYv2JsOjwIGdUAmtGth9bKwxD0SWe3q8/HeD4+HpDcfJdZ2PDABTkqD5+H6c67DljrayR1hpTajmuik1d63y+Fv7GtnmdJSYQQFP7xrklsHN2Awnf5605NMghnCTUH1lFXS8PlsbrYvi4y7c7Dvxknl7CwvAW9Sga5pMRRPk7YMMC/nrd+dFkpjcodTYIBm9iB/2ieoMCHEr34D1RIJji22XKxUuiTVTIEn963QxGPFXwFbWHYExoB+9buynglCLxo4Kp5Yf8Cwbn3+ksY+/qKYlR6xNIFzqD5KiFSoHbJ1ZSf8HWz82iIswY/fyjYMdp1XkwBpkhd2StXxhYpvvInQs7kY28LXqdccld4uPQ7i0fIUurA7PWld2GU7PvoNpdJ6iYQIxFyoydkO3lZeOnbx4tJY3CP75l8vD6Rd/7nsg1avlz/+aMywS7HCy/qTxzTWOqPQE2bFY/VZWQnlfNtGEuvFrM X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(396003)(136003)(376002)(39860400002)(82310400008)(451199021)(40470700004)(46966006)(36840700001)(186003)(426003)(8936002)(66574015)(83380400001)(8676002)(26005)(41300700001)(107886003)(2906002)(5660300002)(36860700001)(47076005)(40460700003)(336012)(2616005)(40480700001)(478600001)(7636003)(86362001)(54906003)(316002)(1076003)(82740400003)(7696005)(70586007)(70206006)(356005)(6666004)(6916009)(4326008)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 08:15:23.5628 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e442ff78-958f-43b6-5a3c-08db93309eee X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F5.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7292 Received-SPF: softfail client-ip=2a01:111:f400:7ea9::60d; envelope-from=avihaih@nvidia.com; helo=NAM02-SN1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Joao Martins Move the PRE_COPY and RUNNING state checks to helper functions. This is in preparation for adding P2P VFIO migration support, where these helpers will also test for PRE_COPY_P2P and RUNNING_P2P states. Signed-off-by: Joao Martins Signed-off-by: Avihai Horon Reviewed-by: Cédric Le Goater --- include/hw/vfio/vfio-common.h | 2 ++ hw/vfio/common.c | 22 ++++++++++++++++++---- hw/vfio/migration.c | 10 ++++------ 3 files changed, 24 insertions(+), 10 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index da43d27352..e9b8954595 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -230,6 +230,8 @@ void vfio_unblock_multiple_devices_migration(void); bool vfio_viommu_preset(VFIODevice *vbasedev); int64_t vfio_mig_bytes_transferred(void); void vfio_reset_bytes_transferred(void); +bool vfio_device_state_is_running(VFIODevice *vbasedev); +bool vfio_device_state_is_precopy(VFIODevice *vbasedev); #ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index, diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 9aac21abb7..16cf79a76c 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -437,6 +437,20 @@ static void vfio_set_migration_error(int err) } } +bool vfio_device_state_is_running(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + return migration->device_state == VFIO_DEVICE_STATE_RUNNING; +} + +bool vfio_device_state_is_precopy(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + return migration->device_state == VFIO_DEVICE_STATE_PRE_COPY; +} + static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) { VFIOGroup *group; @@ -457,8 +471,8 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) } if (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF && - (migration->device_state == VFIO_DEVICE_STATE_RUNNING || - migration->device_state == VFIO_DEVICE_STATE_PRE_COPY)) { + (vfio_device_state_is_running(vbasedev) || + vfio_device_state_is_precopy(vbasedev))) { return false; } } @@ -503,8 +517,8 @@ static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) return false; } - if (migration->device_state == VFIO_DEVICE_STATE_RUNNING || - migration->device_state == VFIO_DEVICE_STATE_PRE_COPY) { + if (vfio_device_state_is_running(vbasedev) || + vfio_device_state_is_precopy(vbasedev)) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 8acd182a8b..48f9c23cbe 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -411,7 +411,7 @@ static void vfio_state_pending_estimate(void *opaque, uint64_t *must_precopy, VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - if (migration->device_state != VFIO_DEVICE_STATE_PRE_COPY) { + if (!vfio_device_state_is_precopy(vbasedev)) { return; } @@ -444,7 +444,7 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, vfio_query_stop_copy_size(vbasedev, &stop_copy_size); *must_precopy += stop_copy_size; - if (migration->device_state == VFIO_DEVICE_STATE_PRE_COPY) { + if (vfio_device_state_is_precopy(vbasedev)) { vfio_query_precopy_size(migration); *must_precopy += @@ -459,9 +459,8 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, static bool vfio_is_active_iterate(void *opaque) { VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - return migration->device_state == VFIO_DEVICE_STATE_PRE_COPY; + return vfio_device_state_is_precopy(vbasedev); } static int vfio_save_iterate(QEMUFile *f, void *opaque) @@ -656,7 +655,6 @@ static const SaveVMHandlers savevm_vfio_handlers = { static void vfio_vmstate_change(void *opaque, bool running, RunState state) { VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; enum vfio_device_mig_state new_state; int ret; @@ -664,7 +662,7 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) new_state = VFIO_DEVICE_STATE_RUNNING; } else { new_state = - (migration->device_state == VFIO_DEVICE_STATE_PRE_COPY && + (vfio_device_state_is_precopy(vbasedev) && (state == RUN_STATE_FINISH_MIGRATE || state == RUN_STATE_PAUSED)) ? VFIO_DEVICE_STATE_STOP_COPY : VFIO_DEVICE_STATE_STOP; From patchwork Wed Aug 2 08:14:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13337837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91F8DC001DF for ; Wed, 2 Aug 2023 08:16:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qR713-0006yf-Vi; Wed, 02 Aug 2023 04:15:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR711-0006y6-UU for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:31 -0400 Received: from mail-dm6nam04on20627.outbound.protection.outlook.com ([2a01:111:f400:7e8b::627] helo=NAM04-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR70z-0006Qu-Iw for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:31 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KBGkqg3YYKGrZsaJw8N8b2SzTkJCu2q5JQCjOopbERXn6hDSN0Ca4wi3+Gb67+C9eWERtQHnnaA97RgQaQ4C91eDMfZw9HnZa7cJqXLJo8NrjDfB7moPC61hbfYzolMMUmNWw35NRBW2kdnRyPLLZ7f1gglKmN7lEBVOtukj05vHlLRr0SSvfM9gKB8rF/Uaoigt+yTM9YpE92oJm4HaJQBd3vV6GF4Dz+/6rV2OlQ9e15xnEvIk4u4opLYmM8B7mC7x91Goco9Tvl34h+GkKEIRdwElX5tcG7tK32RYg7mv9by2qGcIOpFi/0pF+hviHFYu9jtkdZ5ZnvlRIlB90g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ft7Sg6rR10RiUEMIeD7KXwHGhgXN3kPK7nib4fEn7ME=; b=KFYtgX89y1gLQluJP2y2CCQ8e2yIBbpaJ5E0okVmki8VLAaSuiIzd4r+L7Cw8Y/HuHQGUvMj2kRSeuo/BexCaYwMdTskKjaB61827F6hvE2SDJco8nUQWIQJfywgfDnWrfDZsCZIHwV39Ajr6SqsACxgX8P09bzeE9YphVCOWDSZSV44JlHZ0h6U03C6T2ZPiE79eiCYj1LTye+222StzQ4a3zX8lEWNj6XOJ8yp44I9wxXzejncv1fpLmPysYQgEaaYio9yKbiVcYrSUw6uQ/szzu2JnAM+zTXxQzzHduLyS0tpHN146GFfTptlv+Dgb5MbUCy9YR+saBbLKdwk4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ft7Sg6rR10RiUEMIeD7KXwHGhgXN3kPK7nib4fEn7ME=; b=lcIRaBV+lrvJANJaNZmWuNJY3y2mJ07IkjuWcwM8wP0hs2nij7DkhtR35bXOnd1lfoWwbjrEoWhB9jv6b4QZfPFt3hzmdtvCFH85ywrJmw1rLwZ6WLMKe8Gp3cWghmL8Prp+YLuh3bevcHornXe9B+IAGncchSY+PGakxhM2JkBKoGFOwB/NosYh9omu4WLwNVTXJ7zSAjj78a7Si2lD1ZgDxsMKQ/Qg6UN6eK2rQ+xUMJmd6tqT/eZH12xDVwDXoFMdNMDdqpzb3HipXeY/tCU4iqumiPHcW4cxUYOy/QGjv/x5Y8VzCc08dhgcEGB5epr8l/zSDZFljWPjz1fkpw== Received: from CY5P221CA0062.NAMP221.PROD.OUTLOOK.COM (2603:10b6:930:4::34) by SA3PR12MB7902.namprd12.prod.outlook.com (2603:10b6:806:305::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 08:15:26 +0000 Received: from CY4PEPF0000E9CE.namprd03.prod.outlook.com (2603:10b6:930:4:cafe::b7) by CY5P221CA0062.outlook.office365.com (2603:10b6:930:4::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000E9CE.mail.protection.outlook.com (10.167.241.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:25 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 01:15:13 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 01:15:12 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 01:15:08 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Paolo Bonzini , Joao Martins , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH for-8.2 v3 5/6] vfio/migration: Add P2P support for VFIO migration Date: Wed, 2 Aug 2023 11:14:48 +0300 Message-ID: <20230802081449.2528-6-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230802081449.2528-1-avihaih@nvidia.com> References: <20230802081449.2528-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9CE:EE_|SA3PR12MB7902:EE_ X-MS-Office365-Filtering-Correlation-Id: e6808c38-85b8-4455-612a-08db9330a051 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Rr8AclJK4p3/47tQt9kr2aVTlRch+ecFaKRj8w0+tAAxRsHTzhu7g1vMvH8/lkWnT7RwbykO7h6XONjRwmldJmCdaEvtXFB7LM2FgMYaRumPT2ZdDvhGLOzDeICBbgShFzFWWoIx8Lthhi8Q+s7p3Pa++G+7bNj1hQQM/HDpKGb9tIWhd9Gtr2q3xe6KfxtNRNv+eEy0EqdDsmNncb/SQ7Hms84ivSdGzBVmlISbhpjtMcHeXQcgjzwqdz3X38iMPAET3Lv+3ahcmuMi69BDVfw2q+gjQASkn2B4v7Uu5pyCi8eFYCAuh4pcCucN9f1yC51Rc3QoDSUvbeDX1vtLNzKt6M1fv+nB0x2DUeuZmgjBXS59KBkHzufz8G8icsZsfkgIlCvB01EljsKO8pI/ZyYiIckK5MtmrBckMge4VyYz7nKYXWbpsD+BtnrVlMTanPl7t5Ulx+3LVnb2n5YWlxZfZbSN1pa8aWF5PxkmkA4bKbzPZW86sVHf1zLFYMk32SOv5UEjfdWdGlQeVD5kFIcsR4hOnD4nFsKj2YWs0TwgnIuTSp2cu/9G8/TVHNWIzY8Gtom9Ihoh+9LkljcZrqjY4BBAD8Mz7n2vlM7+7fyiriT0+MUlNZsu72nnNkQ287TMbhJJv/xvIVV4ZNu3mXCz2wwQV2zFYlX3J5sT3bBY1TgqxjfQA34N9MGG/EehZwuk5+6+O6hW07FNyybly4O7Pw2Lh/0jypzTSmgNLSTmjQiY8JDLn2hq2A1QcXyM X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(396003)(136003)(376002)(39860400002)(82310400008)(451199021)(40470700004)(46966006)(36840700001)(8936002)(8676002)(426003)(47076005)(30864003)(41300700001)(107886003)(2906002)(5660300002)(83380400001)(36860700001)(40460700003)(2616005)(40480700001)(336012)(186003)(86362001)(7636003)(478600001)(54906003)(316002)(356005)(26005)(1076003)(7696005)(70586007)(70206006)(6666004)(82740400003)(4326008)(6916009)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 08:15:25.9520 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e6808c38-85b8-4455-612a-08db9330a051 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9CE.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7902 Received-SPF: softfail client-ip=2a01:111:f400:7e8b::627; envelope-from=avihaih@nvidia.com; helo=NAM04-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org VFIO migration uAPI defines an optional intermediate P2P quiescent state. While in the P2P quiescent state, P2P DMA transactions cannot be initiated by the device, but the device can respond to incoming ones. Additionally, all outstanding P2P transactions are guaranteed to have been completed by the time the device enters this state. The purpose of this state is to support migration of multiple devices that might do P2P transactions between themselves. Add support for P2P migration by transitioning all the devices to the P2P quiescent state before stopping or starting the devices. Use the new VMChangeStateHandler prepare_cb to achieve that behavior. This will allow migration of multiple VFIO devices if all of them support P2P migration. Signed-off-by: Avihai Horon Reviewed-by: Cédric Le Goater Tested-by: YangHang Liu --- docs/devel/vfio-migration.rst | 93 +++++++++++++++++++++-------------- hw/vfio/common.c | 6 ++- hw/vfio/migration.c | 46 +++++++++++++++-- hw/vfio/trace-events | 1 + 4 files changed, 105 insertions(+), 41 deletions(-) diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst index b433cb5bb2..605fe60e96 100644 --- a/docs/devel/vfio-migration.rst +++ b/docs/devel/vfio-migration.rst @@ -23,9 +23,21 @@ and recommends that the initial bytes are sent and loaded in the destination before stopping the source VM. Enabling this migration capability will guarantee that and thus, can potentially reduce downtime even further. -Note that currently VFIO migration is supported only for a single device. This -is due to VFIO migration's lack of P2P support. However, P2P support is planned -to be added later on. +To support migration of multiple devices that might do P2P transactions between +themselves, VFIO migration uAPI defines an intermediate P2P quiescent state. +While in the P2P quiescent state, P2P DMA transactions cannot be initiated by +the device, but the device can respond to incoming ones. Additionally, all +outstanding P2P transactions are guaranteed to have been completed by the time +the device enters this state. + +All the devices that support P2P migration are first transitioned to the P2P +quiescent state and only then are they stopped or started. This makes migration +safe P2P-wise, since starting and stopping the devices is not done atomically +for all the devices together. + +Thus, multiple VFIO devices migration is allowed only if all the devices +support P2P migration. Single VFIO device migration is allowed regardless of +P2P migration support. A detailed description of the UAPI for VFIO device migration can be found in the comment for the ``vfio_device_mig_state`` structure in the header file @@ -132,54 +144,63 @@ will be blocked. Flow of state changes during Live migration =========================================== -Below is the flow of state change during live migration. +Below is the state change flow during live migration for a VFIO device that +supports both precopy and P2P migration. The flow for devices that don't +support it is similar, except that the relevant states for precopy and P2P are +skipped. The values in the parentheses represent the VM state, the migration state, and the VFIO device state, respectively. -The text in the square brackets represents the flow if the VFIO device supports -pre-copy. Live migration save path ------------------------ :: - QEMU normal running state - (RUNNING, _NONE, _RUNNING) - | + QEMU normal running state + (RUNNING, _NONE, _RUNNING) + | migrate_init spawns migration_thread - Migration thread then calls each device's .save_setup() - (RUNNING, _SETUP, _RUNNING [_PRE_COPY]) - | - (RUNNING, _ACTIVE, _RUNNING [_PRE_COPY]) - If device is active, get pending_bytes by .state_pending_{estimate,exact}() - If total pending_bytes >= threshold_size, call .save_live_iterate() - [Data of VFIO device for pre-copy phase is copied] - Iterate till total pending bytes converge and are less than threshold - | - On migration completion, vCPU stops and calls .save_live_complete_precopy for - each active device. The VFIO device is then transitioned into _STOP_COPY state - (FINISH_MIGRATE, _DEVICE, _STOP_COPY) - | - For the VFIO device, iterate in .save_live_complete_precopy until - pending data is 0 - (FINISH_MIGRATE, _DEVICE, _STOP) - | - (FINISH_MIGRATE, _COMPLETED, _STOP) - Migraton thread schedules cleanup bottom half and exits + Migration thread then calls each device's .save_setup() + (RUNNING, _SETUP, _PRE_COPY) + | + (RUNNING, _ACTIVE, _PRE_COPY) + If device is active, get pending_bytes by .state_pending_{estimate,exact}() + If total pending_bytes >= threshold_size, call .save_live_iterate() + Data of VFIO device for pre-copy phase is copied + Iterate till total pending bytes converge and are less than threshold + | + On migration completion, the vCPUs and the VFIO device are stopped + The VFIO device is first put in P2P quiescent state + (FINISH_MIGRATE, _ACTIVE, _PRE_COPY_P2P) + | + Then the VFIO device is put in _STOP_COPY state + (FINISH_MIGRATE, _ACTIVE, _STOP_COPY) + .save_live_complete_precopy() is called for each active device + For the VFIO device, iterate in .save_live_complete_precopy() until + pending data is 0 + | + (POSTMIGRATE, _COMPLETED, _STOP_COPY) + Migraton thread schedules cleanup bottom half and exits + | + .save_cleanup() is called + (POSTMIGRATE, _COMPLETED, _STOP) Live migration resume path -------------------------- :: - Incoming migration calls .load_setup for each device - (RESTORE_VM, _ACTIVE, _STOP) - | - For each device, .load_state is called for that device section data - (RESTORE_VM, _ACTIVE, _RESUMING) - | - At the end, .load_cleanup is called for each device and vCPUs are started - (RUNNING, _NONE, _RUNNING) + Incoming migration calls .load_setup() for each device + (RESTORE_VM, _ACTIVE, _STOP) + | + For each device, .load_state() is called for that device section data + (RESTORE_VM, _ACTIVE, _RESUMING) + | + At the end, .load_cleanup() is called for each device and vCPUs are started + The VFIO device is first put in P2P quiescent state + (RUNNING, _ACTIVE, _RUNNING_P2P) + | + (RUNNING, _NONE, _RUNNING) Postcopy ======== diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 16cf79a76c..7c3d636025 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -441,14 +441,16 @@ bool vfio_device_state_is_running(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - return migration->device_state == VFIO_DEVICE_STATE_RUNNING; + return migration->device_state == VFIO_DEVICE_STATE_RUNNING || + migration->device_state == VFIO_DEVICE_STATE_RUNNING_P2P; } bool vfio_device_state_is_precopy(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - return migration->device_state == VFIO_DEVICE_STATE_PRE_COPY; + return migration->device_state == VFIO_DEVICE_STATE_PRE_COPY || + migration->device_state == VFIO_DEVICE_STATE_PRE_COPY_P2P; } static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 48f9c23cbe..71855468fe 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -71,8 +71,12 @@ static const char *mig_state_to_str(enum vfio_device_mig_state state) return "STOP_COPY"; case VFIO_DEVICE_STATE_RESUMING: return "RESUMING"; + case VFIO_DEVICE_STATE_RUNNING_P2P: + return "RUNNING_P2P"; case VFIO_DEVICE_STATE_PRE_COPY: return "PRE_COPY"; + case VFIO_DEVICE_STATE_PRE_COPY_P2P: + return "PRE_COPY_P2P"; default: return "UNKNOWN STATE"; } @@ -652,6 +656,39 @@ static const SaveVMHandlers savevm_vfio_handlers = { /* ---------------------------------------------------------------------- */ +static void vfio_vmstate_change_prepare(void *opaque, bool running, + RunState state) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + enum vfio_device_mig_state new_state; + int ret; + + new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? + VFIO_DEVICE_STATE_PRE_COPY_P2P : + VFIO_DEVICE_STATE_RUNNING_P2P; + + /* + * If setting the device in new_state fails, the device should be reset. + * To do so, use ERROR state as a recover state. + */ + ret = vfio_migration_set_state(vbasedev, new_state, + VFIO_DEVICE_STATE_ERROR); + if (ret) { + /* + * Migration should be aborted in this case, but vm_state_notify() + * currently does not support reporting failures. + */ + if (migrate_get_current()->to_dst_file) { + qemu_file_set_error(migrate_get_current()->to_dst_file, ret); + } + } + + trace_vfio_vmstate_change_prepare(vbasedev->name, running, + RunState_str(state), + mig_state_to_str(new_state)); +} + static void vfio_vmstate_change(void *opaque, bool running, RunState state) { VFIODevice *vbasedev = opaque; @@ -758,6 +795,7 @@ static int vfio_migration_init(VFIODevice *vbasedev) char id[256] = ""; g_autofree char *path = NULL, *oid = NULL; uint64_t mig_flags = 0; + VMChangeStateHandler *prepare_cb; if (!vbasedev->ops->vfio_get_object) { return -EINVAL; @@ -798,9 +836,11 @@ static int vfio_migration_init(VFIODevice *vbasedev) register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_handlers, vbasedev); - migration->vm_state = qdev_add_vm_change_state_handler(vbasedev->dev, - vfio_vmstate_change, - vbasedev); + prepare_cb = migration->mig_flags & VFIO_MIGRATION_P2P ? + vfio_vmstate_change_prepare : + NULL; + migration->vm_state = qdev_add_vm_change_state_handler_full( + vbasedev->dev, vfio_vmstate_change, prepare_cb, vbasedev); migration->migration_state.notify = vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index ee7509e68e..329736a738 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -167,3 +167,4 @@ vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data buffer vfio_state_pending_estimate(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64 vfio_state_pending_exact(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t stopcopy_size, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" stopcopy size 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64 vfio_vmstate_change(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s" +vfio_vmstate_change_prepare(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s" From patchwork Wed Aug 2 08:14:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13337839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7CB17C00528 for ; Wed, 2 Aug 2023 08:16:43 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qR716-0006zD-Hk; Wed, 02 Aug 2023 04:15:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR715-0006yt-8r for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:35 -0400 Received: from mail-bn7nam10on2060b.outbound.protection.outlook.com ([2a01:111:f400:7e8a::60b] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qR713-0006RO-Cy for qemu-devel@nongnu.org; Wed, 02 Aug 2023 04:15:35 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HUJWyUfXFkX0685k5WB6f+Ngno1WV/mI1T77Z/q3ete2qZoGtaOXAnbWzp+noL3HQH8r6nr3WbWeERTspqrFSEA6VkMPCNolgFFOetJh/QFqbnLiKZUiGwm2PLXim2r0E27FOQDHF+zWpX3g4mpcH71SRdJQZ7vaKxBvJ3hhDswvvsc/4MhAdMHa32oDK7rDQuDce5BSCCytgA48l9jSmea4pN63TV9HQLHMj+Yn3Uglo2cSFCJARyNRbj2QXoz8xh1l8KHnK3Q2/+D7rb+X5PIJIhzikfDepeUnB2h5Z8n0s8z+T/aEifKq87E/wxYpbi5nxPI9nxqHqJL6vwowBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XOqslVDeYPp1BKRdY4T9f8yy2aeBgRO7yM4QTsRZNBU=; b=hpPCKWu5/G5MAf6oZSuHlRnL1f/Caif7Br7MiorUH1PJLSjZj49i9XTg1gL9GMyn82K/yZ/7SlEWTWa8GDw9+Sshj4j2s8rDRlTVA4tsrOdCUEaTC9ojN6cBE8Rs4RTrWONlesDkRFZuHd6cXGSv6wsLoXuoY3zi/lbf4HUhXAOwpaPaNBvFQyoXVuaJTuhM3RLZqMsfPKL0Y+d0pOf5uVhxjbiFMH9cpAvT3x8sqMnRfl7OKitXyW2dsvHJgTr2oQelS8T9J6rixQr1gDzufFSfNN7RAux/V/X7dVgK3almGV4CTOAgmptkCrx2nJHoOhIU0n+bOQuC2UpN7dyGbA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XOqslVDeYPp1BKRdY4T9f8yy2aeBgRO7yM4QTsRZNBU=; b=hi27GxnoWD7GfymkYmygnMUF4CT+Gba2kBDXVMBmqFQRr2ZEkRPuDXw1Pa9/BVuf2yb/otYSlYalR3UqxsiYIMZVByNPJCg0BsHIwSsVk511HyWQIaALB9HPH0KvFJ3nC9/LY3OuQx7KTMGwHfmoToiO0mqsjhalIkQooSL5KP00NI+e/PTSgambp3GxfMIapJSND2WpBfnXzY3k78rLwGrthssPm+mEC68N6zu65OQThAT1XIP8eqoLKhyjUNN8/m/UqT46IVHNA0XW2V4PC25St5d42RP95UUGo9qoPTYTNT4ZPHhxzOattHp3rxelpt4S0p+6SvMNamSYG6riVw== Received: from CY8PR11CA0002.namprd11.prod.outlook.com (2603:10b6:930:48::9) by PH8PR12MB6842.namprd12.prod.outlook.com (2603:10b6:510:1c9::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 08:15:29 +0000 Received: from CY4PEPF0000E9CD.namprd03.prod.outlook.com (2603:10b6:930:48:cafe::15) by CY8PR11CA0002.outlook.office365.com (2603:10b6:930:48::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.44 via Frontend Transport; Wed, 2 Aug 2023 08:15:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000E9CD.mail.protection.outlook.com (10.167.241.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 08:15:28 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 01:15:17 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 01:15:16 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 01:15:13 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Paolo Bonzini , Joao Martins , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH for-8.2 v3 6/6] vfio/migration: Allow migration of multiple P2P supporting devices Date: Wed, 2 Aug 2023 11:14:49 +0300 Message-ID: <20230802081449.2528-7-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230802081449.2528-1-avihaih@nvidia.com> References: <20230802081449.2528-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9CD:EE_|PH8PR12MB6842:EE_ X-MS-Office365-Filtering-Correlation-Id: f7dcac20-52ef-474a-b5ae-08db9330a21f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8YA24Uxwi/bKly+MJPG1q0mfJidQj2aWe2/F43yAN0BEDt/gzjLhC5KA4e1Zcuf+dWb/9BVFP82XceBrr7RvBgf9Rhgcu30bVhSSrALIhwk5R3JGCMf7eQKvbWbHHA3A9X6fL+T1rb4B6ppKcE5azy03x04CtCNrKOrKUoMfOwOavgly825Ysa3BzePJjOp8Ujgq7vl3L4okdc95Qg8eCkbWpRCc84KBTWV5SEHPdHMCpz3XWIvmbbGL4JqGgGZjdvzjKouH3LUMXXC5mSAhtgttpn46gIum7+MEdXY4iXAsLpz/Y1FBahYyHJSSmiv8Dzqe1zez/psdaD3brxJh0LWElfON2F59b/6f5TTive4zZmNouEal2QOXzMyAI8xXUu628UX0MGEyb6dj1qOmce4QO72J/uTBEHjY0wNd0I9NCKGxU5vSaacPCF100AraVHhxRyhznADEi+efuiYjcVz9LXB8DNgLIqx6d1FhpDNDEvI57gReXVE8XUyan0/Qy+tvjjY896wikElCLun9KwdkU8OZNtpPcM9aVlZM7F6ea8ZOQfdYgGPwx9urHpbPyveJ8aNv5D7zgUjxj4SCyf+AZtQkEqGkmGPS+8GJfmn+4UCExF5Iq+U0D3T/Knv8ecUm+YGu57P+7S/oDVha2rLm+ZmnKQmJ/HBZtZMQszX2W5WNYczKFwcn5j3Zfe54Ah9te9S1du71hh3cNdlw1qBCsHrphPYn9qFRBNXaaiQ= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(396003)(136003)(376002)(39860400002)(82310400008)(451199021)(36840700001)(46966006)(40470700004)(8936002)(8676002)(5660300002)(426003)(26005)(41300700001)(107886003)(36860700001)(2906002)(83380400001)(1076003)(47076005)(36756003)(40460700003)(336012)(2616005)(40480700001)(478600001)(54906003)(7636003)(316002)(86362001)(82740400003)(7696005)(356005)(70586007)(70206006)(6666004)(4326008)(6916009)(186003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 08:15:28.9664 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f7dcac20-52ef-474a-b5ae-08db9330a21f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9CD.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6842 Received-SPF: softfail client-ip=2a01:111:f400:7e8a::60b; envelope-from=avihaih@nvidia.com; helo=NAM10-BN7-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Now that P2P support has been added to VFIO migration, allow migration of multiple devices if all of them support P2P migration. Single device migration is allowed regardless of P2P migration support. Signed-off-by: Avihai Horon Signed-off-by: Joao Martins Reviewed-by: Cédric Le Goater --- hw/vfio/common.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 7c3d636025..8a8d074e18 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -363,21 +363,31 @@ bool vfio_mig_active(void) static Error *multiple_devices_migration_blocker; -static unsigned int vfio_migratable_device_num(void) +/* + * Multiple devices migration is allowed only if all devices support P2P + * migration. Single device migration is allowed regardless of P2P migration + * support. + */ +static bool vfio_multiple_devices_migration_is_supported(void) { VFIOGroup *group; VFIODevice *vbasedev; unsigned int device_num = 0; + bool all_support_p2p = true; QLIST_FOREACH(group, &vfio_group_list, next) { QLIST_FOREACH(vbasedev, &group->device_list, next) { if (vbasedev->migration) { device_num++; + + if (!(vbasedev->migration->mig_flags & VFIO_MIGRATION_P2P)) { + all_support_p2p = false; + } } } } - return device_num; + return all_support_p2p || device_num <= 1; } int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp) @@ -385,19 +395,19 @@ int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp) int ret; if (multiple_devices_migration_blocker || - vfio_migratable_device_num() <= 1) { + vfio_multiple_devices_migration_is_supported()) { return 0; } if (vbasedev->enable_migration == ON_OFF_AUTO_ON) { - error_setg(errp, "Migration is currently not supported with multiple " - "VFIO devices"); + error_setg(errp, "Multiple VFIO devices migration is supported only if " + "all of them support P2P migration"); return -EINVAL; } error_setg(&multiple_devices_migration_blocker, - "Migration is currently not supported with multiple " - "VFIO devices"); + "Multiple VFIO devices migration is supported only if all of " + "them support P2P migration"); ret = migrate_add_blocker(multiple_devices_migration_blocker, errp); if (ret < 0) { error_free(multiple_devices_migration_blocker); @@ -410,7 +420,7 @@ int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp) void vfio_unblock_multiple_devices_migration(void) { if (!multiple_devices_migration_blocker || - vfio_migratable_device_num() > 1) { + !vfio_multiple_devices_migration_is_supported()) { return; }