From patchwork Mon Mar 4 10:53:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13580408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1297C5478C for ; Mon, 4 Mar 2024 10:54:25 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rh5xO-0003BB-Fx; Mon, 04 Mar 2024 05:54:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rh5xM-00037k-U7 for qemu-devel@nongnu.org; Mon, 04 Mar 2024 05:54:04 -0500 Received: from mail-mw2nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:200a::601] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rh5xI-0005sk-Od for qemu-devel@nongnu.org; Mon, 04 Mar 2024 05:54:04 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fJ2L8E+W50OMU5Jp6MpQDq5iUb8z9M7gdVXXPgcyjjVxb46q3d/CJBO37Oj41DH/kCJyh33DEonkJ29fnDhb02vNndZ9/vOD2DpBxohbw7zBYoDJmCAF35JTXFoDRyWUUGhCj7uagx6wFc8EnAiEfBO/Ia7YGIGAsOKmDXwln3floMH7nqdps33qGWXCboL0pernzSZpWS70aFKZhvIo4I0ZhXVohgFCRMotegZjkBnY47oKt6qZ8DUpXjDpV3o7y/dY1UA+IeUbN5OXGX8p/qnxNr3QzjcNncOPjpgyujxM0EAdMgGICZcC+cl+iHdEvtgnzUDTSldolNh9WIskTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6fAaTZkmK5Z4NYyxaOuFMHZxiguoKYxh/qH/JgnxsVY=; b=az2WTb1fZ8aB8nGHKTgzuF5FGMtpNjPM6Tyrfcql4YCVt9JOq0pE944bEb035WT8ejeKMg/wCOykUp03gheNaeTVM3xujUyn7bu9h0iga5cl8oLL4VwbUpK25ePvI/94nAJZs1gaQwgnE3YaabUbMOkVINI/cV1CDKqXlHLjA2XG9IaLq8YFGdsF7iARNywLOhLGbIVAeGeFx0C99qnhCY/bvfWYeltp28I3OzHfQwE1nzyJF6LlspjMYxsnH9umpmfA448iP7sbf0WfrLIucAI2lZNzHSoH/Ta6V+qjD9VTyVrFTk3kIz/THNb2lfcswa8cWKvDDOrgK+MrGfwKtw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6fAaTZkmK5Z4NYyxaOuFMHZxiguoKYxh/qH/JgnxsVY=; b=WhxQuAaCEqQJ2+x1LLMeRbiIQOgckQblOu3frP+2rKU/ayT2nL3aE82RFPKh9UPudEd+2AxMzDERkz82VeONnm/mu6eJWzhNfrL/2ueJEi1FiUOCQ0sLGTQVIIbcu7djR8jdz7XctAju6kb3OcP02ExWHAK8zl41HZZwKr3Jx8EE4I9nd71JimzxsUtKpBkOg0Ukab8NAY3r/VbLyHmaaDOtl6BDvbll/tQR/jIhgjNEM/pUCyNC2Nb2OM8HLdh9XL4mgm06xiLiqd+k6rMDoSvs0e/KZHDfbly63ojLfofvwYMyZu1KUPbEZXkZrBmjJfDqG5qnxfiZmiy1IyQn9g== Received: from BN8PR04CA0024.namprd04.prod.outlook.com (2603:10b6:408:70::37) by SA3PR12MB7783.namprd12.prod.outlook.com (2603:10b6:806:314::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.38; Mon, 4 Mar 2024 10:53:56 +0000 Received: from BN3PEPF0000B06C.namprd21.prod.outlook.com (2603:10b6:408:70:cafe::70) by BN8PR04CA0024.outlook.office365.com (2603:10b6:408:70::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.38 via Frontend Transport; Mon, 4 Mar 2024 10:53:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN3PEPF0000B06C.mail.protection.outlook.com (10.167.243.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.0 via Frontend Transport; Mon, 4 Mar 2024 10:53:55 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 4 Mar 2024 02:53:45 -0800 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Mon, 4 Mar 2024 02:53:45 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 4 Mar 2024 02:53:43 -0800 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Peter Xu , Fabiano Rosas , "Wang, Lei" , Joao Martins , Avihai Horon Subject: [PATCH v2 1/3] migration: Don't serialize devices in qemu_savevm_state_iterate() Date: Mon, 4 Mar 2024 12:53:37 +0200 Message-ID: <20240304105339.20713-2-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20240304105339.20713-1-avihaih@nvidia.com> References: <20240304105339.20713-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B06C:EE_|SA3PR12MB7783:EE_ X-MS-Office365-Filtering-Correlation-Id: 2060c735-f9ed-4adc-b909-08dc3c396381 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YdFgmc3j3sme/EcwdTObs4nIW/oEf4WxEnyf5NJ3cxgU8dK27H/hlGDsXa1Df1NpqAOXfW8YJLcpdeO4wszcxYmIfEOrMG0OPowuxue6Q0WgLDc9jm3iSsp1/grE+WYR4IAYI2SB/4oLgHTUe2jdnC//XqGc6r5NsYSjdethT1dqvVyIqxBM+tqq7C4/ltjpPO05ow/F98JjR6bvyskB2By4ljYSPo6L2kKU1m+bXlcahnIKR3F/ZaIIlln+VPxg/LSw0jFUBcipjexGEn8uUI+R0eFtgoZVDh36+cBViPpOBYNIhhhYDOBj4HxhwcXITnaxIy8ZOat0EfdYOFi5VINR0Dvtb/IwbjoKUQnBCBgNFq0ncXoYYC+jh+roajDyadn05FpJ1UrRg07KBxE5Fds5TuunsKTJuUtD1+1DdSQRHwmI83D8dopOd74cNcPkVQJ022lfT75mAXDisHQhj/hDNj7hKbiPzTb63madCfP07bWX2kSNEAy1eenJVM4mGNLFImfZDLkvajCOCuD2VXLkoiczOcvWPmBK3ewlo4nF7UOjG71N3uDGrK/S7Z3I/TBvleXgZGLo+EF54IEyLBvYX+u0qMkstMpsr8+jb6Tz9HmOe1LIMcwsHUGIZLRnANN3PJF/nwp3Pajz207GC8edXg5q5yVsNpjdqfDzfStWohGb13TT09VanVi1a+kpCt8w0e2gwBIOXawq8CLQJmARFL/XMR1/iJQp6xr6uGm6W/v4fs1u4NhxbwraY/kc X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(36860700004)(376005)(82310400014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2024 10:53:55.8141 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2060c735-f9ed-4adc-b909-08dc3c396381 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B06C.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7783 Received-SPF: softfail client-ip=2a01:111:f403:200a::601; envelope-from=avihaih@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.589, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Commit 90697be8896c ("live migration: Serialize vmstate saving in stage 2") introduced device serialization in qemu_savevm_state_iterate(). The rationale behind it was to first complete migration of slower changing block devices and only then migrate the RAM, to avoid sending fast changing RAM pages over and over. This commit was added a long time ago, and while it was useful back then, it is not the case anymore: 1. Block migration is deprecated, see commit 66db46ca83b8 ("migration: Deprecate block migration"). 2. Today there are other iterative devices besides RAM and block, such as VFIO, which are registered for migration after RAM. With current serialization behavior, a fast changing device can block other devices from sending their data, which may prevent migration from converging in some cases. The issue described in item 2 was observed in several VFIO migration scenarios with switchover-ack capability enabled, where some workload on the VM prevented RAM from ever reaching a hard zero, thus blocking VFIO initial pre-copy data from being sent. Hence, destination could not ack switchover and migration could not converge. Fix that by not serializing iterative devices in qemu_savevm_state_iterate(). Note that this still doesn't fully prevent device starvation. As correctly pointed out by Peter [1], a fast changing device might constantly consume all allocated bandwidth and block the following devices. However, this scenario is more likely to happen only if max-bandwidth is low. [1] https://lore.kernel.org/qemu-devel/Zd6iw9dBhW6wKNxx@x1n/ Signed-off-by: Avihai Horon --- migration/savevm.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/migration/savevm.c b/migration/savevm.c index d612c8a9020..d76d82e7596 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1389,7 +1389,8 @@ int qemu_savevm_state_resume_prepare(MigrationState *s) int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy) { SaveStateEntry *se; - int ret = 1; + bool all_finished = true; + int ret; trace_savevm_state_iterate(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1430,16 +1431,12 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy) "%d(%s): %d", se->section_id, se->idstr, ret); qemu_file_set_error(f, ret); - } - if (ret <= 0) { - /* Do not proceed to the next vmstate before this one reported - completion of the current stage. This serializes the migration - and reduces the probability that a faster changing state is - synchronized over and over again. */ - break; + return ret; + } else if (!ret) { + all_finished = false; } } - return ret; + return all_finished; } static bool should_send_vmdesc(void) From patchwork Mon Mar 4 10:53:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13580412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22A84C5478C for ; Mon, 4 Mar 2024 10:54:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rh5xV-0003CR-11; Mon, 04 Mar 2024 05:54:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rh5xS-0003Bz-SG for qemu-devel@nongnu.org; Mon, 04 Mar 2024 05:54:10 -0500 Received: from mail-co1nam11on20601.outbound.protection.outlook.com ([2a01:111:f403:2416::601] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rh5xO-0005tY-Ha for qemu-devel@nongnu.org; Mon, 04 Mar 2024 05:54:10 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IJCjeyL7T2GP134F5hpZkIk9pmSdZ3pkBPhuCTNcvsEB/7WNRjBw7imsVWQGa2KQjBGRlqc66PNcuBU6vdwaEcLacsW4t7CQB7YpYWuYIksAWujJNf8weYEXmZbvORhpOnWzsYH8kNUEomTcNZ8KJV7MEwoNBsNQtGqqqyLmJxBZPYnlvs8mYiIl6RJ95TFNhzRQC5UbCWLiZ5UvElvtRheKpg4aVgtO/K9irN4L9ePQ/7Yo7YWoZxOcbhMN0TIUofr7thOlNZ6ewYogvOdXQlmPNMzOFjFOwlIZRJMdjlEd57Pyt4nNUPNUQxvYaX9TIjXgDOBBrzqO5HWbnmHV8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JVwuTJShlRAFgGKpN33gMlFEoL8O9HLOKs6/BcfdV4E=; b=fLWMeKf/Z+rhwQg2TsWRVUXRS+DgMyO0Ut26rqUzsQ4g+id9uG1gZxL8uoHOY/BsBqvrXdMIqhAHCiXZf9jveu50BALH6wtr5R/SuwpVxBasvoY2224s/ytRpYrIBhykfUZ2JRuvvWJAKzqzqag8VKdN5fhb1E178ITijTRAIOhs8a5TtJmFQkouoLNHgKyAo23WAimdtnJQupE3TGlB+WwL0z2ewqgzyS3hSBObQqPzbuo2AQSKN4GDLXLcUzGR4EhTxuKzxcJLOw97aq/F8USiWXnYEFCs6PleSX+SydwED6y2idemJ9R66z98wuoNfPHmHfT74DSolRAKAzDvpQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JVwuTJShlRAFgGKpN33gMlFEoL8O9HLOKs6/BcfdV4E=; b=WzWlRc+GaWI/Ov0GnTQhPIvwr7wKLR53C846xOxdYcuxij5ZGC8NZjuZhtXV0L5KNHotcw4LS8xghs0KSdmJY5jNXGEBA4nb3ssiCD9gDK2rx3rKtgNM0itYZjvwt5xAgXpv4LrC2pQEnhM5Ms9tsideWMJL9UFNXB2RJgj/BwYZG9Hr5G1kUJEPgtj1tm1vd2AmJGuQ89YaYA+rIsj7VIREYSHxZI6h/b7udpiCf2QGic3mcfr7kAZtryxJqn/xdz/uGPK7XYErdgrfGzck+qHzZ2DIzIIN5milBY500YNvInAPH52PpS6KuRPQQGo2MnpTkzzx76Azx1qrTkgA2w== Received: from BL0PR03CA0027.namprd03.prod.outlook.com (2603:10b6:208:2d::40) by CH3PR12MB7690.namprd12.prod.outlook.com (2603:10b6:610:14e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.38; Mon, 4 Mar 2024 10:54:02 +0000 Received: from BL02EPF0001A101.namprd05.prod.outlook.com (2603:10b6:208:2d:cafe::4d) by BL0PR03CA0027.outlook.office365.com (2603:10b6:208:2d::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.39 via Frontend Transport; Mon, 4 Mar 2024 10:54:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF0001A101.mail.protection.outlook.com (10.167.241.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7362.11 via Frontend Transport; Mon, 4 Mar 2024 10:54:01 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 4 Mar 2024 02:53:48 -0800 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Mon, 4 Mar 2024 02:53:47 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 4 Mar 2024 02:53:45 -0800 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Peter Xu , Fabiano Rosas , "Wang, Lei" , Joao Martins , Avihai Horon Subject: [PATCH v2 2/3] vfio/migration: Refactor vfio_save_state() return value Date: Mon, 4 Mar 2024 12:53:38 +0200 Message-ID: <20240304105339.20713-3-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20240304105339.20713-1-avihaih@nvidia.com> References: <20240304105339.20713-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A101:EE_|CH3PR12MB7690:EE_ X-MS-Office365-Filtering-Correlation-Id: 07dd0b9c-45de-45ec-bc84-08dc3c3966c7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mZH5Dbt360I7xTYGRCWmC4kD4MnnKWPdWb9z7vkPuY0E2zYtl24B/ljfxgSomWr82CWz0KsuGyYv+Dsrx3sexr//SEayARsycsv9dtdW+4+0gFEOh/G7WtYyrNUHNK2MKthIS+NWYVKKSCUcP9ECjFmYO2RYiUX4sRwsxgMGKpJT355pzsOPZJcoqMX6LMOtQ/lMiMOKYvPh45Btiz9z8ZC1AY9c3yijk9VDw0oWzp/uQDyuT2e5JSewNPxhQvesIVLf1VxTMWq07jRY0T4m/ncH9IcA1+YzUrGGh11LLJ5+khB3pzpo3ZW2JkFW7NOpBQtvjLMSum25fBrUAxzU01PEdp+nEVK7CHv/DxY5Or7umsHfDJqWh6EREIGlnzyL7c/F+ABP1TII1WOpaCXjk4siaSSBc+eKEH+uFSnRy+pjfyfUfi3x2d9MM30knli3hVaYvwknkcRpzIbAA+2FBrpBBOVsh/uiBFe677IxArB8WWMvBITUUIaojrEWqoRIZs1pGnIscnhjVd+nX2tPMrTMRyEba7dK2JHXsu02wC3LH4TX1/GerSGNuw7shgQjbIAenBdPuJnTGjPvLPkwQVKKOuWHpx/jAEBnFfQebF9QJWnalBnZSWTV5IHItrmG1r8l8PHrHkdbBTWMy2ie+HSIvtWyDgRPEYxQEo62masdusW288Ds/rgcnsal1v3FIcGfJoUlZxr8qhYe7zva9aAaVDWF1J+e04I0RcYJWRtMnA1+GcRVp3P2xIRixBxr X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230031)(36860700004)(376005)(82310400014); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2024 10:54:01.2904 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 07dd0b9c-45de-45ec-bc84-08dc3c3966c7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A101.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7690 Received-SPF: softfail client-ip=2a01:111:f403:2416::601; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.589, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently, vfio_save_state() returns 1 regardless of whether there is more data to send or not. This was done to prevent a fast changing VFIO device from potentially blocking other devices from sending their data, as qemu_savevm_state_iterate() serialized devices. Now that qemu_savevm_state_iterate() no longer serializes devices, there is no need for that. Refactor vfio_save_state() to return 0 if more data is available and 1 if no more data is available. Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 50140eda872..0af783a5892 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -529,11 +529,7 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) trace_vfio_save_iterate(vbasedev->name, migration->precopy_init_size, migration->precopy_dirty_size); - /* - * A VFIO device's pre-copy dirty_bytes is not guaranteed to reach zero. - * Return 1 so following handlers will not be potentially blocked. - */ - return 1; + return !migration->precopy_init_size && !migration->precopy_dirty_size; } static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) From patchwork Mon Mar 4 10:53:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13580413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11E2BC48BF6 for ; Mon, 4 Mar 2024 10:54:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rh5xa-0003Ct-Sw; Mon, 04 Mar 2024 05:54:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rh5xX-0003Cc-AJ for qemu-devel@nongnu.org; Mon, 04 Mar 2024 05:54:15 -0500 Received: from mail-dm6nam10on2060f.outbound.protection.outlook.com ([2a01:111:f400:7e88::60f] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rh5xT-0005uD-Ut for qemu-devel@nongnu.org; Mon, 04 Mar 2024 05:54:15 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gsG1vfoheecn/tF594PUkjSQDAmBQvTWqPQQ+Ybb+LtsD0hgU7L0qgiwPbGPCCXIEFWKbop4lP5yvjCqnX51yEN4sr5GlM2MLw3JYQwAPYTu8KB4v8BzbRdcfoeaJGKBMChe/EbbHNE83hy1SHfkx1UKzPxTdiUkP6tVQg0k3fempV4gt5PfUy4UL6Zs2VRnjKNhLeU9o0p4IYwSdOzCKStmth5Oo+GEyN/nh/cogAIs4YRZDkEARxgT5B5Mq5J4EH/KXc9G9lcK9iTgPFScztlRsN6mEYue4TTYlRqwfV29gYn1F7DQs3CMj+U6/Oz+srEMz6yTb0ClI7ktxoQs3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3kM7KgztYCkLtrafdUf6/mkJhTM9jfLsyNBrs6f2Chk=; b=DnkDJcnWfNDDenDWBa40i6XlIvfsAtgUy406VUzFI09MN1uAAhNMGjOzlWjhYi8UCTsE5thxc1LO/8/0glAUJOMOqSqGv9ji9842pSLFiL5rf43ZlEcTpfG5D1yZ/c+QpNCqJBhkuc5axS+LtCyEfYcdjmg97+iwYEfNoJNsk9TGJopmtsHgzqY4/NRO9G4u9CPaqPeq6nA7mE1+t+WW7OgRdKBTXamPnMq8TOV3Wad9kDUhyFf7DGSY/EYblPuZbdSe6OyAN7icqd5WyyaNFFQscHa2M1IkoBHQ/mtYk7Cue1PTTskounyuXYohyqd7c1mwegw5DYMPNurvTQJ6fQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3kM7KgztYCkLtrafdUf6/mkJhTM9jfLsyNBrs6f2Chk=; b=ixiAd/Q/jAKf1lxlH9Nkw3kS8WMSNRPIgArbtqtAIP/G/IEiQ4kY9/YIzGkuhdMYZdlV+845ksy+uXXhp9iLshJFUwyicebyEXWs4kwB+RDkDExh/N+DbObTV508gmHQ/nZc4tCt7obYIkiID7pQbL59cXhmFAgw8Y/piQUMGibLX+USCT/tZl6oW1DUmB04q3YJmw+xH9EmO40P4nB+v0WIPZb6D/DPn1DPI8BLq8v51cIkzeOp8TGeJ4X8WYqE7MfHQTOYUaKNm7P3sleOVyGxTEiMc3yBlHymmcc+V8H2VegnYsGFLPiUSs/hHvsLrrfeyAFrn/mtS4NeUb4mVQ== Received: from BN9PR03CA0928.namprd03.prod.outlook.com (2603:10b6:408:107::33) by CH0PR12MB8506.namprd12.prod.outlook.com (2603:10b6:610:18a::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.36; Mon, 4 Mar 2024 10:54:04 +0000 Received: from BN3PEPF0000B06E.namprd21.prod.outlook.com (2603:10b6:408:107:cafe::14) by BN9PR03CA0928.outlook.office365.com (2603:10b6:408:107::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.39 via Frontend Transport; Mon, 4 Mar 2024 10:54:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN3PEPF0000B06E.mail.protection.outlook.com (10.167.243.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.0 via Frontend Transport; Mon, 4 Mar 2024 10:54:03 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 4 Mar 2024 02:53:50 -0800 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Mon, 4 Mar 2024 02:53:50 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Mon, 4 Mar 2024 02:53:48 -0800 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Peter Xu , Fabiano Rosas , "Wang, Lei" , Joao Martins , Avihai Horon Subject: [PATCH v2 3/3] vfio/migration: Add a note about migration rate limiting Date: Mon, 4 Mar 2024 12:53:39 +0200 Message-ID: <20240304105339.20713-4-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20240304105339.20713-1-avihaih@nvidia.com> References: <20240304105339.20713-1-avihaih@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B06E:EE_|CH0PR12MB8506:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ac8a222-9830-4b06-d8bf-08dc3c396826 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AAmXQSIVMoiT4o+k+xCCMg1SyxpX+Nu0sDRXQbpVZIT8uplKadh6ZJd+V7IzNSwGvUw4xYMAa9kf8TLEas9R67NPHNVhjoqha6w8U07nUzIgVy3evJU2UJ1mnBGpE4V3INOXngsHj5Du7h0nc/M844P7QVx2aBvv4dLw3zLFAfIvseDacWS0PqBvsMgscJz9YI28p1YEG8DWXMw9G9bDsAbKFAWmR68XTyubltzJOjxk4sZ4ZIlofKQYGFvm4v4igWXxsbxFpkIF2sGGlWxM9DH/84nX16ha1U9O1/RdwaNjVyJy1bU+HmxH50OSPyVWTlvTpORIiuXZ8ILf5bDLXv+7x8UQSmTcUx/eO394DZVj5459l//gKHeQi690lPJIjeIcqoYNGH25FFpX8Ozfh0cX6axQaUGlDQDmdPkbocIdMNFomFk0cVSJSAfAE0PPqxKrgcrkJdC64h/ECtAiAjXOQcjExGnsFITA/07NwD231bK1rxbKrEyZTknTouB/rBsXt9RD+10/yXTL4Df8bWulfRmQ0+kJMarcoqVN3nuMWtjfGMvucZAQ9PDmoHGAtuz6Fcs+MleV0l92DW9OGR7tXEsXeUu11foZzWRh7MUsT1k11UWlCYsQg5pWYG2Rc5k+kryrexu5aNVv9CuIUFNAJWDOO1NBq5PjtTe24/CW2Nayd+WktXXZQI2Z/Fz+XsluL2XA3PVLCdhy2NsUaE+G2hxm3C73zlUbMZZLyLpnzPdr4ZQJCFJhzbbS5jKy X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(82310400014)(36860700004)(376005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2024 10:54:03.5915 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2ac8a222-9830-4b06-d8bf-08dc3c396826 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B06E.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8506 Received-SPF: softfail client-ip=2a01:111:f400:7e88::60f; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.589, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org VFIO migration buffer size is currently limited to 1MB. Therefore, there is no need to check if migration rate exceeded, as in the worst case it will exceed by only 1MB. However, if the buffer size is later changed to a bigger value, vfio_save_iterate() should enforce migration rate (similar to migration RAM code). Add a note about this in vfio_save_iterate() to serve as a reminder. Suggested-by: Peter Xu Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 0af783a5892..f82dcabc49a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -505,6 +505,12 @@ static bool vfio_is_active_iterate(void *opaque) return vfio_device_state_is_precopy(vbasedev); } +/* + * Note about migration rate limiting: VFIO migration buffer size is currently + * limited to 1MB, so there is no need to check if migration rate exceeded (as + * in the worst case it will exceed by 1MB). However, if the buffer size is + * later changed to a bigger value, migration rate should be enforced here. + */ static int vfio_save_iterate(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque;