From patchwork Mon Sep 18 04:49:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 13388799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CEB16CD37B0 for ; Mon, 18 Sep 2023 04:51:15 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qi6DH-0003QL-Ul; Mon, 18 Sep 2023 00:50:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6DD-0003Pb-HE for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:50:20 -0400 Received: from mail-mw2nam10on20609.outbound.protection.outlook.com ([2a01:111:f400:7e89::609] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6D7-0003cE-Nh for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:50:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BriSvDKjNc4l48uwtThk3RRdcEj/Lx9In4MdKS96tdlpuwV+ahZTXwqJp5PmidoqCYwRxM5A6Nxij+oS+U6AzhtYpAoKZwkL1McbBuut11++vQ6BLihe4JZTkEMjBUfz90qQECgGK/Vy38RiManOdWFjL2IKetOhuGVCo9iiTaMLi7dI+QNzhnbWiSbi74zMZ2lXvqc93ac9X3M/9RKTPcxVywro2CicmYxlq7tevMP7o/uOwkIYur88t3/IeszOTeln2hT1fvyj3tKozxlp0ZTq4v46s9FNTMDyzs6JV6QcTKKnnBgiE8A1yJb5CBPjX8NkTNZ+MAiVW43tlhyfUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HyvTH2zfw9Z+hRNhzFi1cneA5xIzTM4jiJfgb8FyEgo=; b=S4qYABX+Mfts2OnDgq0hetqCv7LpYIqMtETtaNMnahlF3u8L1rkzO7zb1EeOAUDo1GU7AQCMEcejRMjutMgk6jY+KQBI88+28HJKtP+kDe4r41VU0jluVvS0FJBG+TAueawXV5H7KP5n46S9WES/nSF2XDqqX/e/kSo8Z8UaWf+53ION15e8vrPo00Pb4wWeE5DoYVFpMcE8nWhQglSArXoMsTJzriGk7ujWFemxQNz0rhkOO6TcJ/kZ1lsA0MDWn/aqhziAVHvcMviEpiJNoZxCLX22s6PtHhsQuxdDUD/WWwlJyfYVcic/mrcwAOKJ/qbA08R3b78BCuZ78lEEVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HyvTH2zfw9Z+hRNhzFi1cneA5xIzTM4jiJfgb8FyEgo=; b=bZbmUxYYYu8ob7kAV2WmyuIALYukjz5VRijiYgtlLxWY4cEZwpWh4NiyACURfavpRUe9cfVFfUZZD5FDEDncGXQyEcT2dXa+m1r6vx3DYyz4Z3vCSTclmOFBX+oM51etyH7q2TmMFkW5LeWSFmvkx5jgBPcxHR35ZhAVLAEKiBZ3shv2w3Ql4WfSB3j4zsLbhFRwfuPxzdDjxuM7KHn8yESR108+Z/Ic9EXoleuv9ZOu7VgnVGHbjRtoYdhnXD/mb2lHIgp28JJQcUR/fSZwcOfU2v3Hb1EUoiReglrKCxMrf1KVhgdJm6aqB8PPAvt3RAwCDcg6wdoFeJFbNKYchQ== Received: from DM6PR02CA0103.namprd02.prod.outlook.com (2603:10b6:5:1f4::44) by PH7PR12MB7940.namprd12.prod.outlook.com (2603:10b6:510:275::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 04:50:06 +0000 Received: from DS3PEPF000099D8.namprd04.prod.outlook.com (2603:10b6:5:1f4:cafe::5e) by DM6PR02CA0103.outlook.office365.com (2603:10b6:5:1f4::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26 via Frontend Transport; Mon, 18 Sep 2023 04:50:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF000099D8.mail.protection.outlook.com (10.167.17.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Mon, 18 Sep 2023 04:50:05 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:54 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:52 -0700 From: Yajun Wu To: , , , CC: Avihai Horon , Jiri Pirko Subject: [RFC PATCH 1/5] vhost-user: Add presetup protocol feature and op Date: Mon, 18 Sep 2023 12:49:28 +0800 Message-ID: <20230918044932.1433744-2-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230918044932.1433744-1-yajunw@nvidia.com> References: <20230918044932.1433744-1-yajunw@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail202.nvidia.com (10.129.68.7) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D8:EE_|PH7PR12MB7940:EE_ X-MS-Office365-Filtering-Correlation-Id: c6a22ad2-69e8-4d5c-373d-08dbb802ba2d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 06hPi2sebr9IWqsnKw4IiE7/LGIalVDlPFQOsBO2+2Rh1vyO7Pwd5gksQa420hP2EpXGbbg6lplPodN6guIVwc+3af1XeNbam7BK0MMHC1Tnyl93W/5aKEl+u92N1XLzd4EDxA+KL5qAfB2PbGR8a2/4HeDWvPYQCyh7HvObd3ZzhjfK6xDtZY63rxwtAEYAkizE/uL9qI4SCTUvxyLXp5SlstcaZWG3oPreRnzXUe4ayR6QhM8NABHLy3h3qsee19ojNUqwaQcEg/fSs/CO/V+X5faeFA2jlY08aNo0j14UbOJ6mUDHNwN+q2UKZedJ+gKLieIoO+gaGvBjlRQjMXPAfykow8wtWa72Omr3UpLprwbkGmldzcC2IU4rY4Lgogn4mTvheKDSX+XB+zpdYxv6Nh2ogq/CGVWyxTLH9RErWVViYvIkIfFQ3HBBuM0/Eq3q+ZrvZ9GvZO5QgRl11kclwV3hPQnd+BVKR8zLdBsTeyIxeME7TOXhN0uLWATAb88FLZHTOKSmJ4XJhWfrFPfGITTqqyyvrYyZzmqxfIl9pc1XQ1UrU9lvhzLZj/OiqSbvX3sfqCkY6zUasGs+RLM+tIXlSAPRw5cqELcdmCz4g+hH3kKYHX0gz6mWNtK0cLQRc6S3MiaOniy9IE3UWCaQvsZIAkD7T3nnFZezPw3q+Heek2kNFQygWzEP7r7a6CAPzVbtKpqWkcVIVo91vQdR3J3lG5KbWqwcIvfphGGQKm8kmh8Tv4hwUXMw5gzb X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(136003)(396003)(376002)(39860400002)(346002)(186009)(1800799009)(82310400011)(451199024)(36840700001)(46966006)(40470700004)(7636003)(356005)(26005)(16526019)(6286002)(82740400003)(2616005)(8936002)(1076003)(40460700003)(4326008)(8676002)(107886003)(83380400001)(36860700001)(2906002)(36756003)(47076005)(426003)(336012)(40480700001)(55016003)(5660300002)(86362001)(7696005)(478600001)(6666004)(316002)(54906003)(70586007)(70206006)(110136005)(7049001)(41300700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 04:50:05.4309 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c6a22ad2-69e8-4d5c-373d-08dbb802ba2d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D8.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7940 Received-SPF: softfail client-ip=2a01:111:f400:7e89::609; envelope-from=yajunw@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch implements VHOST_USER_PROTOCOL_F_PRESETUP protocol feature and VHOST_USER_PRESETUP, so that the backend can know the beginning and completion of the early setup phase for the virtio device. Unlike the regular device state load, which occurs in the VM stop phase, this pre-setup takes place in the live migration setup stage. Signed-off-by: Yajun Wu Reviewed-by: Avihai Horon Reviewed-by: Jiri Pirko --- docs/interop/vhost-user.rst | 10 ++++++++++ hw/virtio/vhost-user.c | 30 ++++++++++++++++++++++++++++++ include/hw/virtio/vhost-backend.h | 3 +++ 3 files changed, 43 insertions(+) diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst index 5a070adbc1..70b8e2694c 100644 --- a/docs/interop/vhost-user.rst +++ b/docs/interop/vhost-user.rst @@ -885,6 +885,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15 #define VHOST_USER_PROTOCOL_F_STATUS 16 #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17 + #define VHOST_USER_PROTOCOL_F_PRESETUP 18 Front-end message types ----------------------- @@ -1440,6 +1441,15 @@ Front-end message types query the back-end for its device status as defined in the Virtio specification. +``VHOST_USER_PRESETUP`` + :id: 41 + :equivalent ioctl: N/A + :request payload: ``u64`` + :reply payload: N/A + + When the ``VHOST_USER_PROTOCOL_F_PRESETUP`` protocol feature has been + successfully negotiated, this message is submitted by the front-end to + indicate start or end early setup. Value 1 means start, 2 means end. Back-end message types ---------------------- diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 8dcf049d42..71018d06c1 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -74,6 +74,8 @@ enum VhostUserProtocolFeature { /* Feature 14 reserved for VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS. */ VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS = 15, VHOST_USER_PROTOCOL_F_STATUS = 16, + /* Feature 17 reserved for VHOST_USER_PROTOCOL_F_XEN_MMAP. */ + VHOST_USER_PROTOCOL_F_PRESETUP = 18, VHOST_USER_PROTOCOL_F_MAX }; @@ -121,6 +123,7 @@ typedef enum VhostUserRequest { VHOST_USER_REM_MEM_REG = 38, VHOST_USER_SET_STATUS = 39, VHOST_USER_GET_STATUS = 40, + VHOST_USER_PRESETUP = 41, VHOST_USER_MAX } VhostUserRequest; @@ -132,6 +135,11 @@ typedef enum VhostUserBackendRequest { VHOST_USER_BACKEND_MAX } VhostUserBackendRequest; +typedef enum VhostUserPresetupState { + VHOST_USER_PRESETUP_START = 1, + VHOST_USER_PRESETUP_END = 2, +} VhostUserPresetupState; + typedef struct VhostUserMemoryRegion { uint64_t guest_phys_addr; uint64_t memory_size; @@ -2741,6 +2749,27 @@ static void vhost_user_reset_status(struct vhost_dev *dev) } } +static int vhost_user_set_presetup_state(struct vhost_dev *dev, bool start) +{ + if (start) { + return vhost_user_set_u64(dev, VHOST_USER_PRESETUP, + VHOST_USER_PRESETUP_START, false); + } else { + return vhost_user_set_u64(dev, VHOST_USER_PRESETUP, + VHOST_USER_PRESETUP_END, false); + } +} + +static int vhost_user_presetup(struct vhost_dev *dev, bool start) +{ + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_PRESETUP)) { + return -ENOTSUP; + } + + return vhost_user_set_presetup_state(dev, start); +} + const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_backend_init = vhost_user_backend_init, @@ -2777,4 +2806,5 @@ const VhostOps user_ops = { .vhost_set_inflight_fd = vhost_user_set_inflight_fd, .vhost_dev_start = vhost_user_dev_start, .vhost_reset_status = vhost_user_reset_status, + .vhost_presetup = vhost_user_presetup, }; diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index 31a251a9f5..00dd532df9 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -133,6 +133,8 @@ typedef int (*vhost_set_config_call_op)(struct vhost_dev *dev, typedef void (*vhost_reset_status_op)(struct vhost_dev *dev); +typedef int (*vhost_presetup_op)(struct vhost_dev *dev, bool start); + typedef struct VhostOps { VhostBackendType backend_type; vhost_backend_init vhost_backend_init; @@ -181,6 +183,7 @@ typedef struct VhostOps { vhost_force_iommu_op vhost_force_iommu; vhost_set_config_call_op vhost_set_config_call; vhost_reset_status_op vhost_reset_status; + vhost_presetup_op vhost_presetup; } VhostOps; int vhost_backend_update_device_iotlb(struct vhost_dev *dev, From patchwork Mon Sep 18 04:49:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 13388812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AEB29C46CA1 for ; Mon, 18 Sep 2023 04:55:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qi6I2-00076J-Rb; Mon, 18 Sep 2023 00:55:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6Hy-00075r-3I for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:55:14 -0400 Received: from mail-mw2nam12on2047.outbound.protection.outlook.com ([40.107.244.47] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6Hv-0004dp-Qt for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:55:13 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LI2nSRbfAXo7zgbPTpAT3WYf6zfMFcwhIcTlHDpy3wl9f6vof98h+aGltlyO0twv0nkaSEik6B0+u9gQxhMO6Fb1xWfkYrFznEWkFN8m7m8rMDRmDsLzKIsIe4GpfuurPqACGBuToYboAwQPRjJKWrSZ8XLn78Q/9MqZE/qEYciq8XREo9scUl3+A6Wfrcm0J5TbjVQHNSlzbRSlyXMb0ndGoCInw0EBhwvCKnyHu3+dFjLYXcwV4Z8wTcZoAtSIZ+BRLuNicqKKUKqOLCaKMBKOft6soTMwpOWMWxaeK7eDRfdHOqK2sbvVZh4AdNARCbq/IAJEQO9shwUCpl9Wrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MUyrIQETz3SMEhrkoA5qJfRqTgbj1DJqmlBH5eju3gQ=; b=BOrQRYreIJV06Xq06+V6FukMJffGuou/Y+IJDs+Y00UZroP3vbMdiyoDPGT4r4ZwOiX75A/9bFVSsxJDAG1X3lOtOo8WCw+UpgmybrJPQ3+b6vi/73q4WhmLX7x6SU1AFYbTLH0dsvij2t7u9PgL5780klPTIwlKwyWCq7xwF5Q4da/zE/OKX3A1RMpvIiDrtfdXRWf9BCDewfEVpBBctf7TJkMVI9AQ2k28X1FCJPky2q1LdLcb9XYFJQjEUPtcOSWpQstCOMvAXmEwUiReoaVyytdNxm+KxWh6hv8CDp7cL479K+X9m8sLEE6TvUwMQibdiQ8e2MC3qArLEFuQvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MUyrIQETz3SMEhrkoA5qJfRqTgbj1DJqmlBH5eju3gQ=; b=NEKnSo/63BjMdKTYolfNypSKKgDWfrhqymG6ng8UnMUypotyJFKXG70rziNz5fj8Vkd8ewvKQG8bpGb5KXepuX2SswZXsfvbRmkYmNsy1B/fBRri3DoIzlQ3HEyAg27lNcTiDyYYrwPJk2GECjB7eqQUePkWYrdVd03GFYMK1rHqXc7/sC+XqlRPQp47mhQZs1DCOxUo6RFMfGOj9+bcFITDg/yB12m0Nf50W1CXgslzSJ0sut9YVW+vZD9yta0Dcd5N6joTFKtxRQKqi6Sdt/dhMo6/2B1AUMQfTMe9gv2naQzX/+40f9uCxz8WZlE5wtte48YGEVMDlWqygIdFHQ== Received: from DM6PR02CA0083.namprd02.prod.outlook.com (2603:10b6:5:1f4::24) by MN2PR12MB4270.namprd12.prod.outlook.com (2603:10b6:208:1d9::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 04:50:06 +0000 Received: from DS3PEPF000099D8.namprd04.prod.outlook.com (2603:10b6:5:1f4:cafe::df) by DM6PR02CA0083.outlook.office365.com (2603:10b6:5:1f4::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26 via Frontend Transport; Mon, 18 Sep 2023 04:50:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF000099D8.mail.protection.outlook.com (10.167.17.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Mon, 18 Sep 2023 04:50:06 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:56 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:54 -0700 From: Yajun Wu To: , , , CC: Avihai Horon , Jiri Pirko Subject: [RFC PATCH 2/5] vhost: Add support for presetup Date: Mon, 18 Sep 2023 12:49:29 +0800 Message-ID: <20230918044932.1433744-3-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230918044932.1433744-1-yajunw@nvidia.com> References: <20230918044932.1433744-1-yajunw@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail202.nvidia.com (10.129.68.7) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D8:EE_|MN2PR12MB4270:EE_ X-MS-Office365-Filtering-Correlation-Id: 4652660a-82b0-467f-1d03-08dbb802bab5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Vh/dW8NqUYiW3IuhyMi9TBkJBIKzRyckejWS6w2fXBDORAs/JD6DzbrN58WpCE9B5BibQCwagjijOZnvVDi9ddkMwvbvO7x7jiV9btO0LV4JBs8Epo8woLGtexJKTucHZz/ZLiypJ63jZ7uj1t2nWjB7X/EoCeRx4JDTzErrsztMvtkPkgrzVxXhh/0jB9jcCmNzHrBhgslrZvoPx7lul4WHJrIq2PQcD0stTbxR5tMU97kfZuql0Jbayv6KWvskPpGVLSq2tkBhA32DdT6bCfKFj3j3/Ze471ht+qG6XnUyDtS8PuVf6O3eijddQ5anPvIB1OY4qhDdL2QDsoPO64Ows/2MaqwYUqumq07q0XlqwdnC949tvOYa06fkuvN/vDVAtAuPiT7CQwSoWra1vVu1U6gPiubBgJ7oK44Z6vVaOjGSB7Ty4OU9wAX1iUZ9SpLd0W1rq2CNk1I1kddBn5cLVHr4GMRaEk8JtXabXvsn/RoMvTx0aEcdwW7bjKy+EqXzcmLrx3IAR2MgHVd4zOlX7z+eBLs5cah2Zl+djyK7an236+RPJivpJxqmfiWnzXxUR1XH3qiRfUtVNx3AKVUFYHF1Kk3naGP1fZ9KRlQkLYnp8/jQ+eHg7RVByzKPOmxgCB0opPs3fwTpN5C40QdaUkruXx39zD1dQKvqZyVkYgaXx32XDVhHQ5z5kI5E8GyM/oa4d7BUF1MIdq7/BKsbV+ULvy+RIu8VURjelZPyEh53oZDT49SGrKAgcCEs X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(136003)(396003)(346002)(376002)(39860400002)(451199024)(82310400011)(1800799009)(186009)(46966006)(36840700001)(40470700004)(40480700001)(7636003)(36756003)(356005)(82740400003)(86362001)(55016003)(7049001)(40460700003)(7696005)(478600001)(5660300002)(54906003)(70206006)(70586007)(2906002)(6666004)(8676002)(8936002)(4326008)(426003)(336012)(83380400001)(47076005)(41300700001)(110136005)(36860700001)(316002)(107886003)(6286002)(26005)(16526019)(2616005)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 04:50:06.3371 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4652660a-82b0-467f-1d03-08dbb802bab5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D8.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4270 Received-SPF: softfail client-ip=40.107.244.47; envelope-from=yajunw@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add New API vhost_dev_start_presetup to notify backend the start and end of presetup. API vhost_dev_presetup to send out the device configurations: 1. acked_features 2. memory table 3. vring information 4. disable host/guest notifier. Signed-off-by: Yajun Wu Reviewed-by: Avihai Horon Reviewed-by: Jiri Pirko --- hw/virtio/vhost.c | 166 ++++++++++++++++++++++++++++++++------ include/hw/virtio/vhost.h | 12 +++ 2 files changed, 152 insertions(+), 26 deletions(-) diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index e2f6ffb446..5b162590fb 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1138,24 +1138,71 @@ out: return ret; } -int vhost_virtqueue_start(struct vhost_dev *dev, - struct VirtIODevice *vdev, - struct vhost_virtqueue *vq, - unsigned idx) +static void vhost_virtqueue_memory_unmap(struct vhost_dev *dev, + struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, + unsigned idx) +{ + if (vq->used) { + vhost_memory_unmap(dev, vq->used, + virtio_queue_get_used_size(vdev, idx), + 1, virtio_queue_get_used_size(vdev, idx)); + vq->used = NULL; + } + + if (vq->avail) { + vhost_memory_unmap(dev, vq->avail, + virtio_queue_get_avail_size(vdev, idx), + 0, virtio_queue_get_avail_size(vdev, idx)); + vq->avail = NULL; + } + + if (vq->desc) { + vhost_memory_unmap(dev, vq->desc, + virtio_queue_get_desc_size(vdev, idx), + 0, virtio_queue_get_desc_size(vdev, idx)); + vq->desc = NULL; + } +} + +static int vhost_virtqueue_disable_notify(struct vhost_dev *dev, + struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, + unsigned idx) { - BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev))); - VirtioBusState *vbus = VIRTIO_BUS(qbus); - VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus); - hwaddr s, l, a; - int r; int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx); struct vhost_vring_file file = { .index = vhost_vq_index }; + int r; + + file.fd = -1; + r = dev->vhost_ops->vhost_set_vring_kick(dev, &file); + if (r) { + VHOST_OPS_DEBUG(r, "vhost_set_vring_kick failed"); + return r; + } + + r = dev->vhost_ops->vhost_set_vring_call(dev, &file); + if (r) { + VHOST_OPS_DEBUG(r, "vhost_set_vring_call failed"); + return r; + } + + return 0; +} + +static int vhost_virtqueue_vring_setup(struct vhost_dev *dev, + struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, + unsigned idx) +{ + hwaddr s, l, a; + int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx); struct vhost_vring_state state = { .index = vhost_vq_index }; - struct VirtQueue *vvq = virtio_get_queue(vdev, idx); + int r; a = virtio_queue_get_desc_addr(vdev, idx); if (a == 0) { @@ -1186,6 +1233,10 @@ int vhost_virtqueue_start(struct vhost_dev *dev, } } + if (vq->desc) { + vhost_virtqueue_memory_unmap(dev, vdev, vq, idx); + } + vq->desc_size = s = l = virtio_queue_get_desc_size(vdev, idx); vq->desc_phys = a; vq->desc = vhost_memory_map(dev, a, &l, false); @@ -1212,6 +1263,36 @@ int vhost_virtqueue_start(struct vhost_dev *dev, if (r < 0) { goto fail_alloc; } + return 0; + +fail_alloc: +fail_alloc_used: +fail_alloc_avail: + vhost_virtqueue_memory_unmap(dev, vdev, vq, idx); +fail_alloc_desc: + return r; +} + +int vhost_virtqueue_start(struct vhost_dev *dev, + struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, + unsigned idx) +{ + BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev))); + VirtioBusState *vbus = VIRTIO_BUS(qbus); + VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus); + int r; + int vhost_vq_index = dev->vhost_ops->vhost_get_vq_index(dev, idx); + struct vhost_vring_file file = { + .index = vhost_vq_index + }; + struct VirtQueue *vvq = virtio_get_queue(vdev, idx); + + r = vhost_virtqueue_vring_setup(dev, vdev, vq, idx); + if (r) { + VHOST_OPS_DEBUG(r, "vhost_virtqueue_vring_setup failed"); + goto fail_vring_setup; + } file.fd = event_notifier_get_fd(virtio_queue_get_host_notifier(vvq)); r = dev->vhost_ops->vhost_set_vring_kick(dev, &file); @@ -1245,16 +1326,8 @@ int vhost_virtqueue_start(struct vhost_dev *dev, fail_vector: fail_kick: -fail_alloc: - vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx), - 0, 0); -fail_alloc_used: - vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx), - 0, 0); -fail_alloc_avail: - vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx), - 0, 0); -fail_alloc_desc: + vhost_virtqueue_memory_unmap(dev, vdev, vq, idx); +fail_vring_setup: return r; } @@ -1296,12 +1369,7 @@ void vhost_virtqueue_stop(struct vhost_dev *dev, vhost_vq_index); } - vhost_memory_unmap(dev, vq->used, virtio_queue_get_used_size(vdev, idx), - 1, virtio_queue_get_used_size(vdev, idx)); - vhost_memory_unmap(dev, vq->avail, virtio_queue_get_avail_size(vdev, idx), - 0, virtio_queue_get_avail_size(vdev, idx)); - vhost_memory_unmap(dev, vq->desc, virtio_queue_get_desc_size(vdev, idx), - 0, virtio_queue_get_desc_size(vdev, idx)); + vhost_virtqueue_memory_unmap(dev, vdev, vq, idx); } static int vhost_virtqueue_set_busyloop_timeout(struct vhost_dev *dev, @@ -1921,6 +1989,43 @@ static int vhost_dev_set_vring_enable(struct vhost_dev *hdev, int enable) return hdev->vhost_ops->vhost_set_vring_enable(hdev, enable); } +int vhost_dev_presetup(struct vhost_dev *hdev, VirtIODevice *vdev) +{ + int i, r; + + /* should only be called after backend is connected */ + assert(hdev->vhost_ops); + + r = vhost_dev_set_features(hdev, hdev->log_enabled); + if (r < 0) { + return r; + } + + r = hdev->vhost_ops->vhost_set_mem_table(hdev, hdev->mem); + if (r < 0) { + VHOST_OPS_DEBUG(r, "vhost_set_mem_table failed"); + return r; + } + + for (i = 0; i < hdev->nvqs; ++i) { + r = vhost_virtqueue_vring_setup(hdev, vdev, + hdev->vqs + i, + hdev->vq_index + i); + if (r < 0) { + VHOST_OPS_DEBUG(r, "vhost_virtqueue_setup failed"); + return r; + } + r = vhost_virtqueue_disable_notify(hdev, vdev, + hdev->vqs + i, + hdev->vq_index + i); + if (r < 0) { + return r; + } + } + + return 0; +} + /* Host notifiers must be enabled at this point. */ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings) { @@ -2087,3 +2192,12 @@ int vhost_net_set_backend(struct vhost_dev *hdev, return -ENOSYS; } + +int vhost_dev_set_presetup_state(struct vhost_dev *hdev, bool start) +{ + if (!hdev->vhost_ops->vhost_presetup) { + return -ENOTSUP; + } + + return hdev->vhost_ops->vhost_presetup(hdev, start); +} diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index 6a173cb9fa..95a8031d12 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -192,6 +192,17 @@ void vhost_dev_disable_notifiers(struct vhost_dev *hdev, VirtIODevice *vdev); bool vhost_config_pending(struct vhost_dev *hdev); void vhost_config_mask(struct vhost_dev *hdev, VirtIODevice *vdev, bool mask); +/** + * vhost_dev_presetup() - pre-setup the vhost device in LM + * @hdev: common vhost_dev structure + * @vdev: the VirtIODevice structure + * + * During live migration, send out device information to backend in early + * running state. Backend can have enough time to prepare HW. + * Return: 0 on success, < 0 on error. + */ +int vhost_dev_presetup(struct vhost_dev *hdev, VirtIODevice *vdev); + /** * vhost_dev_is_started() - report status of vhost device * @hdev: common vhost_dev structure @@ -338,4 +349,5 @@ int vhost_dev_set_inflight(struct vhost_dev *dev, int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size, struct vhost_inflight *inflight); bool vhost_dev_has_iommu(struct vhost_dev *dev); +int vhost_dev_set_presetup_state(struct vhost_dev *hdev, bool start); #endif From patchwork Mon Sep 18 04:49:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 13388811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E162ECD37B0 for ; Mon, 18 Sep 2023 04:55:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qi6I2-00076D-QO; Mon, 18 Sep 2023 00:55:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6Hw-00075o-R5 for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:55:12 -0400 Received: from mail-dm6nam10on2075.outbound.protection.outlook.com ([40.107.93.75] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6Hu-0004db-Tq for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:55:12 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nS354S5IG6IWbIvG49BoD2AQX/+LqUP7+NpE1vAoZJVu25mof6pIfshFBTNKwO5um/3vfLnBroOQdE40pbjC21apHlRV9Lfbc3IwaIcTA7mZZVDLsYr5ANuWLctbjLJ0yZ1aUYXPCyj7GNFxB2/QRv65wIue1lk3vcEyxWynyjCvBlskubBPlpWPIC/bLEVGanUZ+tzqiGKxjJEHsMt/zjG0Ho9DwTsKwacfmnFDpQbmdt1ExhxNmteYxjLZsy4HmQM0i9w2sszchqq98AwFiX+HURIWE1WvZd6IE94ednpqJrV5bXCHHf5hmorwr5yZLkTlhYHZNaz8SSsa5ac1RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8YDYZQT7OpYs1vSYzvuaW0SukIHtMZmhmYYrL/tx6Bc=; b=OySBB/Hx7HSWcGHSHsneY6DAC6N/yFtHrWArwDEvrMILdAPP0pATYHFWd43P8tv7UOaY03GHrJpbu3m6TZZBoEjPxHG2t0sDCoprL0LS0w0H6cx+LpGIbwmZc25lqspJmyI8iJU8W8nFxk5qdPm/NxFoqakQe7XcayH6+ih+i6SI8s19qqM02cmzcyO0jGxTYVz5NBeH1I0+JqzubnQtbGk0qgguBB6CU40aCsPnIppirNidrp/Hj/fr/sPpj3Ma4EJrV1cmkSQym+uxoxg3ef6bHoSuOTlMwz40fk6yt98ZvuCA9jfWawbS267FBb7rSaFblggglAgn6LpaHTmg1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8YDYZQT7OpYs1vSYzvuaW0SukIHtMZmhmYYrL/tx6Bc=; b=swVZVrAtUFebpv+s1RVgOHwHPVB707zQLUo7Kabd0wKQUyeSqndS17bCy9g8J8lP2tlMMiozCRUWki2hKXqiGzy7W3jREwyWWbyxw1lBhHw8KJyw5JyeMFOTRFeLNALdI3+QZNOq+eLW2dslcWMn8fG3J4z1sE8mB6d/pPLolzSvDrXEaDEMlw9xBINk3KHMhegNQxodAsOk/wLkNxYbQWya8iuMmXVYG/bVBdDQhYs/qu7IyJtAZ9FSQPRSVjegSatOP5FH+lIomMYAbKz1P3tqps/adwUGbPeP/pkgtKo8zU3bMJELDR/zT5EhNHUyvb9Ct5IhIGtKr/LU8GDlxg== Received: from CY5PR22CA0029.namprd22.prod.outlook.com (2603:10b6:930:16::22) by IA1PR12MB8539.namprd12.prod.outlook.com (2603:10b6:208:446::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 04:50:04 +0000 Received: from CY4PEPF0000EE33.namprd05.prod.outlook.com (2603:10b6:930:16:cafe::a8) by CY5PR22CA0029.outlook.office365.com (2603:10b6:930:16::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.27 via Frontend Transport; Mon, 18 Sep 2023 04:50:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000EE33.mail.protection.outlook.com (10.167.242.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Mon, 18 Sep 2023 04:50:03 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:58 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:56 -0700 From: Yajun Wu To: , , , CC: Avihai Horon , Jiri Pirko Subject: [RFC PATCH 3/5] vhost-net: Add support for presetup Date: Mon, 18 Sep 2023 12:49:30 +0800 Message-ID: <20230918044932.1433744-4-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230918044932.1433744-1-yajunw@nvidia.com> References: <20230918044932.1433744-1-yajunw@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail202.nvidia.com (10.129.68.7) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE33:EE_|IA1PR12MB8539:EE_ X-MS-Office365-Filtering-Correlation-Id: ef03b8d4-a250-4811-27e2-08dbb802b941 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dPTtZbSX6kN2kdtLFWr522hctzs2vFjIRrmuGgbmi4J45hL2dK3SZcvj05R2A0+kcm4X4pBl70RYBS/cy4yIY412geUm9R48UKWbEEYjxnmhPPtWagF6GOFbob4sOTC6tJ4O9+DPzEoAGZOuAfiVN6tRhLh3kiGNJ+DWIiLl63h7JIINFCUNawnrypE1kD03GtoFmwf4egf3ntLfPIa9DPkp2UKasdQjrG1DbsXgDPm4H7RCd5XSTQtIiOYzM7YanS01JQn8oywHJNdKRthVsgW2zdFXsccaU/w/TSf0sUn9MIreiaDoDCeWwiyRD5EwKoglFno8nzS1drWAB/Pa1rVvUXE06hGDfXVHjBo9wjVNb5d3hrzusZWS4h5+4yI1/XG1IhfY9tvnTn5SWf5oEGOI4SJbr8fWMqfniCsuMhFG5CoXhU8A8CgwBhz2c5LFkxLCxab4RoqoUw/v967uhl+IcYcAFJgPqtYS1mCYcMjZLDyyQ+j2XevLqnTevqI3Yvwu/dELj/VbD9BpGCdUXEIsy+QCHzlQmTIVC6hWTX4HtEl4lVin49cWafYq7bz+4MuUTFzApLqSvB13FsorMPzpluNGjiFHciqWMG7qRgMSpbxyI9gZpTGnE0JPkOv4ElkYzG47FP0AZU5jeh4lLiTcE5itJKT+C+RdMHVqEpjlsmMeJqnDWoDb/QjQGyTYl9Ks8R2QfLLxp/KaR2ZFbnhA/7aPVkdm8ye0fjdd7WD34cr5ttOaZSs9SY8d3OWV X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(396003)(376002)(346002)(136003)(39860400002)(451199024)(186009)(1800799009)(82310400011)(36840700001)(46966006)(40470700004)(316002)(7049001)(8936002)(4326008)(8676002)(41300700001)(55016003)(107886003)(1076003)(26005)(16526019)(426003)(2616005)(6286002)(83380400001)(336012)(478600001)(7696005)(36756003)(6666004)(356005)(82740400003)(7636003)(86362001)(36860700001)(47076005)(110136005)(70586007)(70206006)(40460700003)(54906003)(2906002)(40480700001)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 04:50:03.9463 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ef03b8d4-a250-4811-27e2-08dbb802b941 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE33.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8539 Received-SPF: softfail client-ip=40.107.93.75; envelope-from=yajunw@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Introduce New API vhost_net_presetup to send virtio net device configuration to backend in LM setup. Mainly calling vhost_dev_presetup, then sending out vring enable. Signed-off-by: Yajun Wu Reviewed-by: Avihai Horon Reviewed-by: Jiri Pirko --- hw/net/vhost_net.c | 40 ++++++++++++++++++++++++++++++++++++++++ include/net/vhost_net.h | 3 +++ 2 files changed, 43 insertions(+) diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 6b958d6363..dcb818ccf1 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -345,6 +345,46 @@ static void vhost_net_stop_one(struct vhost_net *net, vhost_dev_disable_notifiers(&net->dev, dev); } +int vhost_net_presetup(VirtIODevice *dev, NetClientState *ncs, + int data_queue_pairs, int cvq) +{ + VirtIONet *n = VIRTIO_NET(dev); + int nvhosts = data_queue_pairs + cvq; + struct vhost_net *net; + int r = 0, i, index_end = data_queue_pairs * 2; + NetClientState *peer; + + if (cvq) { + index_end += 1; + } + + for (i = 0; i < nvhosts; i++) { + if (i < data_queue_pairs) { + peer = qemu_get_peer(ncs, i); + } else { /* Control Virtqueue */ + peer = qemu_get_peer(ncs, n->max_queue_pairs); + } + + net = get_vhost_net(peer); + vhost_net_set_vq_index(net, i * 2, index_end); + + r = vhost_dev_presetup(&net->dev, dev); + if (r < 0) { + return r; + } + + if (peer->vring_enable) { + /* restore vring enable state */ + r = vhost_set_vring_enable(peer, peer->vring_enable); + if (r < 0) { + return r; + } + } + } + + return r; +} + int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, int data_queue_pairs, int cvq) { diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h index c37aba35e6..2c9020c5a2 100644 --- a/include/net/vhost_net.h +++ b/include/net/vhost_net.h @@ -26,6 +26,9 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, void vhost_net_stop(VirtIODevice *dev, NetClientState *ncs, int data_queue_pairs, int cvq); +int vhost_net_presetup(VirtIODevice *dev, NetClientState *ncs, + int data_queue_pairs, int cvq); + void vhost_net_cleanup(VHostNetState *net); uint64_t vhost_net_get_features(VHostNetState *net, uint64_t features); From patchwork Mon Sep 18 04:49:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 13388797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0757BCD37B0 for ; Mon, 18 Sep 2023 04:51:09 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qi6DE-0003PV-M5; Mon, 18 Sep 2023 00:50:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6DB-0003PG-JZ for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:50:17 -0400 Received: from mail-dm6nam10on20627.outbound.protection.outlook.com ([2a01:111:f400:7e88::627] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6D7-0003ca-Jv for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:50:16 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FsQf+AYnIPKe9110bw0vYisIqlXcglkTbZXdRc9Za00dnX1biODPJg14FhMXMKR79H+2JBM8MP+o3OSoAXLIRqySzj3nZ+WjC7OVqb2tUahky4MMy6hX83frr5TkvE0F8tuYbREvPY9yYrV89fWlgWJqvtLWEERvrNYyYVhJeBFBIF9FOlILFKPhbkFiCS17o+KklqXOpb1u3tK5xtIeIILtfJptI98KgRzeySMantRc8e7U56/GWhs5myS2SopXN/41a7nSiH/TDj465AMzH2HQsiaT6AXfoDo7jDvYMeyZ0azWJyS6Il/KRq2KbjMi+MlSHuoJT8n5RiFdOWafSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zc94vzKvovPU3QIUbycGTkCqSy0UJtxZMcuTJ1j+Fd8=; b=MVRt0KrAbXk2wnvtQ8w5pPmDaqK8deLuD8Io4sXTJ6mRdQS2ieKQVlRBGSOj0cQOur4MjIn1h9RP2DebyYVCSOBCuzr3DCEVdUqcQeNwzT4pswBO4UcrzjWTeXryYWU5p17FTFotDWCItevR3UOGx+vfIZhElyyREMPB0VJxPk+OmML0kbo0EV8U3IlPyE5K3L9MzCA93mgpqFD51cDmRocwIofL76jHKhoJziYKf1Jcoa6D1/sh2Bx+aF30yxpokmyWD/xWhU/BxK+pn/YlsOu8cmZPYyg+5e+bivwRrS7U7PFB3wFn6XS1uAaEG81OwxnowXPJVMWOtZu1AR61ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zc94vzKvovPU3QIUbycGTkCqSy0UJtxZMcuTJ1j+Fd8=; b=HQ9aL0tvXMQBUQAVVmy+bE7VVPnbSsE1S27qZmL8P20hFYRiQjLSCsHgszT43BjkAbRNyRJ6ONbZlgHfGAbXcGfRWyMC5rit5TOsqj75xTyoekPqsFf+jrHxWaBoRfsSXR81ucTDI1+fALA0G46NGRv9RR5j54/NvENDrfyKohVCHGxi1yOFGNLnASxu3Z8sIvaU7gBs7/IhxLCShfQTbbgtZOhPXAo92GEne2kBYZaGpw90qN31KndvXvDj1Lo+DGwo90rDr9iV3mNvLlKDmpQyi357MvmSV9FvulGxL/7PiJFoceFNTSq9+japZ2/djiEusisrwUidWCqbkjFtjA== Received: from CY5PR22CA0012.namprd22.prod.outlook.com (2603:10b6:930:16::32) by IA1PR12MB8078.namprd12.prod.outlook.com (2603:10b6:208:3f1::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 04:50:08 +0000 Received: from CY4PEPF0000EE33.namprd05.prod.outlook.com (2603:10b6:930:16:cafe::3c) by CY5PR22CA0012.outlook.office365.com (2603:10b6:930:16::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26 via Frontend Transport; Mon, 18 Sep 2023 04:50:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000EE33.mail.protection.outlook.com (10.167.242.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Mon, 18 Sep 2023 04:50:07 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:50:01 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:49:59 -0700 From: Yajun Wu To: , , , CC: Avihai Horon , Jiri Pirko Subject: [RFC PATCH 4/5] virtio: Add VMState for early load Date: Mon, 18 Sep 2023 12:49:31 +0800 Message-ID: <20230918044932.1433744-5-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230918044932.1433744-1-yajunw@nvidia.com> References: <20230918044932.1433744-1-yajunw@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail202.nvidia.com (10.129.68.7) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE33:EE_|IA1PR12MB8078:EE_ X-MS-Office365-Filtering-Correlation-Id: 317d5241-e2a9-460b-996b-08dbb802bb59 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Nkr3bO3lNmYEwNgaJbpVwii5XzRiqhWRzNyozMAvw+ov4Uc7Jc67BjcJGO9pmVCNYzl2xW7mvYT6/ZzEFdFg2VSZgTa5q4k1UGsqkj37srLJP0r/5W6PxSz3PwLuSjQKWpssVrZ2e/+Yvg8wUmig3YoksqfuuDSi9J89aU9Y92+FmzewqPvMcvqX5tBdQMaG8QQ9gqZZsU9ilhPmzM71HZxPQiNYVMn8h0pILcSiy23pKr7cUxlT+lsn64UdjR3iq7FqbzIJ7oFzisFDm0LjfDR66E7Lw+DqJ7zLqUrzmmLarWYhnht3z2sCxbHygbgi+LGllIuObP67bN1VdnhOCtzAos1JfwpbvwIGlddeDz6JGyO89WZEbE0Z9zy4Q8rqTyEtS2Txu2EH7WhhWiDYqPTqYs5/6pk8XzYZwhZNN8Qgm4Gmps1mPeGo76B2hJZAaMDimCgXtrcBYbj9QXszL7/ekNwM/a1JBmc8+u4uGj2iP3y7WlO4JC1piXdxpD4s6pc1/0WOT6eB8Mn9Wemf6PT1rzUJEgTRZlIPxksehZO3dp86TeLlVT6AEGTokNiBbGwzjD5zLH5ddLdC7G+pvTPdJoCNSby1pd6YYuWc7thjiTRs/pN7RRea5FaWy3X92DuIstjSC85e+fMH1DbC+CtHY678QfGncQuf5x1raaTlYOgFgC7bHznSySqDpSUOu7ZJTk2cXyxuuLHBOWPKLml34lBsiWe1JPVGtTeUtr+u5pPgRSkRpDLIXEWjqr/v X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(376002)(396003)(39860400002)(346002)(136003)(82310400011)(1800799009)(186009)(451199024)(46966006)(40470700004)(36840700001)(7049001)(6666004)(7696005)(478600001)(26005)(6286002)(1076003)(16526019)(2616005)(107886003)(336012)(426003)(47076005)(2906002)(316002)(70206006)(54906003)(70586007)(110136005)(41300700001)(4326008)(8676002)(8936002)(5660300002)(40460700003)(82740400003)(356005)(36860700001)(40480700001)(36756003)(86362001)(7636003)(55016003)(83380400001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 04:50:07.4620 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 317d5241-e2a9-460b-996b-08dbb802bb59 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE33.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8078 Received-SPF: softfail client-ip=2a01:111:f400:7e88::627; envelope-from=yajunw@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define new virtio device vmstate for early save/load (happens in LM setup stage). Same as original vmstate, except: In LM setup phase of the destination VM, the guest memory has not been synchronized yet. To address this, a flag has been added to virtio_load function to skip the index check. Signed-off-by: Yajun Wu Reviewed-by: Avihai Horon Reviewed-by: Jiri Pirko --- hw/virtio/virtio.c | 152 +++++++++++++++++++++++-------------- include/hw/virtio/virtio.h | 10 ++- 2 files changed, 103 insertions(+), 59 deletions(-) diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c index 969c25f4cf..8c73c245dd 100644 --- a/hw/virtio/virtio.c +++ b/hw/virtio/virtio.c @@ -2832,7 +2832,17 @@ virtio_device_get(QEMUFile *f, void *opaque, size_t size, VirtIODevice *vdev = VIRTIO_DEVICE(opaque); DeviceClass *dc = DEVICE_CLASS(VIRTIO_DEVICE_GET_CLASS(vdev)); - return virtio_load(vdev, f, dc->vmsd->version_id); + return virtio_load(vdev, f, dc->vmsd->version_id, false); +} + +/* A wrapper for use as a VMState .get function */ +static int virtio_early_device_get(QEMUFile *f, void *opaque, size_t size, + const VMStateField *field) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(opaque); + DeviceClass *dc = DEVICE_CLASS(VIRTIO_DEVICE_GET_CLASS(vdev)); + + return virtio_load(vdev, f, dc->vmsd->version_id, true); } const VMStateInfo virtio_vmstate_info = { @@ -2841,6 +2851,12 @@ const VMStateInfo virtio_vmstate_info = { .put = virtio_device_put, }; +const VMStateInfo virtio_early_vmstate_info = { + .name = "virtio-early", + .get = virtio_early_device_get, + .put = virtio_device_put, +}; + static int virtio_set_features_nocheck(VirtIODevice *vdev, uint64_t val) { VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev); @@ -2940,8 +2956,75 @@ size_t virtio_get_config_size(const VirtIOConfigSizeParams *params, return config_size; } +static int virtio_load_check_index(VirtIODevice *vdev, int num) +{ + int i; + + RCU_READ_LOCK_GUARD(); + + for (i = 0; i < num; i++) { + if (vdev->vq[i].vring.desc) { + uint16_t nheads; + + /* + * VIRTIO-1 devices migrate desc, used, and avail ring addresses so + * only the region cache needs to be set up. Legacy devices need + * to calculate used and avail ring addresses based on the desc + * address. + */ + if (virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1)) { + virtio_init_region_cache(vdev, i); + } else { + virtio_queue_update_rings(vdev, i); + } + + if (virtio_vdev_has_feature(vdev, VIRTIO_F_RING_PACKED)) { + vdev->vq[i].shadow_avail_idx = vdev->vq[i].last_avail_idx; + vdev->vq[i].shadow_avail_wrap_counter = + vdev->vq[i].last_avail_wrap_counter; + continue; + } + + nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx; + /* Check it isn't doing strange things with descriptor numbers. */ + if (nheads > vdev->vq[i].vring.num) { + virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x " + "inconsistent with Host index 0x%x: delta 0x%x", + i, vdev->vq[i].vring.num, + vring_avail_idx(&vdev->vq[i]), + vdev->vq[i].last_avail_idx, nheads); + vdev->vq[i].used_idx = 0; + vdev->vq[i].shadow_avail_idx = 0; + vdev->vq[i].inuse = 0; + continue; + } + vdev->vq[i].used_idx = vring_used_idx(&vdev->vq[i]); + vdev->vq[i].shadow_avail_idx = vring_avail_idx(&vdev->vq[i]); + + /* + * Some devices migrate VirtQueueElements that have been popped + * from the avail ring but not yet returned to the used ring. + * Since max ring size < UINT16_MAX it's safe to use modulo + * UINT16_MAX + 1 subtraction. + */ + vdev->vq[i].inuse = (uint16_t)(vdev->vq[i].last_avail_idx - + vdev->vq[i].used_idx); + if (vdev->vq[i].inuse > vdev->vq[i].vring.num) { + error_report("VQ %d size 0x%x < last_avail_idx 0x%x - " + "used_idx 0x%x", + i, vdev->vq[i].vring.num, + vdev->vq[i].last_avail_idx, + vdev->vq[i].used_idx); + return -1; + } + } + } + + return 0; +} + int coroutine_mixed_fn -virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id) +virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id, bool early) { int i, ret; int32_t config_len; @@ -3078,62 +3161,15 @@ virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id) vdev->start_on_kick = true; } - RCU_READ_LOCK_GUARD(); - for (i = 0; i < num; i++) { - if (vdev->vq[i].vring.desc) { - uint16_t nheads; - - /* - * VIRTIO-1 devices migrate desc, used, and avail ring addresses so - * only the region cache needs to be set up. Legacy devices need - * to calculate used and avail ring addresses based on the desc - * address. - */ - if (virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1)) { - virtio_init_region_cache(vdev, i); - } else { - virtio_queue_update_rings(vdev, i); - } - - if (virtio_vdev_has_feature(vdev, VIRTIO_F_RING_PACKED)) { - vdev->vq[i].shadow_avail_idx = vdev->vq[i].last_avail_idx; - vdev->vq[i].shadow_avail_wrap_counter = - vdev->vq[i].last_avail_wrap_counter; - continue; - } - - nheads = vring_avail_idx(&vdev->vq[i]) - vdev->vq[i].last_avail_idx; - /* Check it isn't doing strange things with descriptor numbers. */ - if (nheads > vdev->vq[i].vring.num) { - virtio_error(vdev, "VQ %d size 0x%x Guest index 0x%x " - "inconsistent with Host index 0x%x: delta 0x%x", - i, vdev->vq[i].vring.num, - vring_avail_idx(&vdev->vq[i]), - vdev->vq[i].last_avail_idx, nheads); - vdev->vq[i].used_idx = 0; - vdev->vq[i].shadow_avail_idx = 0; - vdev->vq[i].inuse = 0; - continue; - } - vdev->vq[i].used_idx = vring_used_idx(&vdev->vq[i]); - vdev->vq[i].shadow_avail_idx = vring_avail_idx(&vdev->vq[i]); - - /* - * Some devices migrate VirtQueueElements that have been popped - * from the avail ring but not yet returned to the used ring. - * Since max ring size < UINT16_MAX it's safe to use modulo - * UINT16_MAX + 1 subtraction. - */ - vdev->vq[i].inuse = (uint16_t)(vdev->vq[i].last_avail_idx - - vdev->vq[i].used_idx); - if (vdev->vq[i].inuse > vdev->vq[i].vring.num) { - error_report("VQ %d size 0x%x < last_avail_idx 0x%x - " - "used_idx 0x%x", - i, vdev->vq[i].vring.num, - vdev->vq[i].last_avail_idx, - vdev->vq[i].used_idx); - return -1; - } + /* + * Early setup happens in LM setup stage when the guest memory hasn't + * synced to target VM yet. So skip all guest memory access and index check + * in early load. + */ + if (!early) { + ret = virtio_load_check_index(vdev, num); + if (ret) { + return ret; } } diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index c8f72850bc..c9e6faf72c 100644 --- a/include/hw/virtio/virtio.h +++ b/include/hw/virtio/virtio.h @@ -280,6 +280,7 @@ void virtio_notify(VirtIODevice *vdev, VirtQueue *vq); int virtio_save(VirtIODevice *vdev, QEMUFile *f); extern const VMStateInfo virtio_vmstate_info; +extern const VMStateInfo virtio_early_vmstate_info; #define VMSTATE_VIRTIO_DEVICE \ { \ @@ -288,7 +289,14 @@ extern const VMStateInfo virtio_vmstate_info; .flags = VMS_SINGLE, \ } -int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id); +#define VMSTATE_EARLY_VIRTIO_DEVICE \ + { \ + .name = "virtio-early", \ + .info = &virtio_early_vmstate_info,\ + .flags = VMS_SINGLE, \ + } + +int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id, bool early); /** * virtio_notify_config() - signal a change to device config From patchwork Mon Sep 18 04:49:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 13388798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 02AD3CD37B0 for ; Mon, 18 Sep 2023 04:51:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qi6DJ-0003RC-Qr; Mon, 18 Sep 2023 00:50:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6DD-0003Pc-Hf for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:50:20 -0400 Received: from mail-bn8nam11on20601.outbound.protection.outlook.com ([2a01:111:f400:7eae::601] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6D8-0003d6-Uq for qemu-devel@nongnu.org; Mon, 18 Sep 2023 00:50:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qj4dcrVrAOarWfHbTelugtfgy7XUHCMhBdWpXAgTOIHMBNFtl/h51gUpsMfjaTZOLRsg4H/k/wByaFzqvN9N4z2U+G/iGDYyzgMcjPJ9TdpyIIjK4zWU9JrCHSyuqfDe7IUaexONoqx9V7vGys9vMTcl0FliWjZetGPFld/OrpdsnCPWBTOBhFKKxxkHh4WA+pyxxlo5S1WUSqbHJB/VIvcXgRdtzYCVmw/FCmBh5bHmpS4sRwZTFRyf0a3CVIKyJhelakmuR0Gu9sSckVGGKB1+1a9zuBpdbX8H712XHDC7Dj2j99KNVWecZ/Mr759DMEER3Nzcm10vAXZ6AyaoGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WSpfpC5j95uMVKOZEhNck1dD0TKKMDMXlNqXac/uXCA=; b=c+6eFy/RteMueDieKCOhVvLqFtsPtc6i1VbL/HM6Z8WbnRy3A+YhM9zGd0vuz6EPC4lxJLWUqJrpMy32keZKsE2JtdjDu9u8p5rkDkGu8SUv69UqRlZvvuHht1x7Y6tYktJWCX6JP6CWBBTwXEPWQ7Weq4iVxyDnf/Q0R/PU+rb/iHzOz/2Cfe5e0HeEtjOf0rAwKHoaPynkDb//+2RbEGKe2jlfc9eUGSwJD8f0364TWToqf081mbXQg9npagfRsXZ8gZngB2T48XMDbqaJm/brVK0cUHH2T7jD0skMeCLOYbFmwCoDi6QSI1v+ZHBV8tRmtty7M0In0hlOLwmkig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WSpfpC5j95uMVKOZEhNck1dD0TKKMDMXlNqXac/uXCA=; b=KTbDffWK5Qv113K/0Mm8X1QmHQI7TwRsqQIDL9pI9AtM+qLVr9TPDTXZVeL0FZkI3ARZ6EdAEujyRtUZJo5PmmtFyxyXRG+Jt7H9ocQOrYEd8wMC3iSxaP29m+JCMWVGyeJuIXMHeN9IywsVnV/5JFLYGqkEXgrAiIw6fWF8LwxUU/2fcbsNx7T5Cj2eT2p2Y1mKqTmvU1Z/ue4nZDyjfOBTNq+2YCbpdydEl8GNARKMqh8BZ31BhMTY+byXqAPq+ChaUwxH1KVkgv0dIwQSHOmCSGZq/JBL+4AaPm/Ui+8pTP0b9delB9fgDvNkbhTH9H5Hca5Bv1NMOMMdH2FWVw== Received: from CY5P221CA0151.NAMP221.PROD.OUTLOOK.COM (2603:10b6:930:6a::24) by SA1PR12MB6917.namprd12.prod.outlook.com (2603:10b6:806:24c::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 04:50:11 +0000 Received: from CY4PEPF0000EE34.namprd05.prod.outlook.com (2603:10b6:930:6a:cafe::29) by CY5P221CA0151.outlook.office365.com (2603:10b6:930:6a::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26 via Frontend Transport; Mon, 18 Sep 2023 04:50:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000EE34.mail.protection.outlook.com (10.167.242.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Mon, 18 Sep 2023 04:50:10 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:50:03 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sun, 17 Sep 2023 21:50:01 -0700 From: Yajun Wu To: , , , CC: Avihai Horon , Jiri Pirko Subject: [RFC PATCH 5/5] virtio-net: Introduce LM early load Date: Mon, 18 Sep 2023 12:49:32 +0800 Message-ID: <20230918044932.1433744-6-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230918044932.1433744-1-yajunw@nvidia.com> References: <20230918044932.1433744-1-yajunw@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail202.nvidia.com (10.129.68.7) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE34:EE_|SA1PR12MB6917:EE_ X-MS-Office365-Filtering-Correlation-Id: 50a78591-2376-43fd-af21-08dbb802bd72 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RizkzscPUMcCUac5LZCCVNd5swmOHGDvhDfcoaGWsDm/cxh468R16XvlxAfJGoGz304+FC9EQSpxo/4pVbVKrBlMqNOEfXRq4T4bQPpDHUkcEB83DFHjxiuA2D/QMh8G12qEsWgTgXyhrwgs6yaY1msZ3NlABcWZY5Cnp/y20v08oRAsbSzC5k9MmegK2PPFft5n3pHxt7WIz4419EFkhe6LH2nJlhZB70aZ7xhiPZ96B7fZQRI54WSbp6H5ICfOXfrERBBQ/7PhFDzsXKon1mROkn3jl838Sqhqe74UCXx5U2DTj1essq+rymrOEzJEUe2HZK7iuS/EsCxxoh60ouuQ8YNMwPguaIusNeTz/YipyH/CANf9kcT+t3U+y9xbyAsWIZX3EeoGVBFH2KLypr3OQ5bVzK9kq58ksYIkmUoRW19IUADJE4T3XGbh8G0/yIXNk2q/omAyiGci1haHfGRadILysLeumahseer83YU7u7jDMG0RGTK3VlH0P6mrNIYK8fpKJkAbZw7lnTo/MsJAwfqAIgxmOky+w1OSeCxKvGnA6F6+CuKqi8k5JMgpUtf+JajTd1VnBNA3t4X8fUstSxDnJXQDLFDZEUaujRT27yTApyx5xiPmRjfl+L8wo0KfgTs/EV1/nN3RbjWycSODaoUIm6JnQTIc7DnF8ux8aX8bv/yPtLFWWQ3BUC7Uq4+u5R1LTAgLNBMd72ZKYH8eqPIr4orAbdBjA2SRP6D3cWWX6X/ImlqzS7uuyu4q X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(396003)(39860400002)(346002)(376002)(136003)(82310400011)(1800799009)(186009)(451199024)(36840700001)(46966006)(40470700004)(7636003)(356005)(26005)(16526019)(82740400003)(6286002)(8936002)(2616005)(1076003)(8676002)(4326008)(40460700003)(107886003)(83380400001)(36860700001)(2906002)(36756003)(47076005)(426003)(336012)(40480700001)(55016003)(5660300002)(86362001)(7696005)(478600001)(6666004)(110136005)(316002)(54906003)(70206006)(70586007)(7049001)(41300700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 04:50:10.9818 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 50a78591-2376-43fd-af21-08dbb802bd72 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE34.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6917 Received-SPF: softfail client-ip=2a01:111:f400:7eae::601; envelope-from=yajunw@nvidia.com; helo=NAM11-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Register a new vmstate for virtio-net with an early_setup flag to send the device state during migration setup. This can reduce the migration downtime of a virtio-net device with a vhost-user backend. This feature is disabled by default and can be enabled by setting the "x-early-migration" device property to on. Signed-off-by: Yajun Wu Reviewed-by: Avihai Horon Reviewed-by: Jiri Pirko --- hw/net/trace-events | 1 + hw/net/virtio-net.c | 100 +++++++++++++++++++++++++++++++++ include/hw/virtio/virtio-net.h | 1 + 3 files changed, 102 insertions(+) diff --git a/hw/net/trace-events b/hw/net/trace-events index 6b5ba669a2..ec89229044 100644 --- a/hw/net/trace-events +++ b/hw/net/trace-events @@ -399,6 +399,7 @@ virtio_net_post_load_device(void) virtio_net_rss_disable(void) virtio_net_rss_error(const char *msg, uint32_t value) "%s, value 0x%08x" virtio_net_rss_enable(uint32_t p1, uint16_t p2, uint8_t p3) "hashes 0x%x, table of %d, key of %d" +virtio_net_load_early_setup(void) "" # tulip.c tulip_reg_write(uint64_t addr, const char *name, int size, uint64_t val) "addr 0x%02"PRIx64" (%s) size %d value 0x%08"PRIx64 diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 7102ec4817..d0b0cc2ffe 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -46,6 +46,7 @@ #include "net_rx_pkt.h" #include "hw/virtio/vhost.h" #include "sysemu/qtest.h" +#include "sysemu/runstate.h" #define VIRTIO_NET_VM_VERSION 11 @@ -3568,6 +3569,95 @@ static bool failover_hide_primary_device(DeviceListener *listener, return qatomic_read(&n->failover_primary_hidden); } +static int virtio_net_load_early_setup(void *opaque, int version_id) +{ + VirtIONet *n = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(n); + NetClientState *nc = qemu_get_queue(n->nic); + int queue_pairs = n->multiqueue ? n->max_queue_pairs : 1; + int cvq = virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) ? + n->max_ncs - n->max_queue_pairs : 0; + VHostNetState *net; + int r; + + assert(nc->peer); + assert(nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER); + + net = get_vhost_net(nc->peer); + assert(net); + assert(net->dev.vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER); + + trace_virtio_net_load_early_setup(); + + /* backend should support presetup */ + r = vhost_dev_set_presetup_state(&net->dev, true); + if (r < 0) { + error_report("Start presetup device fail: %d", r); + return r; + } + + if (virtio_has_feature(vdev->guest_features, VIRTIO_NET_F_MTU)) { + r = vhost_net_set_mtu(get_vhost_net(nc->peer), n->net_conf.mtu); + if (r < 0) { + error_report("%uBytes MTU not supported by the backend", + n->net_conf.mtu); + goto error; + } + } + + r = vhost_net_presetup(vdev, n->nic->ncs, queue_pairs, cvq); + if (r < 0) { + error_report("Presetup device fail: %d", r); + goto error; + } + + r = vhost_dev_set_presetup_state(&net->dev, false); + if (r < 0) { + error_report("Finish presetup device fail: %d", r); + return r; + } + return 0; + +error: + vhost_dev_set_presetup_state(&net->dev, false); + return r; +} + +static bool virtio_net_early_setup_needed(void *opaque) +{ + VirtIONet *n = opaque; + NetClientState *nc = qemu_get_queue(n->nic); + VHostNetState *net = get_vhost_net(nc->peer); + + /* + * Presetup aims to reduce live migration downtime by sync device + * status in setup stage. So only do presetup when source VM is in + * running state. + */ + if (runstate_is_running() && + nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_USER && + net->dev.vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER && + !vhost_dev_has_iommu(&net->dev) && + n->vhost_started && + n->status & VIRTIO_NET_S_LINK_UP) { + return true; + } + return false; +} + +static const VMStateDescription vmstate_virtio_net_early = { + .name = "virtio-net-early", + .minimum_version_id = VIRTIO_NET_VM_VERSION, + .version_id = VIRTIO_NET_VM_VERSION, + .fields = (VMStateField[]) { + VMSTATE_EARLY_VIRTIO_DEVICE, + VMSTATE_END_OF_LIST() + }, + .early_setup = true, + .post_load = virtio_net_load_early_setup, + .needed = virtio_net_early_setup_needed, +}; + static void virtio_net_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev = VIRTIO_DEVICE(dev); @@ -3743,6 +3833,11 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp) if (virtio_has_feature(n->host_features, VIRTIO_NET_F_RSS)) { virtio_net_load_ebpf(n); } + + if (n->early_migration) { + vmstate_register(NULL, VMSTATE_INSTANCE_ID_ANY, + &vmstate_virtio_net_early, n); + } } static void virtio_net_device_unrealize(DeviceState *dev) @@ -3787,6 +3882,10 @@ static void virtio_net_device_unrealize(DeviceState *dev) g_free(n->rss_data.indirections_table); net_rx_pkt_uninit(n->rx_pkt); virtio_cleanup(vdev); + + if (n->early_migration) { + vmstate_unregister(NULL, &vmstate_virtio_net_early, n); + } } static void virtio_net_instance_init(Object *obj) @@ -3922,6 +4021,7 @@ static Property virtio_net_properties[] = { DEFINE_PROP_INT32("speed", VirtIONet, net_conf.speed, SPEED_UNKNOWN), DEFINE_PROP_STRING("duplex", VirtIONet, net_conf.duplex_str), DEFINE_PROP_BOOL("failover", VirtIONet, failover, false), + DEFINE_PROP_BOOL("x-early-migration", VirtIONet, early_migration, false), DEFINE_PROP_END_OF_LIST(), }; diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h index e07a723027..9e6f90b46f 100644 --- a/include/hw/virtio/virtio-net.h +++ b/include/hw/virtio/virtio-net.h @@ -212,6 +212,7 @@ struct VirtIONet { /* primary failover device is hidden*/ bool failover_primary_hidden; bool failover; + bool early_migration; DeviceListener primary_listener; QDict *primary_opts; bool primary_opts_from_json;