From patchwork Mon Dec 5 17:04:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13064817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9DF03C4332F for ; Mon, 5 Dec 2022 17:05:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p2Etk-0000Se-SS; Mon, 05 Dec 2022 12:04:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p2Etd-0000Rr-Th for qemu-devel@nongnu.org; Mon, 05 Dec 2022 12:04:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p2Etc-0002eN-9k for qemu-devel@nongnu.org; Mon, 05 Dec 2022 12:04:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670259887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=9jhb79LTN1zIzvYKNu6vLgQd/A3YUmNVphhijZNBnm0=; b=JvFwDZB1UMeLHVNBw/HyNsFrMqHwy/Mv5SVAMJmRE1yGXw7Smoal3lLtqONbj2U0+ivQX3 VvrXRS/Q96Rl1YJu2vxQlYQhWQ7/oE1sHPlFzOhPZTyvWUXBTd2caBG3gnqrTcujFSzDvm MuuZ1j6uBbV396WAdvAAjIedf6/n0l8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-91-bFn1aNfDPZKqNsiEs5XcCw-1; Mon, 05 Dec 2022 12:04:44 -0500 X-MC-Unique: bFn1aNfDPZKqNsiEs5XcCw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3FD543C1022C; Mon, 5 Dec 2022 17:04:43 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.167]) by smtp.corp.redhat.com (Postfix) with ESMTP id 58416C15BA8; Mon, 5 Dec 2022 17:04:40 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Liuxiangdong , Stefan Hajnoczi , Jason Wang , Harpreet Singh Anand , Gautam Dawar , Zhu Lingshan , Cindy Lu , Si-Wei Liu , "Michael S. Tsirkin" , "Dr. David Alan Gilbert" , Laurent Vivier , Eli Cohen , Stefano Garzarella , Juan Quintela , Parav Pandit Subject: [RFC PATCH for 8.0 00/13] vDPA-net inflight descriptors migration with SVQ Date: Mon, 5 Dec 2022 18:04:23 +0100 Message-Id: <20221205170436.2977336-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The state of the descriptors (avail or used) may not be recoverable just looking at the guest memory. Out of order used descriptor may override previous avail ones in the descriptor table or avail vring. Currently we're not migrating this status in net devices because virtio-net, vhost-kernel etc use the descriptors in order, so the information always recoverable from guest's memory. However, vDPA devices may use them out of order, and other kind of devices like block need this support. Shadow virtqueue is able to track these and resend them at the destination. Add them to the virtio-net migration description so they are not lose in the process. This is a very early RFC just to validate the first draft so expect leftovers. To fetch and request the descriptors from a device without SVQ need to be implemented on top. Some other notable pending items are: * Do not send the descriptors actually recoverable from the guest memory. * Properly version the migrate data. * Properly abstract the descriptors access from virtio-net to SVQ. * Do not use VirtQueueElementOld but migrate directly VirtQueueElement. * Replace lots of assertions with runtime conditionals. * Other TODOs in the patch message or code changes. Thanks. Eugenio Pérez (13): vhost: add available descriptor list in SVQ vhost: iterate only available descriptors at SVQ stop vhost: merge avail list and next avail descriptors detach vhost: add vhost_svq_save_inflight virtio: Specify uint32_t as VirtQueueElementOld members type virtio: refactor qemu_get_virtqueue_element virtio: refactor qemu_put_virtqueue_element virtio: expose VirtQueueElementOld virtio: add vmstate_virtqueue_element_old virtio-net: Migrate vhost inflight descriptors virtio-net: save inflight descriptors at vhost shutdown vhost: expose vhost_svq_add_element vdpa: Recover inflight descriptors hw/virtio/vhost-shadow-virtqueue.h | 9 ++ include/hw/virtio/virtio-net.h | 2 + include/hw/virtio/virtio.h | 32 ++++++ include/migration/vmstate.h | 22 ++++ hw/net/vhost_net.c | 56 ++++++++++ hw/net/virtio-net.c | 129 +++++++++++++++++++++++ hw/virtio/vhost-shadow-virtqueue.c | 52 +++++++-- hw/virtio/vhost-vdpa.c | 11 -- hw/virtio/virtio.c | 162 ++++++++++++++++++----------- 9 files changed, 392 insertions(+), 83 deletions(-)