From patchwork Mon Mar 15 19:48:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 12140563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E1F3C433DB for ; Mon, 15 Mar 2021 19:51:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E12864E4D for ; Mon, 15 Mar 2021 19:51:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E12864E4D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50508 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLtEx-0006Jc-Uv for qemu-devel@archiver.kernel.org; Mon, 15 Mar 2021 15:50:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37544) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lLtD5-00050K-0T for qemu-devel@nongnu.org; Mon, 15 Mar 2021 15:49:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:42752) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1lLtD1-0003r4-88 for qemu-devel@nongnu.org; Mon, 15 Mar 2021 15:49:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615837736; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dyAcr8KFvX5ZXIu4EGLiiqNLB0VmrbBhQ/enTolLbEE=; b=Ou5MhtQFMkooj/wFarvlQ4Wp7pvNxPUyjx4WATZsfTmiD6oNlHsOj+EnqoYnWPiRMEcVwu fZtPnBf/nW4+ipNIm8K4PFMVnVyCpGflHosRXnPhOEgyOUbE/jgRA8i1QIkaf+1lNXWkH2 m22f6mF6dzogZJX9eIMsbwi2xanJ8pM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-572-VEkVYNdUPlq3truCdywumA-1; Mon, 15 Mar 2021 15:48:52 -0400 X-MC-Unique: VEkVYNdUPlq3truCdywumA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 091A319200C5; Mon, 15 Mar 2021 19:48:51 +0000 (UTC) Received: from eperezma.remote.csb (ovpn-112-173.ams2.redhat.com [10.36.112.173]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FEAB1C4; Mon, 15 Mar 2021 19:48:44 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC v2 00/13] vDPA software assisted live migration Date: Mon, 15 Mar 2021 20:48:29 +0100 Message-Id: <20210315194842.277740-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.25, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , "Michael S. Tsirkin" , Guru Prasad , Jason Wang , Juan Quintela , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Eli Cohen , Stefano Garzarella , Michael Lilja , Jim Harford , Rob Miller Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This series enable shadow virtqueue for vhost-net devices. This is a new method of vhost devices migration: Instead of relay on vhost device's dirty logging capability, SW assisted LM intercepts dataplane, forwarding the descriptors between VM and device. Is intended for vDPA devices with no logging, but this address the basic platform to build that support on. In this migration mode, qemu offers a new vring to the device to read and write into, and disable vhost notifiers, processing guest and vhost notifications in qemu. On used buffer relay, qemu will mark the dirty memory as with plain virtio-net devices. This way, devices does not need to have dirty page logging capability. This series is a POC doing SW LM for vhost-net devices, which already have dirty page logging capabilities. For qemu to use shadow virtqueues these vhost-net devices need to be instantiated: * With IOMMU (iommu_platform=on,ats=on) * Without event_idx (event_idx=off) And shadow virtqueue needs to be enabled for them with QMP command like: { "execute": "x-vhost-enable-shadow-vq", "arguments": { "name": "virtio-net", "enable": false } } Just the notification forwarding (with no descriptor relay) can be achieved with patches 5 and 6, and starting SVQ. Previous commits are cleanup ones and declaration of QMP command. Commit 11 introduces the buffer forwarding. Previous one are for preparations again, and laters are for enabling some obvious optimizations. It is based on the ideas of DPDK SW assisted LM, in the series of DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does not map the shadow vq in guest's VA, but in qemu's. Comments are welcome! Especially on: * Different/improved way of synchronization, particularly on the race of masking. TODO: * Event, indirect, packed, and others features of virtio - Waiting for confirmation of the big picture. * vDPA devices: Developing solutions for tracking the available IOVA space for all devices. Small POC available, skipping the get/set status (since vDPA does not support it) and just allocating more and more IOVA addresses in a hardcoded range available for the device. * To sepparate buffers forwarding in its own AIO context, so we can throw more threads to that task and we don't need to stop the main event loop. * IOMMU optimizations, so bacthing and bigger chunks of IOVA can be sent to device. * Automatic kick-in on live-migration. * Proper documentation. Thanks! Changes from v1 RFC: * Use QMP instead of migration to start SVQ mode. * Only accepting IOMMU devices, closer behavior with target devices (vDPA) * Fix invalid masking/unmasking of vhost call fd. * Use of proper methods for synchronization. * No need to modify VirtIO device code, all of the changes are contained in vhost code. * Delete superfluous code. * An intermediate RFC was sent with only the notifications forwarding changes. It can be seen in https://patchew.org/QEMU/20210129205415.876290-1-eperezma@redhat.com/ * v1 at https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.html Eugenio Pérez (13): virtio: Add virtio_queue_is_host_notifier_enabled vhost: Save masked_notifier state vhost: Add VhostShadowVirtqueue vhost: Add x-vhost-enable-shadow-vq qmp vhost: Route guest->host notification through shadow virtqueue vhost: Route host->guest notification through shadow virtqueue vhost: Avoid re-set masked notifier in shadow vq virtio: Add vhost_shadow_vq_get_vring_addr virtio: Add virtio_queue_full vhost: add vhost_kernel_set_vring_enable vhost: Shadow virtqueue buffers forwarding vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue kick vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow virtqueue qapi/net.json | 22 ++ hw/virtio/vhost-shadow-virtqueue.h | 36 ++ include/hw/virtio/vhost.h | 6 + include/hw/virtio/virtio.h | 3 + hw/virtio/vhost-backend.c | 29 ++ hw/virtio/vhost-shadow-virtqueue.c | 551 +++++++++++++++++++++++++++++ hw/virtio/vhost.c | 283 +++++++++++++++ hw/virtio/virtio.c | 23 +- hw/virtio/meson.build | 2 +- 9 files changed, 952 insertions(+), 3 deletions(-) create mode 100644 hw/virtio/vhost-shadow-virtqueue.h create mode 100644 hw/virtio/vhost-shadow-virtqueue.c