From patchwork Fri Feb 3 19:11:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57E7BC61DA4 for ; Fri, 3 Feb 2023 19:14:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U5-0006kD-MY; Fri, 03 Feb 2023 14:12:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1To-0006ZG-P4 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:13 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Ti-0007YH-V3 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XIlHBRUuqaWxdpypKS8k9nPXvNlZJx/L50KFWlh9jH8=; b=eh6F8wUbxNBQuwC8ootOUK8SRxe/lwItWU8jt00QDeGmTdbZXRGdzya7DAAc5J3Fy2Ggke 5vzio+q/O5J/qWUwhmaKBgsGVNi4H9VZPEHKySp8dKTBsduMDhz6treJXdO55c0r1XQ7st 4DuOQb8F1oYY/iFBJOTlmVQ8XB3EaaU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-208-5aRhQeBiOoiHQ2y-sePF2g-1; Fri, 03 Feb 2023 14:11:52 -0500 X-MC-Unique: 5aRhQeBiOoiHQ2y-sePF2g-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 475A4380606C; Fri, 3 Feb 2023 19:11:51 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1EA5492C14; Fri, 3 Feb 2023 19:11:47 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 01/13] vdpa net: move iova tree creation from init to start Date: Fri, 3 Feb 2023 20:11:30 +0100 Message-Id: <20230203191142.362623-2-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Only create iova_tree if and when it is needed. The cleanup keeps being responsible of last VQ but this change allows it to merge both cleanup functions. Signed-off-by: Eugenio Pérez Acked-by: Jason Wang --- net/vhost-vdpa.c | 99 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 71 insertions(+), 28 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index de5ed8ff22..a9e6c8f28e 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -178,13 +178,9 @@ err_init: static void vhost_vdpa_cleanup(NetClientState *nc) { VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); - struct vhost_dev *dev = &s->vhost_net->dev; qemu_vfree(s->cvq_cmd_out_buffer); qemu_vfree(s->status); - if (dev->vq_index + dev->nvqs == dev->vq_index_end) { - g_clear_pointer(&s->vhost_vdpa.iova_tree, vhost_iova_tree_delete); - } if (s->vhost_net) { vhost_net_cleanup(s->vhost_net); g_free(s->vhost_net); @@ -234,10 +230,64 @@ static ssize_t vhost_vdpa_receive(NetClientState *nc, const uint8_t *buf, return size; } +/** From any vdpa net client, get the netclient of first queue pair */ +static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s) +{ + NICState *nic = qemu_get_nic(s->nc.peer); + NetClientState *nc0 = qemu_get_peer(nic->ncs, 0); + + return DO_UPCAST(VhostVDPAState, nc, nc0); +} + +static void vhost_vdpa_net_data_start_first(VhostVDPAState *s) +{ + struct vhost_vdpa *v = &s->vhost_vdpa; + + if (v->shadow_vqs_enabled) { + v->iova_tree = vhost_iova_tree_new(v->iova_range.first, + v->iova_range.last); + } +} + +static int vhost_vdpa_net_data_start(NetClientState *nc) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + struct vhost_vdpa *v = &s->vhost_vdpa; + + assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + + if (v->index == 0) { + vhost_vdpa_net_data_start_first(s); + return 0; + } + + if (v->shadow_vqs_enabled) { + VhostVDPAState *s0 = vhost_vdpa_net_first_nc_vdpa(s); + v->iova_tree = s0->vhost_vdpa.iova_tree; + } + + return 0; +} + +static void vhost_vdpa_net_client_stop(NetClientState *nc) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + struct vhost_dev *dev; + + assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + + dev = s->vhost_vdpa.dev; + if (dev->vq_index + dev->nvqs == dev->vq_index_end) { + g_clear_pointer(&s->vhost_vdpa.iova_tree, vhost_iova_tree_delete); + } +} + static NetClientInfo net_vhost_vdpa_info = { .type = NET_CLIENT_DRIVER_VHOST_VDPA, .size = sizeof(VhostVDPAState), .receive = vhost_vdpa_receive, + .start = vhost_vdpa_net_data_start, + .stop = vhost_vdpa_net_client_stop, .cleanup = vhost_vdpa_cleanup, .has_vnet_hdr = vhost_vdpa_has_vnet_hdr, .has_ufo = vhost_vdpa_has_ufo, @@ -351,7 +401,7 @@ dma_map_err: static int vhost_vdpa_net_cvq_start(NetClientState *nc) { - VhostVDPAState *s; + VhostVDPAState *s, *s0; struct vhost_vdpa *v; uint64_t backend_features; int64_t cvq_group; @@ -425,6 +475,15 @@ out: return 0; } + s0 = vhost_vdpa_net_first_nc_vdpa(s); + if (s0->vhost_vdpa.iova_tree) { + /* SVQ is already configured for all virtqueues */ + v->iova_tree = s0->vhost_vdpa.iova_tree; + } else { + v->iova_tree = vhost_iova_tree_new(v->iova_range.first, + v->iova_range.last); + } + r = vhost_vdpa_cvq_map_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer, vhost_vdpa_net_cvq_cmd_page_len(), false); if (unlikely(r < 0)) { @@ -449,15 +508,9 @@ static void vhost_vdpa_net_cvq_stop(NetClientState *nc) if (s->vhost_vdpa.shadow_vqs_enabled) { vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer); vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->status); - if (!s->always_svq) { - /* - * If only the CVQ is shadowed we can delete this safely. - * If all the VQs are shadows this will be needed by the time the - * device is started again to register SVQ vrings and similar. - */ - g_clear_pointer(&s->vhost_vdpa.iova_tree, vhost_iova_tree_delete); - } } + + vhost_vdpa_net_client_stop(nc); } static ssize_t vhost_vdpa_net_cvq_add(VhostVDPAState *s, size_t out_len, @@ -667,8 +720,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, int nvqs, bool is_datapath, bool svq, - struct vhost_vdpa_iova_range iova_range, - VhostIOVATree *iova_tree) + struct vhost_vdpa_iova_range iova_range) { NetClientState *nc = NULL; VhostVDPAState *s; @@ -690,7 +742,6 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.shadow_vqs_enabled = svq; s->vhost_vdpa.iova_range = iova_range; s->vhost_vdpa.shadow_data = svq; - s->vhost_vdpa.iova_tree = iova_tree; if (!is_datapath) { s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(), vhost_vdpa_net_cvq_cmd_page_len()); @@ -760,7 +811,6 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, uint64_t features; int vdpa_device_fd; g_autofree NetClientState **ncs = NULL; - g_autoptr(VhostIOVATree) iova_tree = NULL; struct vhost_vdpa_iova_range iova_range; NetClientState *nc; int queue_pairs, r, i = 0, has_cvq = 0; @@ -812,12 +862,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, goto err; } - if (opts->x_svq) { - if (!vhost_vdpa_net_valid_svq_features(features, errp)) { - goto err_svq; - } - - iova_tree = vhost_iova_tree_new(iova_range.first, iova_range.last); + if (opts->x_svq && !vhost_vdpa_net_valid_svq_features(features, errp)) { + goto err; } ncs = g_malloc0(sizeof(*ncs) * queue_pairs); @@ -825,7 +871,7 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, for (i = 0; i < queue_pairs; i++) { ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, vdpa_device_fd, i, 2, true, opts->x_svq, - iova_range, iova_tree); + iova_range); if (!ncs[i]) goto err; } @@ -833,13 +879,11 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, if (has_cvq) { nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, vdpa_device_fd, i, 1, false, - opts->x_svq, iova_range, iova_tree); + opts->x_svq, iova_range); if (!nc) goto err; } - /* iova_tree ownership belongs to last NetClientState */ - g_steal_pointer(&iova_tree); return 0; err: @@ -849,7 +893,6 @@ err: } } -err_svq: qemu_close(vdpa_device_fd); return -1; From patchwork Fri Feb 3 19:11:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46601C05027 for ; Fri, 3 Feb 2023 19:14:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U6-0006la-Bc; Fri, 03 Feb 2023 14:12:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tr-0006al-Ea for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tn-0007YW-0U for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451518; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B0P82Kk1stpb6zZU8JsMQ6xnrrkGr7H7LKuOD9O/E50=; b=eVgt7l9wr3JD5ob08pZpqBf91D2oIG4ViBUuhzQheQy+SgBUivp9t9vuAAgYFAfdIMfLc3 nakCX+GfDAIbjjJ2XxCvw6kqYB4eek+ylMxTd72GRx6zBOwFbl27EzO/37fEEGZe6/cvqU /YppgALytS2gPmcaSXFd8/cSQvU/Spw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-177-34RmaiIYMM6sfEULRwTf2g-1; Fri, 03 Feb 2023 14:11:55 -0500 X-MC-Unique: 34RmaiIYMM6sfEULRwTf2g-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 681F285CBE0; Fri, 3 Feb 2023 19:11:54 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 86D4D492C14; Fri, 3 Feb 2023 19:11:51 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 02/13] vdpa: Negotiate _F_SUSPEND feature Date: Fri, 3 Feb 2023 20:11:31 +0100 Message-Id: <20230203191142.362623-3-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This is needed for qemu to know it can suspend the device to retrieve its status and enable SVQ with it, so all the process is transparent to the guest. Signed-off-by: Eugenio Pérez Acked-by: Jason Wang --- hw/virtio/vhost-vdpa.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 542e003101..2e79fbe4b2 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -659,7 +659,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) uint64_t features; uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 | 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | - 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID; + 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | + 0x1ULL << VHOST_BACKEND_F_SUSPEND; int r; if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) { From patchwork Fri Feb 3 19:11:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B59EC61DA4 for ; Fri, 3 Feb 2023 19:13:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U0-0006eR-3F; Fri, 03 Feb 2023 14:12:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tr-0006ai-Eg for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tn-0007Ya-0i for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451523; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t41R//2gXUTGFX/p6w1zeOJkuTeMOogk+bxpMqyKN7U=; b=GmYUk3XX1g20PCgNDMz8+4XcYeQhW5IyjHgRkaGBTSzpfkj9E1IvS3DSk+hh+kArkuyTs8 oSdWyXR69tVEVq1WqlRiF/fgWa6MUXVGA+RQHhnmh2cr2xF6qmNN7EYtEISclQgX2ap6h5 HBjLFY1c3xdWoyJP0BH0Z+mWUWdJBh8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-523-4Fz9PczwOLW3OQA--H8akQ-1; Fri, 03 Feb 2023 14:11:58 -0500 X-MC-Unique: 4Fz9PczwOLW3OQA--H8akQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 92FD180006E; Fri, 3 Feb 2023 19:11:57 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id A859D492C14; Fri, 3 Feb 2023 19:11:54 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 03/13] vdpa: add vhost_vdpa_suspend Date: Fri, 3 Feb 2023 20:11:32 +0100 Message-Id: <20230203191142.362623-4-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The function vhost.c:vhost_dev_stop fetches the vring base so the vq state can be migrated to other devices. However, this is unreliable in vdpa, since we didn't signal the device to suspend the queues, making the value fetched useless. Suspend the device if possible before fetching first and subsequent vring bases. Moreover, vdpa totally reset and wipes the device at the last device before fetch its vrings base, making that operation useless in the last device. This will be fixed in later patches of this series. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 19 +++++++++++++++++++ hw/virtio/trace-events | 1 + 2 files changed, 20 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 2e79fbe4b2..b681841cdc 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1108,6 +1108,24 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) } } +static void vhost_vdpa_suspend(struct vhost_dev *dev) +{ + struct vhost_vdpa *v = dev->opaque; + int r; + + if (!vhost_vdpa_first_dev(dev) || + !(dev->backend_features & BIT_ULL(VHOST_BACKEND_F_SUSPEND))) { + return; + } + + trace_vhost_vdpa_suspend(dev); + r = ioctl(v->device_fd, VHOST_VDPA_SUSPEND); + if (unlikely(r)) { + error_report("Cannot suspend: %s(%d)", g_strerror(errno), errno); + /* Not aborting since we're called from stop context */ + } +} + static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) { struct vhost_vdpa *v = dev->opaque; @@ -1122,6 +1140,7 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) } vhost_vdpa_set_vring_ready(dev); } else { + vhost_vdpa_suspend(dev); vhost_vdpa_svqs_stop(dev); vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs); } diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index a87c5f39a2..8f8d05cf9b 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -50,6 +50,7 @@ vhost_vdpa_set_vring_ready(void *dev) "dev: %p" vhost_vdpa_dump_config(void *dev, const char *line) "dev: %p %s" vhost_vdpa_set_config(void *dev, uint32_t offset, uint32_t size, uint32_t flags) "dev: %p offset: %"PRIu32" size: %"PRIu32" flags: 0x%"PRIx32 vhost_vdpa_get_config(void *dev, void *config, uint32_t config_len) "dev: %p config: %p config_len: %"PRIu32 +vhost_vdpa_suspend(void *dev) "dev: %p" vhost_vdpa_dev_start(void *dev, bool started) "dev: %p started: %d" vhost_vdpa_set_log_base(void *dev, uint64_t base, unsigned long long size, int refcnt, int fd, void *log) "dev: %p base: 0x%"PRIx64" size: %llu refcnt: %d fd: %d log: %p" vhost_vdpa_set_vring_addr(void *dev, unsigned int index, unsigned int flags, uint64_t desc_user_addr, uint64_t used_user_addr, uint64_t avail_user_addr, uint64_t log_guest_addr) "dev: %p index: %u flags: 0x%x desc_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" log_guest_addr: 0x%"PRIx64 From patchwork Fri Feb 3 19:11:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7BBF8C61DA4 for ; Fri, 3 Feb 2023 19:13:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U6-0006lw-Vv; Fri, 03 Feb 2023 14:12:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tr-0006aj-EE for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tm-0007Yj-UR for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451525; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xRR25skpeshYFqOBbCZA5w9c00eqzmw6VsmZSUlCj40=; b=GkBrvPkNd2nHUXRBGAU4HQp+Az3SDjUU5Y2tn5pwIVIIz/CB8Lhc/gIzSbNntPDlB3H6i/ rKQktqtDui3yuiXvTo+mVTqugOZyFJAiaUawYReECQqs3N8FY0ITptrQMiWYsuwfM4hj9u YvaNkslQNbdALfTJdOr8aDGntyVLvQY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-154-aNOk-EefPbWRTm_i9SZmWQ-1; Fri, 03 Feb 2023 14:12:01 -0500 X-MC-Unique: aNOk-EefPbWRTm_i9SZmWQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B4970100F907; Fri, 3 Feb 2023 19:12:00 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id D1BF5492C14; Fri, 3 Feb 2023 19:11:57 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 04/13] vdpa: move vhost reset after get vring base Date: Fri, 3 Feb 2023 20:11:33 +0100 Message-Id: <20230203191142.362623-5-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The function vhost.c:vhost_dev_stop calls vhost operation vhost_dev_start(false). In the case of vdpa it totally reset and wipes the device, making the fetching of the vring base (virtqueue state) totally useless. The kernel backend does not use vhost_dev_start vhost op callback, but vhost-user do. A patch to make vhost_user_dev_start more similar to vdpa is desirable, but it can be added on top. Signed-off-by: Eugenio Pérez --- include/hw/virtio/vhost-backend.h | 4 ++++ hw/virtio/vhost-vdpa.c | 22 ++++++++++++++++------ hw/virtio/vhost.c | 3 +++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index c5ab49051e..ec3fbae58d 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -130,6 +130,9 @@ typedef bool (*vhost_force_iommu_op)(struct vhost_dev *dev); typedef int (*vhost_set_config_call_op)(struct vhost_dev *dev, int fd); + +typedef void (*vhost_reset_status_op)(struct vhost_dev *dev); + typedef struct VhostOps { VhostBackendType backend_type; vhost_backend_init vhost_backend_init; @@ -177,6 +180,7 @@ typedef struct VhostOps { vhost_get_device_id_op vhost_get_device_id; vhost_force_iommu_op vhost_force_iommu; vhost_set_config_call_op vhost_set_config_call; + vhost_reset_status_op vhost_reset_status; } VhostOps; int vhost_backend_update_device_iotlb(struct vhost_dev *dev, diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index b681841cdc..6d20d70601 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1152,14 +1152,23 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) if (started) { memory_listener_register(&v->listener, &address_space_memory); return vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK); - } else { - vhost_vdpa_reset_device(dev); - vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | - VIRTIO_CONFIG_S_DRIVER); - memory_listener_unregister(&v->listener); + } - return 0; + return 0; +} + +static void vhost_vdpa_reset_status(struct vhost_dev *dev) +{ + struct vhost_vdpa *v = dev->opaque; + + if (dev->vq_index + dev->nvqs != dev->vq_index_end) { + return; } + + vhost_vdpa_reset_device(dev); + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | + VIRTIO_CONFIG_S_DRIVER); + memory_listener_unregister(&v->listener); } static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base, @@ -1346,4 +1355,5 @@ const VhostOps vdpa_ops = { .vhost_vq_get_addr = vhost_vdpa_vq_get_addr, .vhost_force_iommu = vhost_vdpa_force_iommu, .vhost_set_config_call = vhost_vdpa_set_config_call, + .vhost_reset_status = vhost_vdpa_reset_status, }; diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index eb8c4c378c..a266396576 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -2049,6 +2049,9 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings) hdev->vqs + i, hdev->vq_index + i); } + if (hdev->vhost_ops->vhost_reset_status) { + hdev->vhost_ops->vhost_reset_status(hdev); + } if (vhost_dev_has_iommu(hdev)) { if (hdev->vhost_ops->vhost_set_iotlb_callback) { From patchwork Fri Feb 3 19:11:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 468CCC61DA4 for ; Fri, 3 Feb 2023 19:12:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1Tx-0006c4-VV; Fri, 03 Feb 2023 14:12:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1To-0006ZF-HN for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:13 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tn-0007Yv-09 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pbe2RQY0RLyncYTUFWY0cPcVrWsqzfZRkDYAiCISaW0=; b=JmMJad1ALCYHYqavzPElE7BGpx6EnwBqRtZQl4OogoxGLaL9UdVSkCSYsLeg5M2zOWaJN8 Jarz1twHPbucowJo1IAgjneymgp9PEwiNSXFApj2BebcTiyAu2G0+2KzyOxpBog67anHnW zVKPGthy5yoNLaCZL+JpN6/pMW/alN4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-448-CgbVeW8FN6aJR3oAH9U1xQ-1; Fri, 03 Feb 2023 14:12:04 -0500 X-MC-Unique: CgbVeW8FN6aJR3oAH9U1xQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E1CB100F909; Fri, 3 Feb 2023 19:12:04 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3D87492C14; Fri, 3 Feb 2023 19:12:00 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 05/13] vdpa: rewind at get_base, not set_base Date: Fri, 3 Feb 2023 20:11:34 +0100 Message-Id: <20230203191142.362623-6-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org At this moment it is only possible to migrate to a vdpa device running with x-svq=on. As a protective measure, the rewind of the inflight descriptors was done at the destination. That way if the source sent a virtqueue with inuse descriptors they are always discarded. Since this series allows to migrate also to passthrough devices with no SVQ, the right thing to do is to rewind at the source so the base of vrings are correct. Support for inflight descriptors may be added in the future. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 6d20d70601..6385c16a55 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1211,18 +1211,7 @@ static int vhost_vdpa_set_vring_base(struct vhost_dev *dev, struct vhost_vring_state *ring) { struct vhost_vdpa *v = dev->opaque; - VirtQueue *vq = virtio_get_queue(dev->vdev, ring->index); - /* - * vhost-vdpa devices does not support in-flight requests. Set all of them - * as available. - * - * TODO: This is ok for networking, but other kinds of devices might - * have problems with these retransmissions. - */ - while (virtqueue_rewind(vq, 1)) { - continue; - } if (v->shadow_vqs_enabled) { /* * Device vring base was set at device start. SVQ base is handled by @@ -1241,6 +1230,19 @@ static int vhost_vdpa_get_vring_base(struct vhost_dev *dev, int ret; if (v->shadow_vqs_enabled) { + VirtQueue *vq = virtio_get_queue(dev->vdev, ring->index); + + /* + * vhost-vdpa devices does not support in-flight requests. Set all of + * them as available. + * + * TODO: This is ok for networking, but other kinds of devices might + * have problems with these retransmissions. + */ + while (virtqueue_rewind(vq, 1)) { + continue; + } + ring->num = virtio_queue_get_last_avail_idx(dev->vdev, ring->index); return 0; } From patchwork Fri Feb 3 19:11:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30BB7C05027 for ; Fri, 3 Feb 2023 19:12:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1Ty-0006d6-OT; Fri, 03 Feb 2023 14:12:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tr-0006ak-Em for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tn-0007Yz-0x for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KjV3DPcBfm8K8qFiu1yK2BdhCAwgI+FicV5cNL33ulc=; b=AvViE1adjRM6qAISAUdsm8qshNZ8actjUMVGOIMiEOAQ/7jWdF/PpCn9mS4nbGGxutzGsh sEv00nBrBFf1zMT91T7kdRQWeX0iS1Ox45UzygXSaFwrmF/fNcjDa9RA+ICEq1fZi001ap FbkTO2V2G9dP8qa6Ta07zwz339LvsQU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-353-fDdFYvf4N2e9u-KMjUehaA-1; Fri, 03 Feb 2023 14:12:08 -0500 X-MC-Unique: fDdFYvf4N2e9u-KMjUehaA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2F35B293249C; Fri, 3 Feb 2023 19:12:07 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D6F9492C14; Fri, 3 Feb 2023 19:12:04 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 06/13] vdpa net: allow VHOST_F_LOG_ALL Date: Fri, 3 Feb 2023 20:11:35 +0100 Message-Id: <20230203191142.362623-7-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Since some actions move to the start function instead of init, the device features may not be the parent vdpa device's, but the one returned by vhost backend. If transition to SVQ is supported, the vhost backend will return _F_LOG_ALL to signal the device is migratable. Add VHOST_F_LOG_ALL. HW dirty page tracking can be added on top of this change if the device supports it in the future. Signed-off-by: Eugenio Pérez Acked-by: Jason Wang --- net/vhost-vdpa.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index a9e6c8f28e..dd686b4514 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -98,6 +98,8 @@ static const uint64_t vdpa_svq_device_features = BIT_ULL(VIRTIO_NET_F_MQ) | BIT_ULL(VIRTIO_F_ANY_LAYOUT) | BIT_ULL(VIRTIO_NET_F_CTRL_MAC_ADDR) | + /* VHOST_F_LOG_ALL is exposed by SVQ */ + BIT_ULL(VHOST_F_LOG_ALL) | BIT_ULL(VIRTIO_NET_F_RSC_EXT) | BIT_ULL(VIRTIO_NET_F_STANDBY); From patchwork Fri Feb 3 19:11:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FABFC61DA4 for ; Fri, 3 Feb 2023 19:13:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U4-0006ht-Lv; Fri, 03 Feb 2023 14:12:28 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tt-0006c3-C8 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:19 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tr-0007Zg-6Z for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cjF4XgacC8zKnOGK0TvXOE59ICgnYZ9SEDqHvfr3++I=; b=HO2glp4C4O8TxvNVnjOsXsGLlt5U7CKksm7IjXI7bWoaixBSYvssxvAIHHBwHdn2AAAmxI nrta4PDKDC3hJk9N7KxhTDA7KyfdkyO6Lb284wgK2bWsTdPutzJ7noIYNizT5JWkRsSiEq 9LdwDU4gV7Z0MzmkGvE+tThuJw1k72A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-113-yU5WxRCCNNC3oACCS05DqA-1; Fri, 03 Feb 2023 14:12:11 -0500 X-MC-Unique: yU5WxRCCNNC3oACCS05DqA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5498C803D62; Fri, 3 Feb 2023 19:12:10 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6ED73492C14; Fri, 3 Feb 2023 19:12:07 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 07/13] vdpa: add vdpa net migration state notifier Date: Fri, 3 Feb 2023 20:11:36 +0100 Message-Id: <20230203191142.362623-8-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This allows net to restart the device backend to configure SVQ on it. Ideally, these changes should not be net specific. However, the vdpa net backend is the one with enough knowledge to configure everything because of some reasons: * Queues might need to be shadowed or not depending on its kind (control vs data). * Queues need to share the same map translations (iova tree). Because of that it is cleaner to restart the whole net backend and configure again as expected, similar to how vhost-kernel moves between userspace and passthrough. If more kinds of devices need dynamic switching to SVQ we can create a callback struct like VhostOps and move most of the code there. VhostOps cannot be reused since all vdpa backend share them, and to personalize just for networking would be too heavy. Signed-off-by: Eugenio Pérez --- v3: * Add TODO to use the resume operation in the future. * Use migration_in_setup and migration_has_failed instead of a complicated switch case. --- net/vhost-vdpa.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index dd686b4514..bca13f97fd 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -26,12 +26,14 @@ #include #include "standard-headers/linux/virtio_net.h" #include "monitor/monitor.h" +#include "migration/misc.h" #include "hw/virtio/vhost.h" /* Todo:need to add the multiqueue support here */ typedef struct VhostVDPAState { NetClientState nc; struct vhost_vdpa vhost_vdpa; + Notifier migration_state; VHostNetState *vhost_net; /* Control commands shadow buffers */ @@ -241,10 +243,79 @@ static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s) return DO_UPCAST(VhostVDPAState, nc, nc0); } +static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable) +{ + struct vhost_vdpa *v = &s->vhost_vdpa; + VirtIONet *n; + VirtIODevice *vdev; + int data_queue_pairs, cvq, r; + NetClientState *peer; + + /* We are only called on the first data vqs and only if x-svq is not set */ + if (s->vhost_vdpa.shadow_vqs_enabled == enable) { + return; + } + + vdev = v->dev->vdev; + n = VIRTIO_NET(vdev); + if (!n->vhost_started) { + return; + } + + data_queue_pairs = n->multiqueue ? n->max_queue_pairs : 1; + cvq = virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) ? + n->max_ncs - n->max_queue_pairs : 0; + /* + * TODO: vhost_net_stop does suspend, get_base and reset. We can be smarter + * in the future and resume the device if read-only operations between + * suspend and reset goes wrong. + */ + vhost_net_stop(vdev, n->nic->ncs, data_queue_pairs, cvq); + + peer = s->nc.peer; + for (int i = 0; i < data_queue_pairs + cvq; i++) { + VhostVDPAState *vdpa_state; + NetClientState *nc; + + if (i < data_queue_pairs) { + nc = qemu_get_peer(peer, i); + } else { + nc = qemu_get_peer(peer, n->max_queue_pairs); + } + + vdpa_state = DO_UPCAST(VhostVDPAState, nc, nc); + vdpa_state->vhost_vdpa.shadow_data = enable; + + if (i < data_queue_pairs) { + /* Do not override CVQ shadow_vqs_enabled */ + vdpa_state->vhost_vdpa.shadow_vqs_enabled = enable; + } + } + + r = vhost_net_start(vdev, n->nic->ncs, data_queue_pairs, cvq); + if (unlikely(r < 0)) { + error_report("unable to start vhost net: %s(%d)", g_strerror(-r), -r); + } +} + +static void vdpa_net_migration_state_notifier(Notifier *notifier, void *data) +{ + MigrationState *migration = data; + VhostVDPAState *s = container_of(notifier, VhostVDPAState, + migration_state); + + if (migration_in_setup(migration)) { + vhost_vdpa_net_log_global_enable(s, true); + } else if (migration_has_failed(migration)) { + vhost_vdpa_net_log_global_enable(s, false); + } +} + static void vhost_vdpa_net_data_start_first(VhostVDPAState *s) { struct vhost_vdpa *v = &s->vhost_vdpa; + add_migration_state_change_notifier(&s->migration_state); if (v->shadow_vqs_enabled) { v->iova_tree = vhost_iova_tree_new(v->iova_range.first, v->iova_range.last); @@ -278,6 +349,10 @@ static void vhost_vdpa_net_client_stop(NetClientState *nc) assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + if (s->vhost_vdpa.index == 0) { + remove_migration_state_change_notifier(&s->migration_state); + } + dev = s->vhost_vdpa.dev; if (dev->vq_index + dev->nvqs == dev->vq_index_end) { g_clear_pointer(&s->vhost_vdpa.iova_tree, vhost_iova_tree_delete); @@ -741,6 +816,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; + s->migration_state.notify = vdpa_net_migration_state_notifier; s->vhost_vdpa.shadow_vqs_enabled = svq; s->vhost_vdpa.iova_range = iova_range; s->vhost_vdpa.shadow_data = svq; From patchwork Fri Feb 3 19:11:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37BEFC05027 for ; Fri, 3 Feb 2023 19:13:25 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U6-0006lb-C6; Fri, 03 Feb 2023 14:12:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Ty-0006eN-SR for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:23 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Tw-0007aN-LG for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451540; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kxC+Vd6JV8f/ObrRy4WUX4ESGQzNBsPdSPtazpTn/24=; b=gRBzn1aQ7FFPMJw2Lh4oOh1ehrFQD2/jaz3KLL7hPwuMnkK3ZW8kxeQthAWEHSOZEI8T3t mMe1NBj5w5I6azF8qZr2P7NXhY7n7VG74DIIf2yIlFGtzQ8FO7HZfN8jNcb2NGj4nP1xLX zghocIv0yUtOMZrdmON3pz3HPa3IkUM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-249-GVFOv-b5MiWS3yuERPqIsQ-1; Fri, 03 Feb 2023 14:12:14 -0500 X-MC-Unique: GVFOv-b5MiWS3yuERPqIsQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7575F80D0E8; Fri, 3 Feb 2023 19:12:13 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 94A4D492C14; Fri, 3 Feb 2023 19:12:10 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 08/13] vdpa: disable RAM block discard only for the first device Date: Fri, 3 Feb 2023 20:11:37 +0100 Message-Id: <20230203191142.362623-9-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Although it does not make a big difference, its more correct and simplifies the cleanup path in subsequent patches. Move ram_block_discard_disable(false) call to the top of vhost_vdpa_cleanup because: * We cannot use vhost_vdpa_first_dev after dev->opaque = NULL assignment. * Improve the stack order in cleanup: since it is the last action taken in init, it should be the first at cleanup. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 6385c16a55..b0d69df9e6 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -431,16 +431,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) trace_vhost_vdpa_init(dev, opaque); int ret; - /* - * Similar to VFIO, we end up pinning all guest memory and have to - * disable discarding of RAM. - */ - ret = ram_block_discard_disable(true); - if (ret) { - error_report("Cannot set discarding of RAM broken"); - return ret; - } - v = opaque; v->dev = dev; dev->opaque = opaque ; @@ -452,6 +442,16 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) return 0; } + /* + * Similar to VFIO, we end up pinning all guest memory and have to + * disable discarding of RAM. + */ + ret = ram_block_discard_disable(true); + if (ret) { + error_report("Cannot set discarding of RAM broken"); + return ret; + } + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); @@ -577,12 +577,15 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev) assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA); v = dev->opaque; trace_vhost_vdpa_cleanup(dev, v); + if (vhost_vdpa_first_dev(dev)) { + ram_block_discard_disable(false); + } + vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs); memory_listener_unregister(&v->listener); vhost_vdpa_svq_cleanup(dev); dev->opaque = NULL; - ram_block_discard_disable(false); return 0; } From patchwork Fri Feb 3 19:11:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3BCB9C05027 for ; Fri, 3 Feb 2023 19:13:02 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U6-0006lv-Uf; Fri, 03 Feb 2023 14:12:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1U2-0006g4-PX for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1Ty-0007aj-U6 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451542; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cqdlab7l8nHdwiCXObkZmhi8oBObwKS48XyRXQYZ434=; b=H2LIZNv3huBO5aI/GivJzxVH+OKBxBtu+Bls1izuOem4XxtgnXWKeQlyJ8xvCQcWQW+ZNt 53ItuU31IrWSe2+uEe32VR3So1z+8/RB8XDSWTZbCk4YwrGGbrXFxzTqWYE7rsLIf3fCrx Y8b2PzNytNqnlV58TK7u1kGZMu/dfQk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-493-w3QyRM5VO8SeJjCqN7vZLg-1; Fri, 03 Feb 2023 14:12:20 -0500 X-MC-Unique: w3QyRM5VO8SeJjCqN7vZLg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9FC2C803CBB; Fri, 3 Feb 2023 19:12:16 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id B5188492C14; Fri, 3 Feb 2023 19:12:13 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 09/13] vdpa net: block migration if the device has CVQ Date: Fri, 3 Feb 2023 20:11:38 +0100 Message-Id: <20230203191142.362623-10-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Devices with CVQ needs to migrate state beyond vq state. Leaving this to future series. Signed-off-by: Eugenio Pérez --- net/vhost-vdpa.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index bca13f97fd..309861e56c 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -955,11 +955,17 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, } if (has_cvq) { + VhostVDPAState *s; + nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, vdpa_device_fd, i, 1, false, opts->x_svq, iova_range); if (!nc) goto err; + + s = DO_UPCAST(VhostVDPAState, nc, nc); + error_setg(&s->vhost_vdpa.dev->migration_blocker, + "net vdpa cannot migrate with MQ feature"); } return 0; From patchwork Fri Feb 3 19:11:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 955A7C05027 for ; Fri, 3 Feb 2023 19:14:01 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U5-0006k6-Kz; Fri, 03 Feb 2023 14:12:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1U4-0006he-KP for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1U2-0007b1-FL for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451545; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQJuvt6i7H5KtDKMtl36daFtmGk30qd1bEA7/xzw5GU=; b=h9ALJRmAQXPYGsM8Xe/U/qSZLRgpgRJdMXILPGfJeqOCuD8Jp1xKW6EUsm48UFKZKSFswK 9rFRDYeQP5Z8wuZYhgVoGyilW6Oau7hXhjZxqBo3Qo+L7vY7KZKsJDogCV6bfADOzBQRy1 dzVWx10RmqxZ8BxQo3NZVRj6186VVr4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-608-KLvAjDqwPsq_iwpQTDehxQ-1; Fri, 03 Feb 2023 14:12:20 -0500 X-MC-Unique: KLvAjDqwPsq_iwpQTDehxQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F2DDA2932480; Fri, 3 Feb 2023 19:12:19 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id DF988492C14; Fri, 3 Feb 2023 19:12:16 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 10/13] vdpa: block migration if device has unsupported features Date: Fri, 3 Feb 2023 20:11:39 +0100 Message-Id: <20230203191142.362623-11-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A vdpa net device must initialize with SVQ in order to be migratable at this moment, and initialization code verifies some conditions. If the device is not initialized with the x-svq parameter, it will not expose _F_LOG so the vhost subsystem will block VM migration from its initialization. Next patches change this, so we need to verify migration conditions differently. QEMU only supports a subset of net features in SVQ, and it cannot migrate state that cannot track or restore in the destination. Add a migration blocker if the device offer an unsupported feature. Signed-off-by: Eugenio Pérez --- net/vhost-vdpa.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 309861e56c..a0c4d5de2c 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -952,6 +952,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, iova_range); if (!ncs[i]) goto err; + + if (i == 0) { + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, ncs[0]); + + if (!s->vhost_vdpa.dev->migration_blocker) { + vhost_vdpa_net_valid_svq_features(features, + &s->vhost_vdpa.dev->migration_blocker); + } + } } if (has_cvq) { From patchwork Fri Feb 3 19:11:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 27E5CC61DA4 for ; Fri, 3 Feb 2023 19:13:30 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1U8-0006p3-P8; Fri, 03 Feb 2023 14:12:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1U8-0006o4-1s for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1U6-0007ba-K3 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451549; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gOLCdGC0Gh19yhW2UeyJgNxuQQgAazN8wQgGc/RaPyw=; b=HexDYtGzBPiv6JUg8+BwLQJ0G+FnRTFRlFv2Pmvwmpp0yFBHTef+FMPtg4O/AwbfryIVVI eFRZ02ZeFBlHJseMtshfygB1d8QAut76hL2sLBSyIguVYsTitBG/p/6qHwLCQDoyrg7KOh 1gR2UbAahHbj4UD2YHHMS897Lo3Fokg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-94-xPdZNPXONYGaKlojH9kFuQ-1; Fri, 03 Feb 2023 14:12:23 -0500 X-MC-Unique: xPdZNPXONYGaKlojH9kFuQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2312D1871D94; Fri, 3 Feb 2023 19:12:23 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3DD8F492C14; Fri, 3 Feb 2023 19:12:20 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 11/13] vdpa: block migration if dev does not have _F_SUSPEND Date: Fri, 3 Feb 2023 20:11:40 +0100 Message-Id: <20230203191142.362623-12-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Next patches enable devices to be migrated even if vdpa netdev has not been started with x-svq. However, not all devices are migratable, so we need to block migration if we detect that. Block vhost-vdpa device migration if it does not offer _F_SUSPEND and it has not been started with x-svq. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index b0d69df9e6..c01411af30 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -442,6 +442,27 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) return 0; } + /* + * If dev->shadow_vqs_enabled at initialization that means the device has + * been started with x-svq=on, so don't block migration + */ + if (dev->migration_blocker == NULL && !v->shadow_vqs_enabled) { + uint64_t backend_features; + + /* We don't have dev->backend_features yet */ + ret = vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, + &backend_features); + if (unlikely(ret)) { + error_setg_errno(errp, -ret, "Could not get backend features"); + return ret; + } + + if (!(backend_features & BIT_ULL(VHOST_BACKEND_F_SUSPEND))) { + error_setg(&dev->migration_blocker, + "vhost-vdpa backend lacks VHOST_BACKEND_F_SUSPEND feature."); + } + } + /* * Similar to VFIO, we end up pinning all guest memory and have to * disable discarding of RAM. From patchwork Fri Feb 3 19:11:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9AE8C61DA4 for ; Fri, 3 Feb 2023 19:14:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1UH-0006ro-GU; Fri, 03 Feb 2023 14:12:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1UF-0006rF-Vj for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:39 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1UE-0007ds-IB for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:39 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451558; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQg8MSSJa4+ceEvuB/cJI4kdCm8a8MxP7tgVMTC8KSU=; b=eF+BIh1YHixprCCGEm9vVaxOYMYNLNwqn4Qv9ht1rDO2UZv8BkjJpTukcwR9NJIzE81Y+K ICikjPVSDJwX6rOweBjml/J5icITas8LmVDxPI7mYt99xwuLlrvAZIxSY1fLK4JlwnpONU dQccVIOCinc0LS3SIjdvf4n8l52XmRw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-673-64fFUQ53MdS9E_s4OCAknQ-1; Fri, 03 Feb 2023 14:12:27 -0500 X-MC-Unique: 64fFUQ53MdS9E_s4OCAknQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 45824811E9C; Fri, 3 Feb 2023 19:12:26 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 617FF492C14; Fri, 3 Feb 2023 19:12:23 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 12/13] vdpa: block migration if SVQ does not admit a feature Date: Fri, 3 Feb 2023 20:11:41 +0100 Message-Id: <20230203191142.362623-13-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Next patches enable devices to be migrated even if vdpa netdev has not been started with x-svq. However, not all devices are migratable, so we need to block migration if we detect that. Block migration if we detect the device expose a feature SVQ does not know how to work with. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index c01411af30..2de0796e66 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -460,6 +460,15 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) if (!(backend_features & BIT_ULL(VHOST_BACKEND_F_SUSPEND))) { error_setg(&dev->migration_blocker, "vhost-vdpa backend lacks VHOST_BACKEND_F_SUSPEND feature."); + } else { + /* We don't have dev->features yet */ + uint64_t features; + ret = vhost_vdpa_get_dev_features(dev, &features); + if (unlikely(ret)) { + error_setg_errno(errp, -ret, "Could not get device features"); + return ret; + } + vhost_svq_valid_features(features, &dev->migration_blocker); } } From patchwork Fri Feb 3 19:11:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13128292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55DFEC05027 for ; Fri, 3 Feb 2023 19:13:47 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pO1UC-0006ph-6Y; Fri, 03 Feb 2023 14:12:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1UB-0006pZ-Ax for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pO1U9-0007dE-V3 for qemu-devel@nongnu.org; Fri, 03 Feb 2023 14:12:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675451553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=smZg9j4UFFdXL4YAzzEz38YcvUAA6en+5p1KHcahDhI=; b=gjzcZLSLwY10xE+19L8QjiRZ1mr3+kF5elkmsNYOH2jsj12VDajduvT2EhW8VDYyKJqj+8 Q0D03FQyeneAqGbj/8o8WrbpXj3NjB66r1iuqb4v8uZtfehG/VrtAYAKT8/mtsDNxUH2IP 9kEdyst33W+KIqSXs6boYS34oF75ZuA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-640-CBoqzWZbOJWgFvtwlmRUsQ-1; Fri, 03 Feb 2023 14:12:30 -0500 X-MC-Unique: CBoqzWZbOJWgFvtwlmRUsQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 672CB100F911; Fri, 3 Feb 2023 19:12:29 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 84AD9492C14; Fri, 3 Feb 2023 19:12:26 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Cindy Lu , Liuxiangdong , alvaro.karsz@solid-run.com, Zhu Lingshan , Jason Wang , longpeng2@huawei.com, "Michael S. Tsirkin" , Shannon Nelson , Eli Cohen , Stefan Hajnoczi , virtualization@lists.linux-foundation.org, Parav Pandit , si-wei.liu@oracle.com, Laurent Vivier , Gautam Dawar , Stefano Garzarella , Harpreet Singh Anand , "Gonglei (Arei)" Subject: [PATCH 13/13] vdpa: return VHOST_F_LOG_ALL in vhost-vdpa devices Date: Fri, 3 Feb 2023 20:11:42 +0100 Message-Id: <20230203191142.362623-14-eperezma@redhat.com> In-Reply-To: <20230203191142.362623-1-eperezma@redhat.com> References: <20230203191142.362623-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vhost-vdpa devices can return this features now that blockers have been set in case some features are not met. Expose VHOST_F_LOG_ALL only in that case. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 2de0796e66..a94ce254a9 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1319,10 +1319,9 @@ static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, static int vhost_vdpa_get_features(struct vhost_dev *dev, uint64_t *features) { - struct vhost_vdpa *v = dev->opaque; int ret = vhost_vdpa_get_dev_features(dev, features); - if (ret == 0 && v->shadow_vqs_enabled) { + if (ret == 0) { /* Add SVQ logging capabilities */ *features |= BIT_ULL(VHOST_F_LOG_ALL); }