From patchwork Thu Jan 11 19:02:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13517721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 692D5C4707B for ; Thu, 11 Jan 2024 19:04:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rO0K5-0001xg-Os; Thu, 11 Jan 2024 14:02:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K4-0001xY-J7 for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:36 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K2-0002Lp-Dm for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704999753; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qQKcPkY6/W3EsLgkAEJ6imao+rgKnctifkMPjHMrgNY=; b=ZYi4jdQujvpFesC1bqcwJ2lUKrDwrWBhgVour2h8eoVvSNeZJ+1XgUYr84l/PDg6FZANdf 1DwsFyu/W/HiZRfjGNl8GRiKQ/ISTEeoGIV40Cjr9Laa/fmVUdWhQItYnjnhXryfQCliN8 OMeKWUv3QHyFyEQo4zUW4NbUD1M9MsE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-695-X8wHvCKHMbOjfGxl6mSmaQ-1; Thu, 11 Jan 2024 14:02:28 -0500 X-MC-Unique: X8wHvCKHMbOjfGxl6mSmaQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 49EC18314F1; Thu, 11 Jan 2024 19:02:28 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7FBAA492BF0; Thu, 11 Jan 2024 19:02:26 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: mst@redhat.com, qemu-devel@nongnu.org Cc: Peter Xu , Dragos Tatulea , Zhu Lingshan , Jason Wang , Lei Yang , Laurent Vivier , si-wei.liu@oracle.com, Stefano Garzarella , Parav Pandit Subject: [PATCH 1/6] vdpa: check for iova tree initialized at net_client_start Date: Thu, 11 Jan 2024 20:02:17 +0100 Message-Id: <20240111190222.496695-2-eperezma@redhat.com> In-Reply-To: <20240111190222.496695-1-eperezma@redhat.com> References: <20240111190222.496695-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -45 X-Spam_score: -4.6 X-Spam_bar: ---- X-Spam_report: (-4.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To map the guest memory while it is migrating we need to create the iova_tree, as long as the destination uses x-svq=on. Checking to not override it. The function vhost_vdpa_net_client_stop clear it if the device is stopped. If the guest starts the device again, the iova tree is recreated by vhost_vdpa_net_data_start_first or vhost_vdpa_net_cvq_start if needed, so old behavior is kept. Signed-off-by: Eugenio Pérez --- net/vhost-vdpa.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 3726ee5d67..e11b390466 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -341,7 +341,9 @@ static void vhost_vdpa_net_data_start_first(VhostVDPAState *s) migration_add_notifier(&s->migration_state, vdpa_net_migration_state_notifier); - if (v->shadow_vqs_enabled) { + + /* iova_tree may be initialized by vhost_vdpa_net_load_setup */ + if (v->shadow_vqs_enabled && !v->shared->iova_tree) { v->shared->iova_tree = vhost_iova_tree_new(v->shared->iova_range.first, v->shared->iova_range.last); } From patchwork Thu Jan 11 19:02:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13517720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 74E13C47258 for ; Thu, 11 Jan 2024 19:04:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rO0K7-0001yR-PC; Thu, 11 Jan 2024 14:02:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K6-0001xh-1K for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:38 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K4-0002Ml-I3 for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704999755; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7hLfAp79U1FIsM1rjGCyedJBUUbAxAWLgcvsVFhJeTs=; b=Ejj5VXwibYOPozO5J494SntUAroVEZds5VepwVthsTk3cHkr9euHR+E7AXPegH34dVrbbX ZMdhuHgT1DNrH5sxV37EUeNSTX+lJ3Xp/OSO8icvmgJoffusRkuV32NnwMGnkeZRvyi1lQ bHIXyqtiT/Hv3owLP7v4tRd7w6Z68AE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-368-Vv7gGK6dNamWkiOyyqEWBg-1; Thu, 11 Jan 2024 14:02:31 -0500 X-MC-Unique: Vv7gGK6dNamWkiOyyqEWBg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 584FE185A782; Thu, 11 Jan 2024 19:02:30 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8ECE6492BF0; Thu, 11 Jan 2024 19:02:28 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: mst@redhat.com, qemu-devel@nongnu.org Cc: Peter Xu , Dragos Tatulea , Zhu Lingshan , Jason Wang , Lei Yang , Laurent Vivier , si-wei.liu@oracle.com, Stefano Garzarella , Parav Pandit Subject: [PATCH 2/6] vdpa: reorder vhost_vdpa_set_backend_cap Date: Thu, 11 Jan 2024 20:02:18 +0100 Message-Id: <20240111190222.496695-3-eperezma@redhat.com> In-Reply-To: <20240111190222.496695-1-eperezma@redhat.com> References: <20240111190222.496695-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -45 X-Spam_score: -4.6 X-Spam_bar: ---- X-Spam_report: (-4.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org It will be used directly by vhost_vdpa_init. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 60 +++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index ddae494ca8..33ae285f87 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -580,6 +580,36 @@ static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v) v->shadow_vqs = g_steal_pointer(&shadow_vqs); } +static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) +{ + struct vhost_vdpa *v = dev->opaque; + + uint64_t features; + uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 | + 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | + 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | + 0x1ULL << VHOST_BACKEND_F_SUSPEND; + int r; + + if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) { + return -EFAULT; + } + + features &= f; + + if (vhost_vdpa_first_dev(dev)) { + r = vhost_vdpa_call(dev, VHOST_SET_BACKEND_FEATURES, &features); + if (r) { + return -EFAULT; + } + } + + dev->backend_cap = features; + v->shared->backend_cap = features; + + return 0; +} + static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) { struct vhost_vdpa *v = opaque; @@ -827,36 +857,6 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev, return vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK); } -static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) -{ - struct vhost_vdpa *v = dev->opaque; - - uint64_t features; - uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 | - 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | - 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | - 0x1ULL << VHOST_BACKEND_F_SUSPEND; - int r; - - if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) { - return -EFAULT; - } - - features &= f; - - if (vhost_vdpa_first_dev(dev)) { - r = vhost_vdpa_call(dev, VHOST_SET_BACKEND_FEATURES, &features); - if (r) { - return -EFAULT; - } - } - - dev->backend_cap = features; - v->shared->backend_cap = features; - - return 0; -} - static int vhost_vdpa_get_device_id(struct vhost_dev *dev, uint32_t *device_id) { From patchwork Thu Jan 11 19:02:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13517724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2476BC47077 for ; Thu, 11 Jan 2024 19:04:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rO0K8-0001yZ-Ig; Thu, 11 Jan 2024 14:02:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K7-0001y5-1D for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:39 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K5-0002NJ-AL for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704999756; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZNYtP0/fqUQrbz1uancjxE/XlEOpkN/LmQB3y+vqoIQ=; b=B2ja8fD9k4WUgXeAWGLtCskCwCzGdBFhwL0c2w+7rReH8V85WbMeXYZkQk+IppEdKzcCpG rodEs36IOlgBYPk0Bv1t4G//eOKWGJDtDUnSh7P9Ak4X0EUkUQw/4tUXbM8DLxraVLoG5g Mcd8uQcvBWGvpwoA1ZzGH/DB6j7HQNo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-582-AeBOwCZUP_O5KmmZjDJ-mQ-1; Thu, 11 Jan 2024 14:02:32 -0500 X-MC-Unique: AeBOwCZUP_O5KmmZjDJ-mQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 63B8C85A58A; Thu, 11 Jan 2024 19:02:32 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9BE65492C25; Thu, 11 Jan 2024 19:02:30 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: mst@redhat.com, qemu-devel@nongnu.org Cc: Peter Xu , Dragos Tatulea , Zhu Lingshan , Jason Wang , Lei Yang , Laurent Vivier , si-wei.liu@oracle.com, Stefano Garzarella , Parav Pandit Subject: [PATCH 3/6] vdpa: set backend capabilities at vhost_vdpa_init Date: Thu, 11 Jan 2024 20:02:19 +0100 Message-Id: <20240111190222.496695-4-eperezma@redhat.com> In-Reply-To: <20240111190222.496695-1-eperezma@redhat.com> References: <20240111190222.496695-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -45 X-Spam_score: -4.6 X-Spam_bar: ---- X-Spam_report: (-4.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The backend does not reset them until the vdpa file descriptor is closed so there is no harm in doing it only once. This allows the destination of a live migration to premap memory in batches, using VHOST_BACKEND_F_IOTLB_BATCH. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 33ae285f87..bd4db7ba5f 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -620,6 +620,12 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) v->dev = dev; dev->opaque = opaque ; v->shared->listener = vhost_vdpa_memory_listener; + + ret = vhost_vdpa_set_backend_cap(dev); + if (unlikely(ret != 0)) { + return ret; + } + vhost_vdpa_init_svq(dev, v); error_propagate(&dev->migration_blocker, v->migration_blocker); @@ -1506,7 +1512,6 @@ const VhostOps vdpa_ops = { .vhost_set_vring_kick = vhost_vdpa_set_vring_kick, .vhost_set_vring_call = vhost_vdpa_set_vring_call, .vhost_get_features = vhost_vdpa_get_features, - .vhost_set_backend_cap = vhost_vdpa_set_backend_cap, .vhost_set_owner = vhost_vdpa_set_owner, .vhost_set_vring_endian = NULL, .vhost_backend_memslots_limit = vhost_vdpa_memslots_limit, From patchwork Thu Jan 11 19:02:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13517722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 611FFC47DA2 for ; Thu, 11 Jan 2024 19:04:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rO0KD-0001zm-FP; Thu, 11 Jan 2024 14:02:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0KB-0001z2-6s for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:43 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K9-0002PH-HU for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704999760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CGgCal5pEcp+8Ne5+Op8F9zTYYgmybpw7H9WYg4oxlA=; b=E3g9NBYSaYQ7sLlSXoEwL1qkngU4133sn0plYnl2CpoSFA1UFIPmPUfgzuyvmOUAFdTuZT tI6/6QpM4LktWYSDdCeAR697kDfcjW/tlIJtGZNu1S52h2h1Y5A0d6dsoxAnX5fsgDFhDy cRLRPADrJYMXt9CbyPWvA9Ej+JCtdfs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-455-jT0-JY_MMGmOUjtxyTMXSg-1; Thu, 11 Jan 2024 14:02:35 -0500 X-MC-Unique: jT0-JY_MMGmOUjtxyTMXSg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C36D485A58E; Thu, 11 Jan 2024 19:02:34 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id A8707492BF0; Thu, 11 Jan 2024 19:02:32 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: mst@redhat.com, qemu-devel@nongnu.org Cc: Peter Xu , Dragos Tatulea , Zhu Lingshan , Jason Wang , Lei Yang , Laurent Vivier , si-wei.liu@oracle.com, Stefano Garzarella , Parav Pandit Subject: [PATCH 4/6] vdpa: add listener_registered Date: Thu, 11 Jan 2024 20:02:20 +0100 Message-Id: <20240111190222.496695-5-eperezma@redhat.com> In-Reply-To: <20240111190222.496695-1-eperezma@redhat.com> References: <20240111190222.496695-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -45 X-Spam_score: -4.6 X-Spam_bar: ---- X-Spam_report: (-4.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Check if the listener has been registered or not, so it needs to be registered again at start. Signed-off-by: Eugenio Pérez --- include/hw/virtio/vhost-vdpa.h | 6 ++++++ hw/virtio/vhost-vdpa.c | 7 ++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 8f54e5edd4..03ed2f2be3 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -45,6 +45,12 @@ typedef struct vhost_vdpa_shared { bool iotlb_batch_begin_sent; + /* + * The memory listener has been registered, so DMA maps have been sent to + * the device. + */ + bool listener_registered; + /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */ bool shadow_data; } VhostVDPAShared; diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index bd4db7ba5f..b08349d57c 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1331,7 +1331,10 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) "IOMMU and try again"); return -1; } - memory_listener_register(&v->shared->listener, dev->vdev->dma_as); + if (!v->shared->listener_registered) { + memory_listener_register(&v->shared->listener, dev->vdev->dma_as); + v->shared->listener_registered = true; + } return vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK); } @@ -1351,6 +1354,8 @@ static void vhost_vdpa_reset_status(struct vhost_dev *dev) vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); memory_listener_unregister(&v->shared->listener); + v->shared->listener_registered = false; + } static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base, From patchwork Thu Jan 11 19:02:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13517723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2495CC47077 for ; Thu, 11 Jan 2024 19:04:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rO0KC-0001zR-Dd; Thu, 11 Jan 2024 14:02:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0KA-0001yt-JG for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:42 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0K8-0002Od-E3 for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704999759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FDU5bDvjCD7qy3GlwYsAJnVVq36TVzsi9e3jkYL+vdU=; b=B8ROowOaM7bm4kiX58rqEeJz0jVIk/8q7phttZHX4suXzw4Uaztgoa/cUMv3eiSugK1snh U7qopsbFiUfl8bdgIHzxT9IbOpeFwT6w9sORzfxGcWSv9tAj1m+RlM7zPMAQPMRqMhkRsY j/TmqW847tkxfgs1y7apCcECV3e6kq8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-Lo9MZ2fIP_KsZNuy1evwRg-1; Thu, 11 Jan 2024 14:02:37 -0500 X-MC-Unique: Lo9MZ2fIP_KsZNuy1evwRg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CEABE811E9C; Thu, 11 Jan 2024 19:02:36 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 13785492BF0; Thu, 11 Jan 2024 19:02:34 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: mst@redhat.com, qemu-devel@nongnu.org Cc: Peter Xu , Dragos Tatulea , Zhu Lingshan , Jason Wang , Lei Yang , Laurent Vivier , si-wei.liu@oracle.com, Stefano Garzarella , Parav Pandit Subject: [PATCH 5/6] vdpa: reorder listener assignment Date: Thu, 11 Jan 2024 20:02:21 +0100 Message-Id: <20240111190222.496695-6-eperezma@redhat.com> In-Reply-To: <20240111190222.496695-1-eperezma@redhat.com> References: <20240111190222.496695-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -45 X-Spam_score: -4.6 X-Spam_bar: ---- X-Spam_report: (-4.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Since commit f6fe3e333f ("vdpa: move memory listener to vhost_vdpa_shared") this piece of code repeatedly assign shared->listener members. This was not a problem as it was not used until device start. However next patches move the listener registration to this vhost_vdpa_init function. When the listener is registered it is added to an embedded linked list, so setting its members again will cause memory corruption to the linked list node. Do the right thing and only set it in the first vdpa device. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index b08349d57c..521a889104 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -619,7 +619,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) v->dev = dev; dev->opaque = opaque ; - v->shared->listener = vhost_vdpa_memory_listener; ret = vhost_vdpa_set_backend_cap(dev); if (unlikely(ret != 0)) { @@ -661,6 +660,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); + v->shared->listener = vhost_vdpa_memory_listener; return 0; } From patchwork Thu Jan 11 19:02:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13517719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40E5CC47077 for ; Thu, 11 Jan 2024 19:03:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rO0KG-0001zx-6a; Thu, 11 Jan 2024 14:02:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0KF-0001zo-2c for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:47 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rO0KD-0002RC-EN for qemu-devel@nongnu.org; Thu, 11 Jan 2024 14:02:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704999764; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xfHj4a+1kkW3IuoRJ+D3rPLEa+55ok5buRdNGYoGEhQ=; b=cFs5RcuGJsCXX/briUVUbYdraCV1hz6xBV5fmk/P43OxGeaC4v5OCodxCoVDXY3qt0DXjL X2iNn2ckNaeJu5uN+LTa4u8HQMngAoaqmJccKaRTMzBciWw6J5fcDXM+xYsCeP4Lq+6SS3 nwdZiGHon4Yh56auIvYyEi6y0PngCr8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-623-RWZ7LSwLOdeF1csfGnAqog-1; Thu, 11 Jan 2024 14:02:39 -0500 X-MC-Unique: RWZ7LSwLOdeF1csfGnAqog-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DCD422999B28; Thu, 11 Jan 2024 19:02:38 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 20559492BF0; Thu, 11 Jan 2024 19:02:36 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: mst@redhat.com, qemu-devel@nongnu.org Cc: Peter Xu , Dragos Tatulea , Zhu Lingshan , Jason Wang , Lei Yang , Laurent Vivier , si-wei.liu@oracle.com, Stefano Garzarella , Parav Pandit Subject: [PATCH 6/6] vdpa: move memory listener register to vhost_vdpa_init Date: Thu, 11 Jan 2024 20:02:22 +0100 Message-Id: <20240111190222.496695-7-eperezma@redhat.com> In-Reply-To: <20240111190222.496695-1-eperezma@redhat.com> References: <20240111190222.496695-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -45 X-Spam_score: -4.6 X-Spam_bar: ---- X-Spam_report: (-4.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Current memory operations like pinning may take a lot of time at the destination. Currently they are done after the source of the migration is stopped, and before the workload is resumed at the destination. This is a period where neigher traffic can flow, nor the VM workload can continue (downtime). We can do better as we know the memory layout of the guest RAM at the destination from the moment that all devices are initializaed. So moving that operation allows QEMU to communicate the kernel the maps while the workload is still running in the source, so Linux can start mapping them. As a small drawback, there is a time in the initialization where QEMU cannot respond to QMP etc. By some testing, this time is about 0.2seconds. This may be further reduced (or increased) depending on the vdpa driver and the platform hardware, and it is dominated by the cost of memory pinning. This matches the time that we move out of the called downtime window. The downtime is measured as checking the trace timestamp from the moment the source suspend the device to the moment the destination starts the eight and last virtqueue pair. For a 39G guest, it goes from ~2.2526 secs to 2.0949. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 521a889104..eae8b790dd 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -660,7 +660,13 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); + /* + * Being optimistic and listening address space memory. If the device + * uses vIOMMU, it is changed at vhost_vdpa_dev_start. + */ v->shared->listener = vhost_vdpa_memory_listener; + memory_listener_register(&v->shared->listener, &address_space_memory); + v->shared->listener_registered = true; return 0; } @@ -1331,6 +1337,11 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) "IOMMU and try again"); return -1; } + if (v->shared->listener_registered && + dev->vdev->dma_as != v->shared->listener.address_space) { + memory_listener_unregister(&v->shared->listener); + v->shared->listener_registered = false; + } if (!v->shared->listener_registered) { memory_listener_register(&v->shared->listener, dev->vdev->dma_as); v->shared->listener_registered = true;