From patchwork Thu Jul 6 19:12:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13304083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30B26EB64DC for ; Thu, 6 Jul 2023 19:14:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHUPD-00063s-QZ; Thu, 06 Jul 2023 15:12:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPC-000636-49 for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPA-0003fM-KB for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688670759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n4LtMziDSwqhV9XrrxLcKL5CUlV/p+z5tjPX/N35b1U=; b=V9kWFDabbSps8YTU3R+9yHT4LepUj/X+3XTLfRnbyE+UhAu3tBkiqOkIQc1lMfRvjj5aCO R7zcyDmo+shUmPUmJub5HlAuJfR/KU+DLb3/nMl47z0ninBtPql7YHeF4V/aGDjzhwbsQp m964WKjBuagzA+1JJq26LULGPByBVrQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-ufwXmBxZPhK9UqhmEOBj8w-1; Thu, 06 Jul 2023 15:12:34 -0400 X-MC-Unique: ufwXmBxZPhK9UqhmEOBj8w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BEEA5101A528; Thu, 6 Jul 2023 19:12:33 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id A65A64087C6A; Thu, 6 Jul 2023 19:12:31 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Laurent Vivier , "Michael S. Tsirkin" , Cindy Lu , si-wei.liu@oracle.com, Stefano Garzarella , Shannon Nelson , Gautam Dawar , Jason Wang , Harpreet Singh Anand , Parav Pandit , Dragos Tatulea , Zhu Lingshan , Lei Yang Subject: [RFC PATCH 1/6] vdpa: export vhost_vdpa_set_vring_ready Date: Thu, 6 Jul 2023 21:12:22 +0200 Message-Id: <20230706191227.835526-2-eperezma@redhat.com> In-Reply-To: <20230706191227.835526-1-eperezma@redhat.com> References: <20230706191227.835526-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The vhost-vdpa net backend needs to enable vrings in a different order than default, so export it. No functional change intended except for tracing, that now includes the (virtio) index being enabled and the return value of the ioctl. Still ignoring return value of this function if called from vhost_vdpa_dev_start, as reorganize calling code around it is out of the scope of this series. Signed-off-by: Eugenio Pérez --- include/hw/virtio/vhost-vdpa.h | 1 + hw/virtio/vhost-vdpa.c | 28 ++++++++++++++++++++-------- hw/virtio/trace-events | 2 +- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index e64bfc7f98..5407d54fd7 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -57,6 +57,7 @@ typedef struct vhost_vdpa { } VhostVDPA; int vhost_vdpa_get_iova_range(int fd, struct vhost_vdpa_iova_range *iova_range); +int vhost_vdpa_set_vring_ready(struct vhost_vdpa *v, unsigned idx); int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova, hwaddr size, void *vaddr, bool readonly); diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 3c575a9a6e..5978d970ee 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -528,6 +528,19 @@ int vhost_vdpa_get_iova_range(int fd, struct vhost_vdpa_iova_range *iova_range) return ret < 0 ? -errno : 0; } +int vhost_vdpa_set_vring_ready(struct vhost_vdpa *v, unsigned idx) +{ + struct vhost_dev *dev = v->dev; + struct vhost_vring_state state = { + .index = idx, + .num = 1, + }; + int r = vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state); + + trace_vhost_vdpa_set_vring_ready(dev, idx, r); + return r; +} + /* * The use of this function is for requests that only need to be * applied once. Typically such request occurs at the beginning @@ -872,16 +885,15 @@ static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx) return idx; } -static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev) +static int vhost_vdpa_set_vrings_ready(struct vhost_dev *dev) { + struct vhost_vdpa *v = dev->opaque; int i; - trace_vhost_vdpa_set_vring_ready(dev); + + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA); + for (i = 0; i < dev->nvqs; ++i) { - struct vhost_vring_state state = { - .index = dev->vq_index + i, - .num = 1, - }; - vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state); + vhost_vdpa_set_vring_ready(v, dev->vq_index + i); } return 0; } @@ -1294,7 +1306,7 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) if (unlikely(!ok)) { return -1; } - vhost_vdpa_set_vring_ready(dev); + vhost_vdpa_set_vrings_ready(dev); } else { vhost_vdpa_suspend(dev); vhost_vdpa_svqs_stop(dev); diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 8f8d05cf9b..4f6a6ba428 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -46,7 +46,7 @@ vhost_vdpa_set_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRI vhost_vdpa_get_device_id(void *dev, uint32_t device_id) "dev: %p device_id %"PRIu32 vhost_vdpa_reset_device(void *dev, uint8_t status) "dev: %p status: 0x%"PRIx8 vhost_vdpa_get_vq_index(void *dev, int idx, int vq_idx) "dev: %p idx: %d vq idx: %d" -vhost_vdpa_set_vring_ready(void *dev) "dev: %p" +vhost_vdpa_set_vring_ready(void *dev, unsigned i, int r) "dev: %p, idx: %u, r: %d" vhost_vdpa_dump_config(void *dev, const char *line) "dev: %p %s" vhost_vdpa_set_config(void *dev, uint32_t offset, uint32_t size, uint32_t flags) "dev: %p offset: %"PRIu32" size: %"PRIu32" flags: 0x%"PRIx32 vhost_vdpa_get_config(void *dev, void *config, uint32_t config_len) "dev: %p config: %p config_len: %"PRIu32 From patchwork Thu Jul 6 19:12:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13304080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C4E0EB64DC for ; Thu, 6 Jul 2023 19:13:56 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHUPF-00064O-Hm; Thu, 06 Jul 2023 15:12:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPE-00063v-1W for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPC-0003gV-It for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688670762; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s1mv7uzcPXcJ81XaXK92RJXYFUYLlxlqghXDFZnxUMI=; b=CETF/eUkNzo8ynhZQKezU1wszBF79bA1I9m7HonNDD9U4PvDXCP+rayR6oN0uOvdfjG1q/ pEZTq/sdSsX6LHSsCIWpRgzcTtUJ5xZl2kMHYX1BnTfHmiCyhYSf/7djNPhBSJHOAh7zgF yOuiCd2oqG/mWd8eduWdeQ69SBLYKXI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-256-zMBrBhsRPeiKunbh1W-Wlg-1; Thu, 06 Jul 2023 15:12:36 -0400 X-MC-Unique: zMBrBhsRPeiKunbh1W-Wlg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1F45881DB6C; Thu, 6 Jul 2023 19:12:36 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 075CC4087C6A; Thu, 6 Jul 2023 19:12:33 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Laurent Vivier , "Michael S. Tsirkin" , Cindy Lu , si-wei.liu@oracle.com, Stefano Garzarella , Shannon Nelson , Gautam Dawar , Jason Wang , Harpreet Singh Anand , Parav Pandit , Dragos Tatulea , Zhu Lingshan , Lei Yang Subject: [RFC PATCH 2/6] vdpa: add should_enable op Date: Thu, 6 Jul 2023 21:12:23 +0200 Message-Id: <20230706191227.835526-3-eperezma@redhat.com> In-Reply-To: <20230706191227.835526-1-eperezma@redhat.com> References: <20230706191227.835526-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To restore the device at the destination of a live migration we send the commands through control virtqueue. For a device to read CVQ it must have received the DRIVER_OK status bit. However this opens a window where the device could start receiving packets in rx queue 0 before it receives the RSS configuration. To avoid that, we will not send vring_enable until all configuration is used by the device. As a first step, enable a new vitio ops per vhost_vdpa device to know if we should enable a virtqueue or not. This srtuct can be reused in the future to add more actions to vhost_vdpa that depend on the virtIO kind of device. Signed-off-by: Eugenio Pérez --- include/hw/virtio/vhost-vdpa.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 5407d54fd7..3d330d439a 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -30,6 +30,13 @@ typedef struct VhostVDPAHostNotifier { void *addr; } VhostVDPAHostNotifier; +struct vhost_vdpa; +typedef bool (*vhost_vdpa_virtio_should_enable_op)(const struct vhost_vdpa *v); + +typedef struct VhostVDPAVirtIOOps { + vhost_vdpa_virtio_should_enable_op should_enable; +} VhostVDPAVirtIOOps; + typedef struct vhost_vdpa { int device_fd; int index; @@ -48,6 +55,7 @@ typedef struct vhost_vdpa { VhostIOVATree *iova_tree; GPtrArray *shadow_vqs; const VhostShadowVirtqueueOps *shadow_vq_ops; + const VhostVDPAVirtIOOps *virtio_ops; void *shadow_vq_ops_opaque; struct vhost_dev *dev; Error *migration_blocker; From patchwork Thu Jul 6 19:12:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13304077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3CB97EB64D9 for ; Thu, 6 Jul 2023 19:13:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHUPD-00063t-U4; Thu, 06 Jul 2023 15:12:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPC-00063T-Jj for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPB-0003fh-9j for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688670760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E+dfnhoTgwmKeq2jhencyAMRE+H1c2Baij8rVaINU/c=; b=UGrxvvy3XpxxPrNVBcaRaFvQIong2LOVoDIL3smBjSjhUt/QLqxxZhOB02zHglTnlDeiRY ZE796FdBou8MqGZsPTVcawNvxberOzxaNFU6zIWyhvZhQoSvgmoVNi6FZ9LiOonu2h8L6i LxN5N6i9bW8M6akU9MqTpMkHG9lm1xk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-80-6cIh9tNtMAiEjqCfKIiP-A-1; Thu, 06 Jul 2023 15:12:39 -0400 X-MC-Unique: 6cIh9tNtMAiEjqCfKIiP-A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7D2B63803914; Thu, 6 Jul 2023 19:12:38 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5BBFB4087C6A; Thu, 6 Jul 2023 19:12:36 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Laurent Vivier , "Michael S. Tsirkin" , Cindy Lu , si-wei.liu@oracle.com, Stefano Garzarella , Shannon Nelson , Gautam Dawar , Jason Wang , Harpreet Singh Anand , Parav Pandit , Dragos Tatulea , Zhu Lingshan , Lei Yang Subject: [RFC PATCH 3/6] vdpa: use virtio_ops->should_enable at vhost_vdpa_set_vrings_ready Date: Thu, 6 Jul 2023 21:12:24 +0200 Message-Id: <20230706191227.835526-4-eperezma@redhat.com> In-Reply-To: <20230706191227.835526-1-eperezma@redhat.com> References: <20230706191227.835526-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This allow to skip some rings that qemu does not want to enable. Signed-off-by: Eugenio Pérez --- hw/virtio/vhost-vdpa.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 5978d970ee..f51f5d9969 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -892,6 +892,11 @@ static int vhost_vdpa_set_vrings_ready(struct vhost_dev *dev) assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA); + if (v->virtio_ops && v->virtio_ops->should_enable + && !(v->virtio_ops->should_enable(v))) { + return 0; + } + for (i = 0; i < dev->nvqs; ++i) { vhost_vdpa_set_vring_ready(v, dev->vq_index + i); } From patchwork Thu Jul 6 19:12:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13304079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D11F6EB64DD for ; Thu, 6 Jul 2023 19:13:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHUPL-00065J-D1; Thu, 06 Jul 2023 15:12:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPI-000654-Qf for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:48 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPH-0003i4-EL for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688670766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vyUVMV+DHTYwE8hKMtLmbmVFq90NewPlv7cTYYGrs7A=; b=i81UqxbatRo4pXZPHhoz4id6sQt3efr0S13YpoO9VLiMUuRRsFmhpk8L2koNwCsI/fgl2l TjzJMN1qfhN+s8JoZPR3U7szWOa7XYra/ObIBO//ZdZYiPttppgHR72Pqe5pZmQEm3yBXL 51P83h91b5Dg0TEfchhibX9ERYb1Hoc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-59-_mK3B_cjOb-gsGItYi-U-A-1; Thu, 06 Jul 2023 15:12:41 -0400 X-MC-Unique: _mK3B_cjOb-gsGItYi-U-A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DCAD51C17100; Thu, 6 Jul 2023 19:12:40 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id BC5D24087C6A; Thu, 6 Jul 2023 19:12:38 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Laurent Vivier , "Michael S. Tsirkin" , Cindy Lu , si-wei.liu@oracle.com, Stefano Garzarella , Shannon Nelson , Gautam Dawar , Jason Wang , Harpreet Singh Anand , Parav Pandit , Dragos Tatulea , Zhu Lingshan , Lei Yang Subject: [RFC PATCH 4/6] vdpa: add stub vhost_vdpa_should_enable Date: Thu, 6 Jul 2023 21:12:25 +0200 Message-Id: <20230706191227.835526-5-eperezma@redhat.com> In-Reply-To: <20230706191227.835526-1-eperezma@redhat.com> References: <20230706191227.835526-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To restore the device at the destination of a live migration we send the commands through control virtqueue. For a device to read CVQ it must have received the DRIVER_OK status bit. However this opens a window where the device could start receiving packets in rx queue 0 before it receives the RSS configuration. To avoid that, we will not send vring_enable until all configuration is used by the device. Add vhost-vdpa net vhost_vdpa_should_enable. Do not change the behavior in this commit, only introduce the op. Signed-off-by: Eugenio Pérez --- net/vhost-vdpa.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index e19ab063fa..536bab8613 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -826,6 +826,15 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +static bool vhost_vdpa_should_enable(const struct vhost_vdpa *v) +{ + return true; +} + +static const VhostVDPAVirtIOOps vhost_vdpa_virtio_net_ops = { + .should_enable = vhost_vdpa_should_enable, +}; + /** * Probe if CVQ is isolated * @@ -949,6 +958,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.shadow_vqs_enabled = svq; s->vhost_vdpa.iova_range = iova_range; s->vhost_vdpa.shadow_data = svq; + s->vhost_vdpa.virtio_ops = &vhost_vdpa_virtio_net_ops; if (queue_pair_index == 0) { vhost_vdpa_net_valid_svq_features(features, &s->vhost_vdpa.migration_blocker); From patchwork Thu Jul 6 19:12:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13304081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3AC3FEB64D9 for ; Thu, 6 Jul 2023 19:14:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHUPN-00066B-Ak; Thu, 06 Jul 2023 15:12:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPL-00065N-GN for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:51 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPJ-0003it-Uf for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688670769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WZ0T9Bj7ivsWGJG0O+gF5mf9U2T53/pSkGsxT8Ys2kQ=; b=TggXe6CH34RnY5r0I2YYqSWprQIS0XT8fb8BlLPRnebO4aJeLFX3UhYiOw+Ph+ZZkwgL7H OPdtaWTRLZKl2/xSxWXydnFuM/+29AoMS8iSP+QUjlr8xXp9dHlUIzdH845Y2kqF3yau+w 7+aqdmODwG8PMNfH5mB0XHJsr7TRPzg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-250-9lz0WDnDNWq3RbPewj_Wkw-1; Thu, 06 Jul 2023 15:12:43 -0400 X-MC-Unique: 9lz0WDnDNWq3RbPewj_Wkw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 43FF7104458E; Thu, 6 Jul 2023 19:12:43 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2789D4087C6A; Thu, 6 Jul 2023 19:12:41 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Laurent Vivier , "Michael S. Tsirkin" , Cindy Lu , si-wei.liu@oracle.com, Stefano Garzarella , Shannon Nelson , Gautam Dawar , Jason Wang , Harpreet Singh Anand , Parav Pandit , Dragos Tatulea , Zhu Lingshan , Lei Yang Subject: [RFC PATCH 5/6] vdpa: delay enable of data vqs Date: Thu, 6 Jul 2023 21:12:26 +0200 Message-Id: <20230706191227.835526-6-eperezma@redhat.com> In-Reply-To: <20230706191227.835526-1-eperezma@redhat.com> References: <20230706191227.835526-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To restore the device at the destination of a live migration we send the commands through control virtqueue. For a device to read CVQ it must have received the DRIVER_OK status bit. However this opens a window where the device could start receiving packets in rx queue 0 before it receives the RSS configuration. To avoid that, we do not send vring_enable until all configuration is used by the device. Signed-off-by: Eugenio Pérez --- net/vhost-vdpa.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 536bab8613..d5970e9f06 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -739,6 +739,13 @@ static int vhost_vdpa_net_load(NetClientState *nc) return r; } + for (int i = 0; i < v->dev->vq_index; ++i) { + r = vhost_vdpa_set_vring_ready(v, i); + if (unlikely(r)) { + return r; + } + } + return 0; } @@ -826,9 +833,25 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +/** + * Check if a vhost_vdpa device should enable before DRIVER_OK + * + * CVQ must always start first if we want to restore the state safely. Do not + * start data vqs if the device has CVQ. + */ static bool vhost_vdpa_should_enable(const struct vhost_vdpa *v) { - return true; + struct vhost_dev *dev = v->dev; + + if (!dev->vq_index_end % 2) { + /* vDPA device does not have CVQ */ + return true; + } + + /* + * We're evaluating CVQ, enable to send control cmds to load device state. + */ + return dev->vq_index + 1 == dev->vq_index_end; } static const VhostVDPAVirtIOOps vhost_vdpa_virtio_net_ops = { From patchwork Thu Jul 6 19:12:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 13304078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ACD77EB64D9 for ; Thu, 6 Jul 2023 19:13:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHUPO-00066F-8L; Thu, 06 Jul 2023 15:12:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPM-00065o-94 for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHUPK-0003ix-0J for qemu-devel@nongnu.org; Thu, 06 Jul 2023 15:12:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688670769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=27cB9bWF+h14/jQtV/P+lGLeqj4W8ligToLP5nYNQo4=; b=Qhfd8UXqoINHxsxz/FmzYSwo9Kykxa+u6L7t8ggkBmfEVT9KLjHU7Ygu3ErefICZzKZI/q bmLYUGxNAeQdcKU5OpDaOdjN/6JKjuzEo+lnnm8Z/f8q/CpSZuF8Nx8MGeD7dScoxU/5Eu 899BgGIZEZhxI/2VVi5d/DZHN+wtO8M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-131-m_Le2UpBPK6aZeaHcXSR1g-1; Thu, 06 Jul 2023 15:12:46 -0400 X-MC-Unique: m_Le2UpBPK6aZeaHcXSR1g-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A09E981D9EC; Thu, 6 Jul 2023 19:12:45 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.194.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 823B44087C6A; Thu, 6 Jul 2023 19:12:43 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: qemu-devel@nongnu.org Cc: Laurent Vivier , "Michael S. Tsirkin" , Cindy Lu , si-wei.liu@oracle.com, Stefano Garzarella , Shannon Nelson , Gautam Dawar , Jason Wang , Harpreet Singh Anand , Parav Pandit , Dragos Tatulea , Zhu Lingshan , Lei Yang Subject: [RFC PATCH 6/6] vdpa: remove net cvq migration blocker Date: Thu, 6 Jul 2023 21:12:27 +0200 Message-Id: <20230706191227.835526-7-eperezma@redhat.com> In-Reply-To: <20230706191227.835526-1-eperezma@redhat.com> References: <20230706191227.835526-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.129.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Now that we have add migration blockers if the device does not support all the needed features, remove the general blocker applied to all net devices with CVQ. Signed-off-by: Eugenio Pérez --- net/vhost-vdpa.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index d5970e9f06..5432b50498 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -996,18 +996,6 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.shadow_vq_ops = &vhost_vdpa_net_svq_ops; s->vhost_vdpa.shadow_vq_ops_opaque = s; s->cvq_isolated = cvq_isolated; - - /* - * TODO: We cannot migrate devices with CVQ and no x-svq enabled as - * there is no way to set the device state (MAC, MQ, etc) before - * starting the datapath. - * - * Migration blocker ownership now belongs to s->vhost_vdpa. - */ - if (!svq) { - error_setg(&s->vhost_vdpa.migration_blocker, - "net vdpa cannot migrate with CVQ feature"); - } } ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa, queue_pair_index, nvqs); if (ret) {