From patchwork Fri Jun 3 10:09:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 12868920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A948C43334 for ; Fri, 3 Jun 2022 10:10:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243426AbiFCKKG (ORCPT ); Fri, 3 Jun 2022 06:10:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243415AbiFCKKE (ORCPT ); Fri, 3 Jun 2022 06:10:04 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id D3F5F3B02E for ; Fri, 3 Jun 2022 03:10:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654251003; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0BBrHrIvz5xwm3AvObGALT/JpyyQvxptTTPXovns/7Y=; b=f1Qwek/SA+Ci7v/l+qT0nfxnDZmdCPK6t2BpodYkPGwjp7/7s2dSLt8Uukwsq7/vWUgojJ 4a7hJFAlvh/HENyQjA2CA8wfPl2ZWfpSxHRD8VYxBaBtAZ9rajkzPhPMWiAebs/zPCqmex 6jRSZ6EVR7y63hEORzWlgi689UtyQYU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-641-lmu9olJ3Pg-wG03v9Vfw6g-1; Fri, 03 Jun 2022 06:09:59 -0400 X-MC-Unique: lmu9olJ3Pg-wG03v9Vfw6g-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7F8361C06EDD; Fri, 3 Jun 2022 10:09:58 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.40.192.190]) by smtp.corp.redhat.com (Postfix) with ESMTP id 964FE492C3B; Fri, 3 Jun 2022 10:09:53 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: kvm@vger.kernel.org, Jason Wang , "Michael S. Tsirkin" , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, virtualization@lists.linux-foundation.org Cc: Christophe JAILLET , Longpeng , Stefano Garzarella , dinang@xilinx.com, Piotr.Uminski@intel.com, martinpo@xilinx.com, tanuj.kamde@amd.com, Parav Pandit , Zhang Min , habetsm.xilinx@gmail.com, Zhu Lingshan , lulu@redhat.com, hanand@xilinx.com, martinh@xilinx.com, Si-Wei Liu , gautam.dawar@amd.com, Xie Yongji , ecree.xilinx@gmail.com, pabloc@xilinx.com, lvivier@redhat.com, Eli Cohen , Wu Zongyong , Dan Carpenter Subject: [PATCH v5 1/4] vdpa: Add suspend operation Date: Fri, 3 Jun 2022 12:09:41 +0200 Message-Id: <20220603100944.871727-2-eperezma@redhat.com> In-Reply-To: <20220603100944.871727-1-eperezma@redhat.com> References: <20220603100944.871727-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This operation is optional: It it's not implemented, backend feature bit will not be exposed. Signed-off-by: Eugenio Pérez --- include/linux/vdpa.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index 15af802d41c4c..8f4559795bf9f 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -215,6 +215,10 @@ struct vdpa_map_file { * @reset: Reset device * @vdev: vdpa device * Returns integer: success (0) or error (< 0) + * @suspend: Suspend or resume the device (optional) + * @vdev: vdpa device + * @suspend: suspend (true), resume (false) + * Returns integer: success (0) or error (< 0) * @get_config_size: Get the size of the configuration space includes * fields that are conditional on feature bits. * @vdev: vdpa device @@ -316,6 +320,7 @@ struct vdpa_config_ops { u8 (*get_status)(struct vdpa_device *vdev); void (*set_status)(struct vdpa_device *vdev, u8 status); int (*reset)(struct vdpa_device *vdev); + int (*suspend)(struct vdpa_device *vdev, bool suspend); size_t (*get_config_size)(struct vdpa_device *vdev); void (*get_config)(struct vdpa_device *vdev, unsigned int offset, void *buf, unsigned int len); From patchwork Fri Jun 3 10:09:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 12868922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6ED5DC433EF for ; Fri, 3 Jun 2022 10:10:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243399AbiFCKKU (ORCPT ); Fri, 3 Jun 2022 06:10:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243448AbiFCKKN (ORCPT ); Fri, 3 Jun 2022 06:10:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B99D93B03B for ; Fri, 3 Jun 2022 03:10:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654251010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dgH0bATG0Fq2ckdWH4abkwGoSauQEOe0ygUhjAt/u2U=; b=NVeYhfJJHqXusNh0XoAoeWMPS4/iNXte1a8s8S3TVRlx0hgme1P4fBrinBWYrF6Ag4IsU8 gGzMnvut4yHXPCCVwGZrcpa7P/ZdG/UFkLKpo9zRw2bno1YD4/HvCBoZAvq+mtUvHEt4IP LmeVk3jYJk471W1O8fnKF4YIFfWFatM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-v6ozWTpGMLOl0YnaXEqvSw-1; Fri, 03 Jun 2022 06:10:05 -0400 X-MC-Unique: v6ozWTpGMLOl0YnaXEqvSw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DF5523C0F74E; Fri, 3 Jun 2022 10:10:03 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.40.192.190]) by smtp.corp.redhat.com (Postfix) with ESMTP id CC61C492C3B; Fri, 3 Jun 2022 10:09:58 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: kvm@vger.kernel.org, Jason Wang , "Michael S. Tsirkin" , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, virtualization@lists.linux-foundation.org Cc: Christophe JAILLET , Longpeng , Stefano Garzarella , dinang@xilinx.com, Piotr.Uminski@intel.com, martinpo@xilinx.com, tanuj.kamde@amd.com, Parav Pandit , Zhang Min , habetsm.xilinx@gmail.com, Zhu Lingshan , lulu@redhat.com, hanand@xilinx.com, martinh@xilinx.com, Si-Wei Liu , gautam.dawar@amd.com, Xie Yongji , ecree.xilinx@gmail.com, pabloc@xilinx.com, lvivier@redhat.com, Eli Cohen , Wu Zongyong , Dan Carpenter Subject: [PATCH v5 2/4] vhost-vdpa: introduce SUSPEND backend feature bit Date: Fri, 3 Jun 2022 12:09:42 +0200 Message-Id: <20220603100944.871727-3-eperezma@redhat.com> In-Reply-To: <20220603100944.871727-1-eperezma@redhat.com> References: <20220603100944.871727-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Userland knows if it can suspend the device or not by checking this feature bit. It's only offered if the vdpa driver backend implements the suspend() operation callback, and to offer it or userland to ack it if the backend does not offer that callback is an error. Signed-off-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 16 +++++++++++++++- include/uapi/linux/vhost_types.h | 2 ++ 2 files changed, 17 insertions(+), 1 deletion(-) -- 2.31.1 diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 935a1d0ddb97e..f4b492526c6f8 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -347,6 +347,14 @@ static long vhost_vdpa_set_config(struct vhost_vdpa *v, return 0; } +static bool vhost_vdpa_can_suspend(const struct vhost_vdpa *v) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + + return ops->suspend; +} + static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep) { struct vdpa_device *vdpa = v->vdpa; @@ -575,7 +583,11 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, if (cmd == VHOST_SET_BACKEND_FEATURES) { if (copy_from_user(&features, featurep, sizeof(features))) return -EFAULT; - if (features & ~VHOST_VDPA_BACKEND_FEATURES) + if (features & ~(VHOST_VDPA_BACKEND_FEATURES | + BIT_ULL(VHOST_BACKEND_F_SUSPEND))) + return -EOPNOTSUPP; + if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) && + !vhost_vdpa_can_suspend(v)) return -EOPNOTSUPP; vhost_set_backend_features(&v->vdev, features); return 0; @@ -626,6 +638,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, break; case VHOST_GET_BACKEND_FEATURES: features = VHOST_VDPA_BACKEND_FEATURES; + if (vhost_vdpa_can_suspend(v)) + features |= BIT_ULL(VHOST_BACKEND_F_SUSPEND); if (copy_to_user(featurep, &features, sizeof(features))) r = -EFAULT; break; diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h index 634cee485abbc..1bdd6e363f4c9 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -161,5 +161,7 @@ struct vhost_vdpa_iova_range { * message */ #define VHOST_BACKEND_F_IOTLB_ASID 0x3 +/* Device can be suspended */ +#define VHOST_BACKEND_F_SUSPEND 0x4 #endif From patchwork Fri Jun 3 10:09:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 12868924 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74F45C433EF for ; Fri, 3 Jun 2022 10:10:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243506AbiFCKKv (ORCPT ); Fri, 3 Jun 2022 06:10:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243487AbiFCKKi (ORCPT ); Fri, 3 Jun 2022 06:10:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7753B3B3FF for ; Fri, 3 Jun 2022 03:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654251025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VJfV/hotJTL7ZePv0RbAazmYZHrUQpKxi4Jb173Q20w=; b=Owgufx5oHAQ/94ZPzQ3GtGh6WRkfhSgqE6A54U/HjhLhBkX7/4QFtfjf7xduIxd33sRLIR WXdqXH95X0XE2IVbpYahZSypcViks9jcqVEszAjgad8DWPiGIFGCAvyiqM+g4jJ0IZ3RkK TFXQiIyu7BzUZBkZVVJxvTsYGiOTyJw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-619-y528ceswNBifW8e9WJwpEQ-1; Fri, 03 Jun 2022 06:10:21 -0400 X-MC-Unique: y528ceswNBifW8e9WJwpEQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4ABF038005D1; Fri, 3 Jun 2022 10:10:09 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.40.192.190]) by smtp.corp.redhat.com (Postfix) with ESMTP id 39019492C3B; Fri, 3 Jun 2022 10:10:04 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: kvm@vger.kernel.org, Jason Wang , "Michael S. Tsirkin" , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, virtualization@lists.linux-foundation.org Cc: Christophe JAILLET , Longpeng , Stefano Garzarella , dinang@xilinx.com, Piotr.Uminski@intel.com, martinpo@xilinx.com, tanuj.kamde@amd.com, Parav Pandit , Zhang Min , habetsm.xilinx@gmail.com, Zhu Lingshan , lulu@redhat.com, hanand@xilinx.com, martinh@xilinx.com, Si-Wei Liu , gautam.dawar@amd.com, Xie Yongji , ecree.xilinx@gmail.com, pabloc@xilinx.com, lvivier@redhat.com, Eli Cohen , Wu Zongyong , Dan Carpenter Subject: [PATCH v5 3/4] vhost-vdpa: uAPI to suspend the device Date: Fri, 3 Jun 2022 12:09:43 +0200 Message-Id: <20220603100944.871727-4-eperezma@redhat.com> In-Reply-To: <20220603100944.871727-1-eperezma@redhat.com> References: <20220603100944.871727-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The ioctl adds support for suspending the device from userspace. This is a must before getting virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa). After a successful return of ioctl with suspend = 1, the device must not process more virtqueue descriptors, and it must not send any config interrupt. The device can answer to read or writes of config fields as if it were not suspended. In particular, writing to "queue_enable" with a value of 1 will not make the device start processing buffers of the virtqueue until the device is resumed (suspend = 0). After a successful return of ioctl with suspend = 0, the device will start processing data of the virtqueues if other expected conditions are met (queue is enabled, DRIVER_OK has already been set to status, etc.) If not, the device should be in the same state as if no call to suspend callback with suspend = 1 has been performed. Signed-off-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 31 +++++++++++++++++++++++++++++++ include/uapi/linux/vhost.h | 14 ++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index f4b492526c6f8..cb47c10bbf471 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -478,6 +478,34 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp) return 0; } +/* After a successful return of ioctl with suspend = 1, the device must not + * process more virtqueue descriptors, and it must not send any config + * interrupt. The device can answer to read or writes of config fields as if it + * were not suspended. In particular, writing to "queue_enable" with a value of + * 1 will not make the device start processing buffers of the virtqueue until + * the device is resumed (suspend = 0). + * + * After a successful return of ioctl with suspend = 0, the device will start + * processing data of the virtqueues if other expected conditions are met + * (queue is enabled, DRIVER_OK has already been set to status, etc.) If not, + * the device should be in the same state as if no call to suspend callback + * with suspend = 1 has been performed. + */ +static long vhost_vdpa_suspend(struct vhost_vdpa *v, u32 __user *argp) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + int suspend; + + if (!ops->suspend) + return -EOPNOTSUPP; + + if (copy_from_user(&suspend, argp, sizeof(suspend))) + return -EFAULT; + + return ops->suspend(vdpa, suspend); +} + static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, void __user *argp) { @@ -652,6 +680,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, case VHOST_VDPA_GET_VQS_COUNT: r = vhost_vdpa_get_vqs_count(v, argp); break; + case VHOST_VDPA_SUSPEND: + r = vhost_vdpa_suspend(v, argp); + break; default: r = vhost_dev_ioctl(&v->vdev, cmd, argp); if (r == -ENOIOCTLCMD) diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h index cab645d4a6455..6d9f451631557 100644 --- a/include/uapi/linux/vhost.h +++ b/include/uapi/linux/vhost.h @@ -171,4 +171,18 @@ #define VHOST_VDPA_SET_GROUP_ASID _IOW(VHOST_VIRTIO, 0x7C, \ struct vhost_vring_state) +/* Suspend or resume a device so it does not process virtqueue requests anymore + * + * After the return of ioctl with suspend != 0, the device must finish any + * pending operations like in flight requests. It must also preserve all the + * necessary state (the virtqueue vring base plus the possible device specific + * states) that is required for restoring in the future. The device must not + * change its configuration after that point. + * + * After the return of ioctl with suspend == 0, the device can continue + * processing buffers as long as typical conditions are met (vq is enabled, + * DRIVER_OK status bit is enabled, etc). + */ +#define VHOST_VDPA_SUSPEND _IOW(VHOST_VIRTIO, 0x7D, int) + #endif From patchwork Fri Jun 3 10:09:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eugenio Perez Martin X-Patchwork-Id: 12868923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2E4EC43334 for ; Fri, 3 Jun 2022 10:10:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243513AbiFCKKk (ORCPT ); Fri, 3 Jun 2022 06:10:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243387AbiFCKKc (ORCPT ); Fri, 3 Jun 2022 06:10:32 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 506413B2BD for ; Fri, 3 Jun 2022 03:10:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654251019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u3+EP9PD/XGXq5/0hvrOKGwV2S5RKvIGOAb6+LG+/kc=; b=Eybym9fWcF9GWENdKKzjAq/poJJ91l1e7MlH2NB8l0xxhjn42ZIp9RJ8oe9+MGVIFKG5Pi 42ARvEA92u31dGx8NTzyso6tSpR1sGEfLmelHuSGIqMKghpi4PZT1WXAGZ+qKZ17DzY67F SBaQQSTxBbR8D1lrc5UydBXuWNZnfvY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-352-5xAbDisiOb6gLvcBpaFx1A-1; Fri, 03 Jun 2022 06:10:15 -0400 X-MC-Unique: 5xAbDisiOb6gLvcBpaFx1A-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AB74380A0B9; Fri, 3 Jun 2022 10:10:14 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.40.192.190]) by smtp.corp.redhat.com (Postfix) with ESMTP id 97681403371; Fri, 3 Jun 2022 10:10:09 +0000 (UTC) From: =?utf-8?q?Eugenio_P=C3=A9rez?= To: kvm@vger.kernel.org, Jason Wang , "Michael S. Tsirkin" , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, virtualization@lists.linux-foundation.org Cc: Christophe JAILLET , Longpeng , Stefano Garzarella , dinang@xilinx.com, Piotr.Uminski@intel.com, martinpo@xilinx.com, tanuj.kamde@amd.com, Parav Pandit , Zhang Min , habetsm.xilinx@gmail.com, Zhu Lingshan , lulu@redhat.com, hanand@xilinx.com, martinh@xilinx.com, Si-Wei Liu , gautam.dawar@amd.com, Xie Yongji , ecree.xilinx@gmail.com, pabloc@xilinx.com, lvivier@redhat.com, Eli Cohen , Wu Zongyong , Dan Carpenter Subject: [PATCH v5 4/4] vdpa_sim: Implement suspend vdpa op Date: Fri, 3 Jun 2022 12:09:44 +0200 Message-Id: <20220603100944.871727-5-eperezma@redhat.com> In-Reply-To: <20220603100944.871727-1-eperezma@redhat.com> References: <20220603100944.871727-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Implement suspend operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively suspend the device. This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa). Reviewed-by: Stefano Garzarella Signed-off-by: Eugenio Pérez --- drivers/vdpa/vdpa_sim/vdpa_sim.c | 21 +++++++++++++++++++++ drivers/vdpa/vdpa_sim/vdpa_sim.h | 1 + drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 3 +++ drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 3 +++ 4 files changed, 28 insertions(+) diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c index 0f28658996472..01f9377830b3e 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c @@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim) for (i = 0; i < vdpasim->dev_attr.nas; i++) vhost_iotlb_reset(&vdpasim->iommu[i]); + vdpasim->running = true; spin_unlock(&vdpasim->iommu_lock); vdpasim->features = 0; @@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa) return 0; } +static int vdpasim_suspend(struct vdpa_device *vdpa, bool suspend) +{ + struct vdpasim *vdpasim = vdpa_to_sim(vdpa); + int i; + + spin_lock(&vdpasim->lock); + vdpasim->running = !suspend; + if (vdpasim->running) { + /* Check for missed buffers */ + for (i = 0; i < vdpasim->dev_attr.nvqs; ++i) + vdpasim_kick_vq(vdpa, i); + + } + spin_unlock(&vdpasim->lock); + + return 0; +} + static size_t vdpasim_get_config_size(struct vdpa_device *vdpa) { struct vdpasim *vdpasim = vdpa_to_sim(vdpa); @@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = { .get_status = vdpasim_get_status, .set_status = vdpasim_set_status, .reset = vdpasim_reset, + .suspend = vdpasim_suspend, .get_config_size = vdpasim_get_config_size, .get_config = vdpasim_get_config, .set_config = vdpasim_set_config, @@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = { .get_status = vdpasim_get_status, .set_status = vdpasim_set_status, .reset = vdpasim_reset, + .suspend = vdpasim_suspend, .get_config_size = vdpasim_get_config_size, .get_config = vdpasim_get_config, .set_config = vdpasim_set_config, diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h index 622782e922391..061986f30911a 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim.h +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h @@ -66,6 +66,7 @@ struct vdpasim { u32 generation; u64 features; u32 groups; + bool running; /* spinlock to synchronize iommu table */ spinlock_t iommu_lock; }; diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c index 42d401d439117..bcdb1982c378e 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c @@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work) if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) goto out; + if (!vdpasim->running) + goto out; + for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) { struct vdpasim_virtqueue *vq = &vdpasim->vqs[i]; diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c index 5125976a4df87..886449e885026 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c @@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work) spin_lock(&vdpasim->lock); + if (!vdpasim->running) + goto out; + if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) goto out;