From patchwork Wed Sep 19 06:28:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tiwei Bie X-Patchwork-Id: 10605369 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1A01112B for ; Wed, 19 Sep 2018 06:31:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8F4F82B647 for ; Wed, 19 Sep 2018 06:31:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 818F12B64E; Wed, 19 Sep 2018 06:31:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E96102B647 for ; Wed, 19 Sep 2018 06:31:40 +0000 (UTC) Received: from localhost ([::1]:43856 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g2W1U-0006gB-6Q for patchwork-qemu-devel@patchwork.kernel.org; Wed, 19 Sep 2018 02:31:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45347) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g2Vzn-0006eX-W8 for qemu-devel@nongnu.org; Wed, 19 Sep 2018 02:29:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g2Vzm-0002ek-UC for qemu-devel@nongnu.org; Wed, 19 Sep 2018 02:29:55 -0400 Received: from mga11.intel.com ([192.55.52.93]:1984) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g2Vzm-0002dt-L6 for qemu-devel@nongnu.org; Wed, 19 Sep 2018 02:29:54 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Sep 2018 23:29:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,392,1531810800"; d="scan'208";a="71164021" Received: from btwcube1.sh.intel.com ([10.67.104.151]) by fmsmga007.fm.intel.com with ESMTP; 18 Sep 2018 23:29:43 -0700 From: Tiwei Bie To: mst@redhat.com, alex.williamson@redhat.com, jasowang@redhat.com, qemu-devel@nongnu.org Date: Wed, 19 Sep 2018 14:28:32 +0800 Message-Id: <20180919062834.30103-3-tiwei.bie@intel.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180919062834.30103-1-tiwei.bie@intel.com> References: <20180919062834.30103-1-tiwei.bie@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.93 Subject: [Qemu-devel] [RFC v2 2/4] vhost-user: support programming VFIO container in master X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tiwei.bie@intel.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces a slave message to allow slave to share its VFIO container fd to master and do the IOMMU programming based on virtio device's DMA address space for the VFIO groups inside this VFIO container in QEMU. For the vhost backends which support vDPA, they can leverage this message to ask master to do the IOMMU programming in QEMU for the vDPA device in backend. Signed-off-by: Tiwei Bie --- docs/interop/vhost-user.txt | 21 ++++++++++++++++ hw/virtio/vhost-user.c | 45 ++++++++++++++++++++++++++++++++++ include/hw/virtio/vhost-user.h | 2 ++ 3 files changed, 68 insertions(+) diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index f59667f498..2ac250aa01 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -397,6 +397,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_CONFIG 9 #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10 #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 +#define VHOST_USER_PROTOCOL_F_VFIO_CONTAINER 12 Master message types -------------------- @@ -815,6 +816,26 @@ Slave message types This request should be sent only when VHOST_USER_PROTOCOL_F_HOST_NOTIFIER protocol feature has been successfully negotiated. + * VHOST_USER_SLAVE_VFIO_CONTAINER_MSG + + Id: 4 + Equivalent ioctl: N/A + Slave payload: N/A + Master payload: N/A + + When VHOST_USER_PROTOCOL_F_VFIO_CONTAINER is negotiated, vhost-user + slave could send this request to share its VFIO container fd via + ancillary data to master. Before sending a VFIO container fd to + master, slave should make sure that IOMMU type has already been + set correctly. After receiving this request from slave, master will + destroy the existing VFIO container (including clearing all the DMA + mappings and closing the container fd) if any and setup a new VFIO + container (including clearing all the existing DMA mappings in this + new container and setup DMA mappings for this container based on + virtio device's DMA address space) if the request is sent with a + file descriptor. + + VHOST_USER_PROTOCOL_F_REPLY_ACK: ------------------------------- The original vhost-user specification only demands replies for certain diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index b041343632..d53787529d 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -52,6 +52,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_CONFIG = 9, VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD = 10, VHOST_USER_PROTOCOL_F_HOST_NOTIFIER = 11, + VHOST_USER_PROTOCOL_F_VFIO_CONTAINER = 12, VHOST_USER_PROTOCOL_F_MAX }; @@ -97,6 +98,7 @@ typedef enum VhostUserSlaveRequest { VHOST_USER_SLAVE_IOTLB_MSG = 1, VHOST_USER_SLAVE_CONFIG_CHANGE_MSG = 2, VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG = 3, + VHOST_USER_SLAVE_VFIO_CONTAINER_MSG = 4, VHOST_USER_SLAVE_MAX } VhostUserSlaveRequest; @@ -949,6 +951,41 @@ static int vhost_user_slave_handle_vring_host_notifier(struct vhost_dev *dev, return 0; } +static int vhost_user_slave_handle_vfio_container(struct vhost_dev *dev, + int *fd) +{ + struct vhost_user *u = dev->opaque; + VhostUserState *user = u->user; + VirtIODevice *vdev = dev->vdev; + int container_fd = fd[0]; + VFIOContainer *container; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_VFIO_CONTAINER) || + vdev == NULL) { + return -1; + } + + if (user->vfio_container) { + vfio_free_container(user->vfio_container); + user->vfio_container = NULL; + } + + if (container_fd < 0) { + return 0; + } + + container = vfio_new_container(container_fd, vdev->dma_as, NULL); + if (container == NULL) { + return -1; + } + + user->vfio_container = container; + fd[0] = -1; + + return 0; +} + static void slave_read(void *opaque) { struct vhost_dev *dev = opaque; @@ -1021,6 +1058,9 @@ static void slave_read(void *opaque) ret = vhost_user_slave_handle_vring_host_notifier(dev, &payload.area, fd[0]); break; + case VHOST_USER_SLAVE_VFIO_CONTAINER_MSG: + ret = vhost_user_slave_handle_vfio_container(dev, fd); + break; default: error_report("Received unexpected msg type."); ret = -EINVAL; @@ -1761,6 +1801,11 @@ void vhost_user_cleanup(VhostUserState *user) user->notifier[i].addr = NULL; } } + + if (user->vfio_container) { + vfio_free_container(user->vfio_container); + user->vfio_container = NULL; + } } const VhostOps user_ops = { diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h index fd660393a0..99c6dbbbff 100644 --- a/include/hw/virtio/vhost-user.h +++ b/include/hw/virtio/vhost-user.h @@ -10,6 +10,7 @@ #include "chardev/char-fe.h" #include "hw/virtio/virtio.h" +#include "hw/vfio/vfio-common.h" typedef struct VhostUserHostNotifier { MemoryRegion mr; @@ -20,6 +21,7 @@ typedef struct VhostUserHostNotifier { typedef struct VhostUserState { CharBackend *chr; VhostUserHostNotifier notifier[VIRTIO_QUEUE_MAX]; + VFIOContainer *vfio_container; } VhostUserState; VhostUserState *vhost_user_init(void);