From patchwork Thu Mar 16 12:41:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D38E3C74A5B for ; Thu, 16 Mar 2023 12:42:21 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 216E610E0E8; Thu, 16 Mar 2023 12:42:21 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id 35DC610E0FA; Thu, 16 Mar 2023 12:42:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970539; x=1710506539; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+VKOyERweJfYAdfXicxL5YrmE4b4662Jc+V+KNNxwM8=; b=aL9fqJJTxJEIq/9PZVPo4cxq2osnpCtGZm/5Gu+uTZBGFSFFyo3Fc2KQ Bc9VZPqz8WU1PcnzYXlasmVd9+VtzaADvbW+oZByWFkmFh5gS4gKmG8kx hlUwkeyoEQDhu6zEZx41LJncOBkejO8BrYa9FkiuoWU6G9WX6OgHgF/2x IudH5Qmg+DFOtp2pqcziF1pzOWdFJMGbmPcqbjfdZQzrtnNFjhBsN8vcH Q22RkiL3GGCT2001FuYvSFYMabWW8AORyo4K6MgEgahPj+Q/eaQjdCNpG zVYe4uBRgugZ4w2vmuQgYY1U2OgBi9Ew41Z24yK/8Y0Xk3XQY9YGa8ip+ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321811996" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321811996" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009207813" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009207813" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:41:59 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:50 -0700 Message-Id: <20230316124156.12064-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 1/7] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" this suits more on what the code does. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a5ab416cf476..65bbef562268 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } /* - * For each group_fd, get the group through the vfio external user - * interface and store the group and iommu ID. This ensures the group - * is held across the reset. + * Get the group file for each fd to ensure the group held across + * the reset */ for (file_idx = 0; file_idx < hdr.count; file_idx++) { struct file *file = fget(group_fds[file_idx]); From patchwork Thu Mar 16 12:41:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3F465C7618E for ; Thu, 16 Mar 2023 12:42:29 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5430A10E189; Thu, 16 Mar 2023 12:42:27 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id 59FA910E19F; Thu, 16 Mar 2023 12:42:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970542; x=1710506542; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AEvwWlClzX1rnKaz+xfqG9UfvXCn9+h9Ct8yRH9iLyU=; b=akyDYj/TrtfTsI6aGEhwaW5/Zc5HK1CSUpKocK7/JrHkaR3YRTpQqxmm OzcuPjAQt5yGtLyFXJaY8cGowDT+vaOf12c339uVKe2fmeUf+E1gZRW44 OUW6bOGOzDSEuBicRsiZlv8/OrZQnM2xcRo9O1EEP+cbxjxSRkgk/ICtG vYEAESCr9SHwd21FTybtdt5RZNygsngc00rl9rfUCneoaagH4TaGAW0tK zXIPvqRJTFe53nhNHi2Qjx2YGiOuVyrazxbP3GZFP9II33W9BFwSdxhev w438wx/nhrB7a/mN5OO/4L3yXE19dFHWqWG16RVrJFOqL9l4CSDpGwfyH g==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321812059" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321812059" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009207887" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009207887" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:42:08 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:51 -0700 Message-Id: <20230316124156.12064-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 2/7] vfio/pci: Only check ownership of opened devices in hot reset X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" If the affected device is not opened by any user, it's safe to reset it given it's not in use. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++++--- include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 65bbef562268..5d745c9abf05 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2429,10 +2429,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) { /* - * Test whether all the affected devices are contained by the - * set of groups provided by the user. + * Test whether all the affected devices can be reset by the + * user. + * + * Resetting an unused device (not opened) is safe, because + * dev_set->lock is held in hot reset path so this device + * cannot race being opened by another user simultaneously. + * + * Otherwise all opened devices in the dev_set must be + * contained by the set of groups provided by the user. */ - if (!vfio_dev_in_groups(cur_vma, groups)) { + if (cur_vma->vdev.open_count && + !vfio_dev_in_groups(cur_vma, groups)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0552e8dcf0cb..f96e5689cffc 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -673,6 +673,14 @@ struct vfio_pci_hot_reset_info { * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * + * Userspace requests hot reset for the devices it uses. Due to the + * underlying topology, multiple devices can be affected in the reset + * while some might be opened by another user. To avoid interference + * the calling user must ensure all affected devices, if opened, are + * owned by itself. + * + * The ownership is proved by an array of group fds. + * * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset { From patchwork Thu Mar 16 12:41:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72C48C6FD19 for ; Thu, 16 Mar 2023 12:42:27 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C3E3110E0FA; Thu, 16 Mar 2023 12:42:26 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id B0C4F10E19F; Thu, 16 Mar 2023 12:42:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970543; x=1710506543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/P7IVz4x0RFbFGWsYEmkd4pnPBYbK35RmNGyDNdhegc=; b=Lpec+h4OYRTjkDhTlHZdA4ULFBh7f7VFPahuGkyBOFW1k0VmknfZudss R4XgHKg+G8jikdOTYhSAkA9Z/YUEv7Ip1s/rm823HcAYIEqrAyR1FpK6P 3TPOp7GWn0ymjHsoYZVw7aqdkoZNatg660VRVDBFJNBfnyPupUOzmLU3H nYw1oJ8YH6mK8xLgfJQGA151yUlMpcOYTpCUS7YMA63YTh86SuBszETGu ec0Mg/TzRfbuL4KKeconHZh8LrYjnJTxsaRahWGLoR36VtspmCyLYAdUG M08EGxUCGtsqX9h1bh1hqdbFgXs5ZqQAyYaBCzZOWoaSo7wjv3pfuPfKs Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321812073" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321812073" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009207924" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009207924" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:42:09 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:52 -0700 Message-Id: <20230316124156.12064-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 3/7] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds. a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/iommu/iommufd/device.c | 6 ++ drivers/vfio/iommufd.c | 8 +++ drivers/vfio/pci/vfio_pci_core.c | 94 +++++++++++++++++++++++--------- include/linux/iommufd.h | 1 + include/linux/vfio.h | 3 + include/uapi/linux/vfio.h | 9 ++- 6 files changed, 93 insertions(+), 28 deletions(-) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 0295140dd384..2ca12716db98 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -131,6 +131,12 @@ void iommufd_device_unbind(struct iommufd_device *idev) } EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev) +{ + return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD); + static int iommufd_device_setup_msi(struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt, phys_addr_t sw_msi_start) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 9aabd8b31c15..ca0c16bb747e 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -66,6 +66,14 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd(vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + if (!vdev->iommufd_device) + return NULL; + return iommufd_device_to_ictx(vdev->iommufd_device); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx); + /* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 5d745c9abf05..b68fcba67a4b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) struct vfio_pci_group_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups); + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx); /* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND @@ -1255,29 +1256,17 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return ret; } -static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); - struct vfio_pci_hot_reset hdr; int32_t *group_fds; struct file **files; struct vfio_pci_group_info info; - bool slot = false; int file_idx, count = 0, ret = 0; - if (copy_from_user(&hdr, arg, minsz)) - return -EFAULT; - - if (hdr.argsz < minsz || hdr.flags) - return -EINVAL; - - /* Can we do a slot or bus reset or neither? */ - if (!pci_probe_reset_slot(vdev->pdev->slot)) - slot = true; - else if (pci_probe_reset_bus(vdev->pdev->bus)) - return -ENODEV; - /* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have @@ -1289,11 +1278,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr.count || hdr.count > count) + if (hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL); - files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL); + group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); if (!group_fds || !files) { kfree(group_fds); kfree(files); @@ -1301,7 +1290,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } if (copy_from_user(group_fds, arg->group_fds, - hdr.count * sizeof(*group_fds))) { + hdr->count * sizeof(*group_fds))) { kfree(group_fds); kfree(files); return -EFAULT; @@ -1311,7 +1300,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, * Get the group file for each fd to ensure the group held across * the reset */ - for (file_idx = 0; file_idx < hdr.count; file_idx++) { + for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(group_fds[file_idx]); if (!file) { @@ -1335,10 +1324,10 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = hdr->count; info.files = files; - ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); + ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL); hot_reset_release: for (file_idx--; file_idx >= 0; file_idx--) @@ -1348,6 +1337,34 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); + struct vfio_pci_hot_reset hdr; + struct iommufd_ctx *iommufd; + bool slot = false; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz || hdr.flags) + return -EINVAL; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + if (hdr.count) + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, iommufd); +} + static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, struct vfio_device_ioeventfd __user *arg) { @@ -2317,6 +2334,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, { unsigned int i; + if (!groups) + return false; + for (i = 0; i < groups->count; i++) if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) return true; @@ -2392,13 +2412,25 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) return ret; } +static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, + struct iommufd_ctx *iommufd_ctx) +{ + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + if (!iommufd) + return false; + + return iommufd == iommufd_ctx; +} + /* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups) + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; struct vfio_pci_core_device *cur_vma; @@ -2438,9 +2470,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * * Otherwise all opened devices in the dev_set must be * contained by the set of groups provided by the user. + * + * If user provides a zero-length array, then all the + * opened devices must be bound to a same iommufd_ctx. + * + * If all above checks are failed, reset is allowed only if + * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups)) { + !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && + (dev_set->device_count > 1)) { ret = -EINVAL; goto err_undo; } diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 1129a36a74c4..035d5d28e612 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -20,6 +20,7 @@ struct file; struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, struct device *dev, u32 *id); void iommufd_device_unbind(struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); void iommufd_device_detach(struct iommufd_device *idev); diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 3188d8a374bd..f0a5ff317b20 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -116,6 +116,7 @@ struct vfio_device_ops { int vfio_iommufd_physical_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_physical_unbind(struct vfio_device *vdev); +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev); int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id); int vfio_iommufd_emulated_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); @@ -127,6 +128,8 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); u32 *out_device_id)) NULL) #define vfio_iommufd_physical_unbind \ ((void (*)(struct vfio_device *vdev)) NULL) +#define vfio_iommufd_physical_ictx \ + ((struct iommufd_ctx * (*)(struct vfio_device *vdev)) NULL) #define vfio_iommufd_physical_attach_ioas \ ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) #define vfio_iommufd_emulated_bind \ diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index f96e5689cffc..17aa5d09db41 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -679,7 +679,14 @@ struct vfio_pci_hot_reset_info { * the calling user must ensure all affected devices, if opened, are * owned by itself. * - * The ownership is proved by an array of group fds. + * The ownership can be proved by: + * - An array of group fds + * - A zero-length array + * + * In the last case all affected devices which are opened by this user + * must have been bound to a same iommufd. If the calling device is in + * noiommu mode (no valid iommufd) then it can be reset only if the reset + * doesn't affect other devices. * * Return: 0 on success, -errno on failure. */ From patchwork Thu Mar 16 12:41:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B712C7618D for ; Thu, 16 Mar 2023 12:42:38 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E994F10E203; Thu, 16 Mar 2023 12:42:35 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4D36910E1EB; Thu, 16 Mar 2023 12:42:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970544; x=1710506544; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5VOwMbZ7NFEp3Vz3lWCBa2rwgc7032UbcA9JSi84oFw=; b=PKPVLC3qtPzSd9lh+0OEdioQxPJRET9bELqHs0zmVgnQErJp/BQmZA8X BOhF7J0Rp+1+qjxZQ3HGBfxjsA/CC4FpQMjGvVUsdWJroTrsKE0VUEhuj g4S789RnC77YKnJSqk6PVNZoq4kOFL6H68bUfe8QxLjy/pyORLvJkzV6N elAp+9QkLQ+IwQeGh0WCpeybDSx1rAEz5XbflGRcSSD2swfPy+yUn6ej5 T2eYL9EpsJqWCWY5agMx7YEr2d08FfYmkbVhN+zxnwKKTlRSyOwIy9Xw4 ChNpnjdhz2hjbOSfCroJ2hqVFgMbDznz+xaOZ9J9orw9gfD3rr7wVq8rh A==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321812086" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321812086" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009207959" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009207959" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:42:11 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:53 -0700 Message-Id: <20230316124156.12064-5-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 4/7] vfio/pci: Renaming for accepting device fd in hot reset path X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" No functional change is intended. Signed-off-by: Yi Liu Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 55 ++++++++++++++++---------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b68fcba67a4b..b6b5624c8b15 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -177,10 +177,10 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) } } -struct vfio_pci_group_info; +struct vfio_pci_file_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx); /* @@ -800,7 +800,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) return 0; } -struct vfio_pci_group_info { +struct vfio_pci_file_info { int count; struct file **files; }; @@ -1257,14 +1257,14 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( } static int -vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset *hdr, - bool slot, - struct vfio_pci_hot_reset __user *arg) +vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - int32_t *group_fds; + int32_t *fds; struct file **files; - struct vfio_pci_group_info info; + struct vfio_pci_file_info info; int file_idx, count = 0, ret = 0; /* @@ -1281,17 +1281,17 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, if (hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + fds = kcalloc(hdr->count, sizeof(*fds), GFP_KERNEL); files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); - if (!group_fds || !files) { - kfree(group_fds); + if (!fds || !files) { + kfree(fds); kfree(files); return -ENOMEM; } - if (copy_from_user(group_fds, arg->group_fds, - hdr->count * sizeof(*group_fds))) { - kfree(group_fds); + if (copy_from_user(fds, arg->group_fds, + hdr->count * sizeof(*fds))) { + kfree(fds); kfree(files); return -EFAULT; } @@ -1301,7 +1301,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, * the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { - struct file *file = fget(group_fds[file_idx]); + struct file *file = fget(fds[file_idx]); if (!file) { ret = -EBADF; @@ -1318,9 +1318,9 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, files[file_idx] = file; } - kfree(group_fds); + kfree(fds); - /* release reference to groups on error */ + /* release reference to fds on error */ if (ret) goto hot_reset_release; @@ -1358,7 +1358,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return -ENODEV; if (hdr.count) - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + return vfio_pci_ioctl_pci_hot_reset_files(vdev, &hdr, slot, arg); iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); @@ -2329,16 +2329,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { }; EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_group_info *groups) +static bool vfio_dev_in_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_file_info *info) { unsigned int i; - if (!groups) + if (!info) return false; - for (i = 0; i < groups->count; i++) - if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) + for (i = 0; i < info->count; i++) + if (vfio_file_has_dev(info->files[i], &vdev->vdev)) return true; return false; } @@ -2429,7 +2429,7 @@ static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; @@ -2469,7 +2469,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * cannot race being opened by another user simultaneously. * * Otherwise all opened devices in the dev_set must be - * contained by the set of groups provided by the user. + * contained by the set of groups/devices provided by + * the user. * * If user provides a zero-length array, then all the * opened devices must be bound to a same iommufd_ctx. @@ -2478,7 +2479,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_files(cur_vma, info) && !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && (dev_set->device_count > 1)) { ret = -EINVAL; From patchwork Thu Mar 16 12:41:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3792BC7619A for ; Thu, 16 Mar 2023 12:42:31 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CD90910E205; Thu, 16 Mar 2023 12:42:28 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id A6F1B10E19F; Thu, 16 Mar 2023 12:42:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970544; x=1710506544; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NicSx/xyczYjqPJzsnHFFZdxot+M1Q+E4aTGTrK/kVA=; b=OBopAghLRk/hw/5uQRUnV41J8g31RvlPz3znKCLMIEzNqR2wwtlzsmuQ mRll5IHIxULtLHvAsfQmZUnEcVW+mwyP1Q4sI4rp5DLcTgWebqi52hxCN YKfKtPQeAH1lvvZpnyol3cCYRQpKVZ7B6COXrvahDWxgs6FLlJBD7EX3n H83D8yx2UUygq165bGS8Y13S0cdHfrk7litle85DGuw6h27khZy+OQfE7 w/kbm7FY99ByyijZwOw663kNKe9aoGoIGALNqzw7NjXFoP3bot7RTHzo0 FW57mjCOHh8Z2pMXWuRVIupw2TZozQ5EqJcu4MpNYCQjYwr5zicvjcTXL Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321812102" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321812102" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009207983" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009207983" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:42:12 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:54 -0700 Message-Id: <20230316124156.12064-6-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 5/7] vfio: Refine vfio file kAPIs for vfio PCI hot reset X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage. Signed-off-by: Yi Liu Reviewed-by: Kevin Tian --- drivers/vfio/group.c | 32 ++++++++++++++------------------ drivers/vfio/pci/vfio_pci_core.c | 4 ++-- drivers/vfio/vfio.h | 2 ++ drivers/vfio/vfio_main.c | 29 +++++++++++++++++++++++++++++ include/linux/vfio.h | 1 + 5 files changed, 48 insertions(+), 20 deletions(-) diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c index 27d5ba7cf9dc..d0c95d033605 100644 --- a/drivers/vfio/group.c +++ b/drivers/vfio/group.c @@ -745,6 +745,15 @@ bool vfio_device_has_container(struct vfio_device *device) return device->group->container; } +struct vfio_group *vfio_group_from_file(struct file *file) +{ + struct vfio_group *group = file->private_data; + + if (file->f_op != &vfio_group_fops) + return NULL; + return group; +} + /** * vfio_file_iommu_group - Return the struct iommu_group for the vfio group file * @file: VFIO group file @@ -755,13 +764,13 @@ bool vfio_device_has_container(struct vfio_device *device) */ struct iommu_group *vfio_file_iommu_group(struct file *file) { - struct vfio_group *group = file->private_data; + struct vfio_group *group = vfio_group_from_file(file); struct iommu_group *iommu_group = NULL; if (!IS_ENABLED(CONFIG_SPAPR_TCE_IOMMU)) return NULL; - if (!vfio_file_is_group(file)) + if (!group) return NULL; mutex_lock(&group->group_lock); @@ -775,12 +784,12 @@ struct iommu_group *vfio_file_iommu_group(struct file *file) EXPORT_SYMBOL_GPL(vfio_file_iommu_group); /** - * vfio_file_is_group - True if the file is usable with VFIO aPIS + * vfio_file_is_group - True if the file is a vfio group file * @file: VFIO group file */ bool vfio_file_is_group(struct file *file) { - return file->f_op == &vfio_group_fops; + return vfio_group_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_group); @@ -842,23 +851,10 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm) } EXPORT_SYMBOL_GPL(vfio_file_set_kvm); -/** - * vfio_file_has_dev - True if the VFIO file is a handle for device - * @file: VFIO file to check - * @device: Device that must be part of the file - * - * Returns true if given file has permission to manipulate the given device. - */ -bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device) { - struct vfio_group *group = file->private_data; - - if (!vfio_file_is_group(file)) - return false; - return group == device->group; } -EXPORT_SYMBOL_GPL(vfio_file_has_dev); static char *vfio_devnode(const struct device *dev, umode_t *mode) { diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b6b5624c8b15..b7de1816b97b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,8 +1308,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, break; } - /* Ensure the FD is a vfio group FD.*/ - if (!vfio_file_is_group(file)) { + /* Ensure the FD is a vfio FD. vfio group or vfio device */ + if (!vfio_file_is_valid(file)) { fput(file); ret = -EINVAL; break; diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h index 7b19c621e0e6..c0aeea24fbd6 100644 --- a/drivers/vfio/vfio.h +++ b/drivers/vfio/vfio.h @@ -84,6 +84,8 @@ void vfio_device_group_unregister(struct vfio_device *device); int vfio_device_group_use_iommu(struct vfio_device *device); void vfio_device_group_unuse_iommu(struct vfio_device *device); void vfio_device_group_close(struct vfio_device *device); +struct vfio_group *vfio_group_from_file(struct file *file); +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device); bool vfio_device_has_container(struct vfio_device *device); int __init vfio_group_init(void); void vfio_group_cleanup(void); diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 89497c933490..fe7446805afd 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,6 +1154,35 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +/** + * vfio_file_is_valid - True if the file is valid vfio file + * @file: VFIO group file or VFIO device file + */ +bool vfio_file_is_valid(struct file *file) +{ + return vfio_group_from_file(file); +} +EXPORT_SYMBOL_GPL(vfio_file_is_valid); + +/** + * vfio_file_has_dev - True if the VFIO file is a handle for device + * @file: VFIO file to check + * @device: Device that must be part of the file + * + * Returns true if given file has permission to manipulate the given device. + */ +bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +{ + struct vfio_group *group; + + group = vfio_group_from_file(file); + if (!group) + return false; + + return vfio_group_has_dev(group, device); +} +EXPORT_SYMBOL_GPL(vfio_file_has_dev); + /* * Sub-module support */ diff --git a/include/linux/vfio.h b/include/linux/vfio.h index f0a5ff317b20..5a2e8a9d538d 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -248,6 +248,7 @@ int vfio_mig_get_next_state(struct vfio_device *device, */ struct iommu_group *vfio_file_iommu_group(struct file *file); bool vfio_file_is_group(struct file *file); +bool vfio_file_is_valid(struct file *file); bool vfio_file_enforced_coherent(struct file *file); void vfio_file_set_kvm(struct file *file, struct kvm *kvm); bool vfio_file_has_dev(struct file *file, struct vfio_device *device); From patchwork Thu Mar 16 12:41:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E13D7C6FD1F for ; Thu, 16 Mar 2023 12:42:36 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C1AE110E19D; Thu, 16 Mar 2023 12:42:35 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id 800F610E19F; Thu, 16 Mar 2023 12:42:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970545; x=1710506545; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=21n59NueoZg+YCLGA7KIVcAZ1swA7rO+1yGotBMrp8E=; b=C2VHYO2GNbE0sx9KG1RKO1UgBA0jPFCSUGLFijqhwlgv73vIhvh0LPHY I5HF5nWlCJLG0cYlzbDb4hLlu1SrVLq9BWySm80FpxzgUBUWGURgF8SrY t5ElKKCi77etfkk8mlqMe6yjcWINgq64GNf3Dbb6shBa9bLO6ETtfZ5rS yl/SCgD0UrOK6CegEkkbxtuE9QjnqdqBzC6pY3sMALraV7NSDTRjbPHcV VlvedpnadZmmMRAu36E71mdC5Wr8/w6v17NMDU09R/Vz9wzGocBJWxA2O WVrLeDUwbdrhBTv6wKMcIKqgUhu1c+bUR89aPYr78WlquETZ4FhWyJwxp g==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321812113" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321812113" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009208007" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009208007" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:42:13 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:55 -0700 Message-Id: <20230316124156.12064-7-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 6/7] vfio: Accpet device file from vfio PCI hot reset path X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset. Signed-off-by: Yi Liu Reviewed-by: Kevin Tian --- drivers/vfio/vfio_main.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index fe7446805afd..ebbb6b91a498 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,13 +1154,23 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +static struct vfio_device *vfio_device_from_file(struct file *file) +{ + struct vfio_device *device = file->private_data; + + if (file->f_op != &vfio_device_fops) + return NULL; + return device; +} + /** * vfio_file_is_valid - True if the file is valid vfio file * @file: VFIO group file or VFIO device file */ bool vfio_file_is_valid(struct file *file) { - return vfio_group_from_file(file); + return vfio_group_from_file(file) || + vfio_device_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_valid); @@ -1174,12 +1184,17 @@ EXPORT_SYMBOL_GPL(vfio_file_is_valid); bool vfio_file_has_dev(struct file *file, struct vfio_device *device) { struct vfio_group *group; + struct vfio_device *vdev; group = vfio_group_from_file(file); - if (!group) - return false; + if (group) + return vfio_group_has_dev(group, device); + + vdev = vfio_device_from_file(file); + if (vdev) + return vdev == device; - return vfio_group_has_dev(group, device); + return false; } EXPORT_SYMBOL_GPL(vfio_file_has_dev); From patchwork Thu Mar 16 12:41:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13177543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CE674C6FD19 for ; Thu, 16 Mar 2023 12:42:36 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E5A6D10E19F; Thu, 16 Mar 2023 12:42:35 +0000 (UTC) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by gabe.freedesktop.org (Postfix) with ESMTPS id DAFC310E189; Thu, 16 Mar 2023 12:42:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678970546; x=1710506546; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=z/3DeM31wos4hPFJcOhBa/QKGeGtDvnA4E5EZwXXXyw=; b=NISSkGkia/pircvDK/6Rat2xbxNmOCAd7NDjNIS1AfAZQVemalzqtqso f15aOjOt5HtGlEZuEfjxPfdR6uAs257cy9nIvbST/iDSzZ+DPR91h7Z6n 0VTjQF7bv9/MnFw9CHIv1vixK1lZwJygYajtqTykur31DNq/A5YDoFxxd yhNaYZmQOGl4T7x/C/u0AoF6qDOjuPFqKiNwcK+MQ1Cx9/DazE64JZ7cg E/8oM0KCpMifoZG5CNjqkjq8EiiNi6vpxCH0LvaibioztSj7E8CmrsT8j UzlYPisJ4Ok0sHf5cypMmoGoDSmC8rVnFkZknGocjgdBV9AbBSxexufZH A==; X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="321812128" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="321812128" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2023 05:42:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10650"; a="1009208018" X-IronPort-AV: E=Sophos;i="5.98,265,1673942400"; d="scan'208";a="1009208018" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga005.fm.intel.com with ESMTP; 16 Mar 2023 05:42:14 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Thu, 16 Mar 2023 05:41:56 -0700 Message-Id: <20230316124156.12064-8-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230316124156.12064-1-yi.l.liu@intel.com> References: <20230316124156.12064-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 7/7] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, mjrosato@linux.ibm.com, kvm@vger.kernel.org, intel-gvt-dev@lists.freedesktop.org, joro@8bytes.org, cohuck@redhat.com, xudong.hao@intel.com, peterx@redhat.com, yan.y.zhao@intel.com, eric.auger@redhat.com, terrence.xu@intel.com, nicolinc@nvidia.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, intel-gfx@lists.freedesktop.org, chao.p.peng@linux.intel.com, lulu@redhat.com, robin.murphy@arm.com, jasowang@redhat.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages. Clarify it in uAPI. Signed-off-by: Yi Liu Reviewed-by: Kevin Tian --- drivers/vfio/pci/vfio_pci_core.c | 6 +++--- include/uapi/linux/vfio.h | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b7de1816b97b..19f5b075d70a 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1289,7 +1289,7 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, return -ENOMEM; } - if (copy_from_user(fds, arg->group_fds, + if (copy_from_user(fds, arg->fds, hdr->count * sizeof(*fds))) { kfree(fds); kfree(files); @@ -1297,8 +1297,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, } /* - * Get the group file for each fd to ensure the group held across - * the reset + * Get the file for each fd to ensure the group/device file + * is held across the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(fds[file_idx]); diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 17aa5d09db41..25432ef213ee 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -681,6 +681,7 @@ struct vfio_pci_hot_reset_info { * * The ownership can be proved by: * - An array of group fds + * - An array of device fds * - A zero-length array * * In the last case all affected devices which are opened by this user @@ -694,7 +695,7 @@ struct vfio_pci_hot_reset { __u32 argsz; __u32 flags; __u32 count; - __s32 group_fds[]; + __s32 fds[]; }; #define VFIO_DEVICE_PCI_HOT_RESET _IO(VFIO_TYPE, VFIO_BASE + 13)