From patchwork Mon Mar 27 09:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C691C76195 for ; Mon, 27 Mar 2023 09:35:03 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 93F7910E344; Mon, 27 Mar 2023 09:35:02 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id ABE4310E363; Mon, 27 Mar 2023 09:35:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909701; x=1711445701; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+VKOyERweJfYAdfXicxL5YrmE4b4662Jc+V+KNNxwM8=; b=Wwt6Vkocz+dhK/RdOPpFjDhaCE3C1I9dUnM+KEko/Tpvm3HCSbb2XZ1a etYOWbrAAM7GyGGAr7mzCcljg5uJffImC8xyoYX+y1GUUwGwBG9EbggZa Yr3t1QvnyOcFXHy3xpguktibbj74coppkTDlV2ba9MKMARoryc0RorkLD vaMkCa+U7gb35hmY1D4pBLwiGzWTpfIjFq2osJvuDoR4CveuKsTduOQNr LSzy8pgNSvAajJmGKP3SBuCim9pg/TVqvoMnmSzxJ+w3yAImRQLAEQ72+ dglfFDCj6HsXg2DuW+9aBzesD7cg0efHfiZSZL2c/hLXA4o12bDAknfqx Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879489" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879489" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554605" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554605" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:00 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:49 -0700 Message-Id: <20230327093458.44939-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 01/10] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" this suits more on what the code does. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a5ab416cf476..65bbef562268 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } /* - * For each group_fd, get the group through the vfio external user - * interface and store the group and iommu ID. This ensures the group - * is held across the reset. + * Get the group file for each fd to ensure the group held across + * the reset */ for (file_idx = 0; file_idx < hdr.count; file_idx++) { struct file *file = fget(group_fds[file_idx]); From patchwork Mon Mar 27 09:34:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A56AC761A6 for ; Mon, 27 Mar 2023 09:35:05 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0962D10E363; Mon, 27 Mar 2023 09:35:05 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9F94910E363; Mon, 27 Mar 2023 09:35:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909702; x=1711445702; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=d7FNJi55AD86wuPAjqm6ryTv0JA15p82567AVXSPbzY=; b=lFuKqodkXF8Xz2jHnvDaU/ES0Z/pR/dakRiPzmgNekj8XLs4Hptpq7lM jOgsmpCWx4D3TWd30fdNLiTGfSOr/30V4QaqtwmXB+VBfy1S5ijRQ79hE JKNSFroR4eLMAJjdukhnKnL0XD2/dEx/8MGItMRDIc8p1Ur3M4cdSycC2 jVBrEjKML38Wt1P/c+D9a+J6f6iGpXnG0N4PP+tLt0TrU3GJpRII2bi+H TvFyY4jid8+hM9Qx5cVLiRd7GVN75b3s47rNTBK3YCzkYZ6SHZuY8TF/4 mXo3EMM9H2U7kt/2RIMxFHnUpbeKbfmSB4q1OOr2zPJgk+AD8F3JM0QuK A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879502" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879502" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554616" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554616" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:01 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:50 -0700 Message-Id: <20230327093458.44939-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 02/10] vfio/pci: Only check ownership of opened devices in hot reset X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" If the affected device is not opened by any user, it's safe to reset it given it's not in use. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++++--- include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 65bbef562268..5d745c9abf05 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2429,10 +2429,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) { /* - * Test whether all the affected devices are contained by the - * set of groups provided by the user. + * Test whether all the affected devices can be reset by the + * user. + * + * Resetting an unused device (not opened) is safe, because + * dev_set->lock is held in hot reset path so this device + * cannot race being opened by another user simultaneously. + * + * Otherwise all opened devices in the dev_set must be + * contained by the set of groups provided by the user. */ - if (!vfio_dev_in_groups(cur_vma, groups)) { + if (cur_vma->vdev.open_count && + !vfio_dev_in_groups(cur_vma, groups)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0552e8dcf0cb..f96e5689cffc 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -673,6 +673,14 @@ struct vfio_pci_hot_reset_info { * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * + * Userspace requests hot reset for the devices it uses. Due to the + * underlying topology, multiple devices can be affected in the reset + * while some might be opened by another user. To avoid interference + * the calling user must ensure all affected devices, if opened, are + * owned by itself. + * + * The ownership is proved by an array of group fds. + * * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset { From patchwork Mon Mar 27 09:34:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 33E9EC7619A for ; Mon, 27 Mar 2023 09:35:17 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6D25410E3D9; Mon, 27 Mar 2023 09:35:16 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7530610E363; Mon, 27 Mar 2023 09:35:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909703; x=1711445703; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jbMe+F9CADAi8l9/BVZxSAdSYRuI/B7FG31MAnL6+cI=; b=Riwf1Mzlb+GFZxwjqF+EbyhCL0/aWtscF9BeFk0dhMk44O7Z+IvCLNZI KXSXyMoiVIqBvfD27JdtMBPLNCIZ3byoAsGuZLSTJ7rnoq/f8/WHTEfkD /9ae4G7j81hZ82Y+37zZOZCBmXnsGV3gMJIOUcQ+RRtZfc4KnrV+ukpE+ cGjoUoI3lR9o47Nv6HlMySQcGBrgAHKr2R9R4rJP/9zSuhHYWHqgWFiKl SfmRNUhLUdz1KppVxZyOc30bPguH+PD13/DXTsLStkvGbRsVxVUbO3yab 84oxqBMtGijAWtvEFMeZZjDqqPZAjJ2AgfG4oj3Z15FX7yVcJH0ZPgY8q A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879511" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879511" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554626" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554626" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:02 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:51 -0700 Message-Id: <20230327093458.44939-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 03/10] vfio/pci: Move the existing hot reset logic to be a helper X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups(). No functional change is intended. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++------------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 5d745c9abf05..3696b8e58445 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1255,29 +1255,17 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return ret; } -static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); - struct vfio_pci_hot_reset hdr; int32_t *group_fds; struct file **files; struct vfio_pci_group_info info; - bool slot = false; int file_idx, count = 0, ret = 0; - if (copy_from_user(&hdr, arg, minsz)) - return -EFAULT; - - if (hdr.argsz < minsz || hdr.flags) - return -EINVAL; - - /* Can we do a slot or bus reset or neither? */ - if (!pci_probe_reset_slot(vdev->pdev->slot)) - slot = true; - else if (pci_probe_reset_bus(vdev->pdev->bus)) - return -ENODEV; - /* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have @@ -1289,11 +1277,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr.count || hdr.count > count) + if (!hdr->count || hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL); - files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL); + group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); if (!group_fds || !files) { kfree(group_fds); kfree(files); @@ -1301,7 +1289,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } if (copy_from_user(group_fds, arg->group_fds, - hdr.count * sizeof(*group_fds))) { + hdr->count * sizeof(*group_fds))) { kfree(group_fds); kfree(files); return -EFAULT; @@ -1311,7 +1299,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, * Get the group file for each fd to ensure the group held across * the reset */ - for (file_idx = 0; file_idx < hdr.count; file_idx++) { + for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(group_fds[file_idx]); if (!file) { @@ -1335,7 +1323,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = hdr->count; info.files = files; ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); @@ -1348,6 +1336,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); + struct vfio_pci_hot_reset hdr; + bool slot = false; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz || hdr.flags) + return -EINVAL; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); +} + static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, struct vfio_device_ioeventfd __user *arg) { From patchwork Mon Mar 27 09:34:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D37C5C7619A for ; Mon, 27 Mar 2023 09:35:19 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7F32910E556; Mon, 27 Mar 2023 09:35:18 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 32BBF10E363; Mon, 27 Mar 2023 09:35:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909704; x=1711445704; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GInm+3DnElLmSvJJLYAMBhnzd2opXH4SR3j570LhFkU=; b=lbRZ0NNW1mwTU6I+vRNTxRPBKrBij3sPN0JvKNM1Vsldt+d/vFFNWBMj AD8TEv4ABDrxbFDFYyM6HIpdOlGyJp8p3mh97msix8dBJ5tMESCrKj9Hk KSb9Xm/s26J9HwOWRs7zoECjXySmwo1EYIiu7rdEZsnf4wKYQsKSuRimt o0uAY6NJnnnV/NBpYiN8qaF5npAqYPjHk6TVUG0RIRa/4RqWCZ/BlGfqi CkKPGJlZDsUKo7vb8VbpNDMNDcniHXzK7YzKFGPmsyJU10cCL6L0O5dcn KSxtrLo1bvZhDpfP/00DKCgOqjTV40QpPF2aJYkOfiYMbR4Wuh44y3kKW Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879523" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879523" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554634" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554634" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:03 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:52 -0700 Message-Id: <20230327093458.44939-5-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 04/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This is needed by the vfio pci driver to report affected devices in the hot reset for a given device. Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/iommu/iommufd/device.c | 12 ++++++++++++ drivers/vfio/iommufd.c | 16 ++++++++++++++++ include/linux/iommufd.h | 3 +++ include/linux/vfio.h | 13 +++++++++++++ 4 files changed, 44 insertions(+) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 25115d401d8f..04a57aa1ae2c 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -131,6 +131,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) } EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev) +{ + return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD); + +u32 iommufd_device_to_id(struct iommufd_device *idev) +{ + return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD); + static int iommufd_device_setup_msi(struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt, phys_addr_t sw_msi_start) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 88b00c501015..44088049dbb1 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -66,6 +66,22 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd(vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + if (!vdev->iommufd_device) + return NULL; + return iommufd_device_to_ictx(vdev->iommufd_device); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) +{ + if (!vdev->iommufd_device) + return -EINVAL; + *id = iommufd_device_to_id(vdev->iommufd_device); + return 0; +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_devid); /* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 1129a36a74c4..ac96df406833 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev); int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); void iommufd_device_detach(struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); +u32 iommufd_device_to_id(struct iommufd_device *idev); + struct iommufd_access_ops { u8 needs_pin_pages : 1; void (*unmap)(void *data, unsigned long iova, unsigned long length); diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 3188d8a374bd..e54bef5489a0 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -113,6 +113,8 @@ struct vfio_device_ops { }; #if IS_ENABLED(CONFIG_IOMMUFD) +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev); +int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id); int vfio_iommufd_physical_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_physical_unbind(struct vfio_device *vdev); @@ -122,6 +124,17 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev, void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); #else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + return NULL; +} + +static inline int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) +{ + return -EOPNOTSUPP; +} + #define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL) From patchwork Mon Mar 27 09:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2168C76195 for ; Mon, 27 Mar 2023 09:35:20 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B951110E557; Mon, 27 Mar 2023 09:35:18 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 98BA210E388; Mon, 27 Mar 2023 09:35:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909705; x=1711445705; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DspUHqM3GniNFDpFuKlLGJpDgU4kjJg6COLxwo46xbI=; b=ak8BV5ncrlYJMQnM78Vvg57MSPcJQIkmX2PdXXgH8gk721etlI12L/Tt kiU8hRnzzI6aF7tUWXnGrxzxupZqcCdT5CG62ItTaMxc5G/gw3IWO7mUI 0SlD7jblO5lD2codSx2HI4lZruovzHQ447PTKuLqad2f5jmT5zUjUG5IA ZaK+kegrXDu1gUmXrtyijdNC0JcYhNofvUUSeJSy0rGi4suQ3su6KzZ+y rNLPHicc1Ah8qCwbObMD70WGWFPrNqCaPYc/4cZOLjPJU3h5vZm72VDQo OfBfMHhcV9Xtb+JnftxAB7RfHN8ENsGy2Oc/1xRPsJvxDzPJXKgwSAcBv w==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879536" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879536" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554643" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554643" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:04 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:53 -0700 Message-Id: <20230327093458.44939-6-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 05/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds. a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++----- include/uapi/linux/vfio.h | 9 ++++++- 2 files changed, 44 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 3696b8e58445..b68fcba67a4b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) struct vfio_pci_group_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups); + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx); /* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND @@ -1277,7 +1278,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr->count || hdr->count > count) + if (hdr->count > count) return -EINVAL; group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); @@ -1326,7 +1327,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = hdr->count; info.files = files; - ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); + ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL); hot_reset_release: for (file_idx--; file_idx >= 0; file_idx--) @@ -1341,6 +1342,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, { unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); struct vfio_pci_hot_reset hdr; + struct iommufd_ctx *iommufd; bool slot = false; if (copy_from_user(&hdr, arg, minsz)) @@ -1355,7 +1357,12 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, else if (pci_probe_reset_bus(vdev->pdev->bus)) return -ENODEV; - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + if (hdr.count) + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, iommufd); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, @@ -2327,6 +2334,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, { unsigned int i; + if (!groups) + return false; + for (i = 0; i < groups->count; i++) if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) return true; @@ -2402,13 +2412,25 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) return ret; } +static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, + struct iommufd_ctx *iommufd_ctx) +{ + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + if (!iommufd) + return false; + + return iommufd == iommufd_ctx; +} + /* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups) + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; struct vfio_pci_core_device *cur_vma; @@ -2448,9 +2470,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * * Otherwise all opened devices in the dev_set must be * contained by the set of groups provided by the user. + * + * If user provides a zero-length array, then all the + * opened devices must be bound to a same iommufd_ctx. + * + * If all above checks are failed, reset is allowed only if + * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups)) { + !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && + (dev_set->device_count > 1)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index f96e5689cffc..17aa5d09db41 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -679,7 +679,14 @@ struct vfio_pci_hot_reset_info { * the calling user must ensure all affected devices, if opened, are * owned by itself. * - * The ownership is proved by an array of group fds. + * The ownership can be proved by: + * - An array of group fds + * - A zero-length array + * + * In the last case all affected devices which are opened by this user + * must have been bound to a same iommufd. If the calling device is in + * noiommu mode (no valid iommufd) then it can be reset only if the reset + * doesn't affect other devices. * * Return: 0 on success, -errno on failure. */ From patchwork Mon Mar 27 09:34:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A27BC7619A for ; Mon, 27 Mar 2023 09:35:08 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7EF8210E36D; Mon, 27 Mar 2023 09:35:07 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 018D910E390; Mon, 27 Mar 2023 09:35:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909706; x=1711445706; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lPjbsC97jhsaoOPvq0BOI8tQ/LPFyBBuvflvbc0YitU=; b=mFndsoipozb1e7GRr6+i8cA0yJP3uTUeAbZLFWLPX3LiDqPHWb6Z20jh b10cXip7ABcBCQmUjclXsgL+h3P9wNVcmGiK+AYF5TV28kynPjNgAB3nU lo2jMopx5Z6ElGgaElCsmlhwBcoMSsdK6XxrS9mI+83OiDt3wvuSTaYLH 3AkpBm2Sl0coCAQnmu/L2c6Dy74efrOe/ksJX/59ASQ/KyyXtqn6kP8Ld 3VbQdXQ+FtBkyHOtSsFetdYCBwmQxuMtfXo6hrJs6loQzUa3suy5iZq5y HCCCQOCfRkLv4dDn5U5Vj5+9nV5VlIlOGrGOyBlPFFeIE+GOGfEOtMGrJ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879546" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879546" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554649" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554649" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:05 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:54 -0700 Message-Id: <20230327093458.44939-7-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 06/10] vfio: Refine vfio file kAPIs for vfio PCI hot reset X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/group.c | 32 ++++++++++++++------------------ drivers/vfio/pci/vfio_pci_core.c | 4 ++-- drivers/vfio/vfio.h | 2 ++ drivers/vfio/vfio_main.c | 29 +++++++++++++++++++++++++++++ include/linux/vfio.h | 1 + 5 files changed, 48 insertions(+), 20 deletions(-) diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c index 27d5ba7cf9dc..d0c95d033605 100644 --- a/drivers/vfio/group.c +++ b/drivers/vfio/group.c @@ -745,6 +745,15 @@ bool vfio_device_has_container(struct vfio_device *device) return device->group->container; } +struct vfio_group *vfio_group_from_file(struct file *file) +{ + struct vfio_group *group = file->private_data; + + if (file->f_op != &vfio_group_fops) + return NULL; + return group; +} + /** * vfio_file_iommu_group - Return the struct iommu_group for the vfio group file * @file: VFIO group file @@ -755,13 +764,13 @@ bool vfio_device_has_container(struct vfio_device *device) */ struct iommu_group *vfio_file_iommu_group(struct file *file) { - struct vfio_group *group = file->private_data; + struct vfio_group *group = vfio_group_from_file(file); struct iommu_group *iommu_group = NULL; if (!IS_ENABLED(CONFIG_SPAPR_TCE_IOMMU)) return NULL; - if (!vfio_file_is_group(file)) + if (!group) return NULL; mutex_lock(&group->group_lock); @@ -775,12 +784,12 @@ struct iommu_group *vfio_file_iommu_group(struct file *file) EXPORT_SYMBOL_GPL(vfio_file_iommu_group); /** - * vfio_file_is_group - True if the file is usable with VFIO aPIS + * vfio_file_is_group - True if the file is a vfio group file * @file: VFIO group file */ bool vfio_file_is_group(struct file *file) { - return file->f_op == &vfio_group_fops; + return vfio_group_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_group); @@ -842,23 +851,10 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm) } EXPORT_SYMBOL_GPL(vfio_file_set_kvm); -/** - * vfio_file_has_dev - True if the VFIO file is a handle for device - * @file: VFIO file to check - * @device: Device that must be part of the file - * - * Returns true if given file has permission to manipulate the given device. - */ -bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device) { - struct vfio_group *group = file->private_data; - - if (!vfio_file_is_group(file)) - return false; - return group == device->group; } -EXPORT_SYMBOL_GPL(vfio_file_has_dev); static char *vfio_devnode(const struct device *dev, umode_t *mode) { diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b68fcba67a4b..2a510b71edcb 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,8 +1308,8 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, break; } - /* Ensure the FD is a vfio group FD.*/ - if (!vfio_file_is_group(file)) { + /* Ensure the FD is a vfio FD. vfio group or vfio device */ + if (!vfio_file_is_valid(file)) { fput(file); ret = -EINVAL; break; diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h index 7b19c621e0e6..c0aeea24fbd6 100644 --- a/drivers/vfio/vfio.h +++ b/drivers/vfio/vfio.h @@ -84,6 +84,8 @@ void vfio_device_group_unregister(struct vfio_device *device); int vfio_device_group_use_iommu(struct vfio_device *device); void vfio_device_group_unuse_iommu(struct vfio_device *device); void vfio_device_group_close(struct vfio_device *device); +struct vfio_group *vfio_group_from_file(struct file *file); +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device); bool vfio_device_has_container(struct vfio_device *device); int __init vfio_group_init(void); void vfio_group_cleanup(void); diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 89497c933490..fe7446805afd 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,6 +1154,35 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +/** + * vfio_file_is_valid - True if the file is valid vfio file + * @file: VFIO group file or VFIO device file + */ +bool vfio_file_is_valid(struct file *file) +{ + return vfio_group_from_file(file); +} +EXPORT_SYMBOL_GPL(vfio_file_is_valid); + +/** + * vfio_file_has_dev - True if the VFIO file is a handle for device + * @file: VFIO file to check + * @device: Device that must be part of the file + * + * Returns true if given file has permission to manipulate the given device. + */ +bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +{ + struct vfio_group *group; + + group = vfio_group_from_file(file); + if (!group) + return false; + + return vfio_group_has_dev(group, device); +} +EXPORT_SYMBOL_GPL(vfio_file_has_dev); + /* * Sub-module support */ diff --git a/include/linux/vfio.h b/include/linux/vfio.h index e54bef5489a0..79c47733ae0d 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -258,6 +258,7 @@ int vfio_mig_get_next_state(struct vfio_device *device, */ struct iommu_group *vfio_file_iommu_group(struct file *file); bool vfio_file_is_group(struct file *file); +bool vfio_file_is_valid(struct file *file); bool vfio_file_enforced_coherent(struct file *file); void vfio_file_set_kvm(struct file *file, struct kvm *kvm); bool vfio_file_has_dev(struct file *file, struct vfio_device *device); From patchwork Mon Mar 27 09:34:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14DAEC76195 for ; Mon, 27 Mar 2023 09:35:17 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3A36E10E388; Mon, 27 Mar 2023 09:35:16 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id C497E10E398; Mon, 27 Mar 2023 09:35:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909706; x=1711445706; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=prR90xwsDCfDNxJjL9hXgdF5o6pMTiJmwWoTjfPFxLM=; b=Ide1G07qrFIf/qC5ex7HwL+PjPlSGRkq2afL/BI4nPI6BvaNvX2JMWzS 0TzHbEAVgNZYmkmt2Byc9EbACF3nJQdcR2Euq/ZYVVFHuzBClJi30n8eI RM9Hq6VyQQVmf5goT1w+hBZd53xBdEvxBgNd0HVfMv1IxRdmh3PTMQlAw 3c0BfGZlcQ0Nc8BVP7ivMapW4RU7ovUO8Lk8fCXs8MAKd7sOvoWniZ4C4 QouoCa2qCW46cUN2C0zckm00s/IV8k+Eo90fAe7+IgrwFkBDgaPyLvIcX fVboqu7CpFmKV3qqVwkvgUghoFQaFlVqEHNaaP1deWn+wm5Zg5SUANB2H A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879559" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879559" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554656" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554656" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:06 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:55 -0700 Message-Id: <20230327093458.44939-8-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 07/10] vfio: Accpet device file from vfio PCI hot reset path X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/vfio_main.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index fe7446805afd..ebbb6b91a498 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,13 +1154,23 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +static struct vfio_device *vfio_device_from_file(struct file *file) +{ + struct vfio_device *device = file->private_data; + + if (file->f_op != &vfio_device_fops) + return NULL; + return device; +} + /** * vfio_file_is_valid - True if the file is valid vfio file * @file: VFIO group file or VFIO device file */ bool vfio_file_is_valid(struct file *file) { - return vfio_group_from_file(file); + return vfio_group_from_file(file) || + vfio_device_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_valid); @@ -1174,12 +1184,17 @@ EXPORT_SYMBOL_GPL(vfio_file_is_valid); bool vfio_file_has_dev(struct file *file, struct vfio_device *device) { struct vfio_group *group; + struct vfio_device *vdev; group = vfio_group_from_file(file); - if (!group) - return false; + if (group) + return vfio_group_has_dev(group, device); + + vdev = vfio_device_from_file(file); + if (vdev) + return vdev == device; - return vfio_group_has_dev(group, device); + return false; } EXPORT_SYMBOL_GPL(vfio_file_has_dev); From patchwork Mon Mar 27 09:34:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B7DA5C7619A for ; Mon, 27 Mar 2023 09:35:11 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 380C110E3B0; Mon, 27 Mar 2023 09:35:11 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id D3A3910E338; Mon, 27 Mar 2023 09:35:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909708; x=1711445708; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j5nwiM6RBu75u+6frQmtDejnHzhEKYNU9/P5Y8zUpgE=; b=C2IXfQppkF4wmSrtiR83kFeaVuEt7PA8XvtA+TVjHr7OL4+wvs+ZLvh8 z1FY7Icgw+7GOyJq8TPuy+5uIvGHlnEAVx5ARI1YOASfEh7G6m1gNN3kt 5UvYow7UHDRfm65WAz03xxqJxLQBal04Tm5ZR97t2oM33d3GNaS7M1MbB fSlM8/V8QO45nIRaRyiAEHJy3wBiPFcxxh6t8TOVHpB3t6DdjGTqI0G3m /yJM80nwORSqAAbFhHGgxVB0s/KdAeWeEAP3FFdKTlgdzpjy5TSFeW3Aq SWfKcNu5cG0f2iKBxTAA5KdjR3RXKoICxhVbfVle8jfSNJcQafcdK6PW3 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879580" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879580" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554668" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554668" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:07 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:56 -0700 Message-Id: <20230327093458.44939-9-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 08/10] vfio/pci: Renaming for accepting device fd in hot reset path X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" No functional change is intended. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++---------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 2a510b71edcb..da6325008872 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -177,10 +177,10 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) } } -struct vfio_pci_group_info; +struct vfio_pci_file_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx); /* @@ -800,7 +800,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) return 0; } -struct vfio_pci_group_info { +struct vfio_pci_file_info { int count; struct file **files; }; @@ -1257,14 +1257,14 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( } static int -vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset *hdr, - bool slot, - struct vfio_pci_hot_reset __user *arg) +vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - int32_t *group_fds; + int32_t *fds; struct file **files; - struct vfio_pci_group_info info; + struct vfio_pci_file_info info; int file_idx, count = 0, ret = 0; /* @@ -1281,17 +1281,17 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, if (hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + fds = kcalloc(hdr->count, sizeof(*fds), GFP_KERNEL); files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); - if (!group_fds || !files) { - kfree(group_fds); + if (!fds || !files) { + kfree(fds); kfree(files); return -ENOMEM; } - if (copy_from_user(group_fds, arg->group_fds, - hdr->count * sizeof(*group_fds))) { - kfree(group_fds); + if (copy_from_user(fds, arg->group_fds, + hdr->count * sizeof(*fds))) { + kfree(fds); kfree(files); return -EFAULT; } @@ -1301,7 +1301,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, * the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { - struct file *file = fget(group_fds[file_idx]); + struct file *file = fget(fds[file_idx]); if (!file) { ret = -EBADF; @@ -1318,9 +1318,9 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, files[file_idx] = file; } - kfree(group_fds); + kfree(fds); - /* release reference to groups on error */ + /* release reference to fds on error */ if (ret) goto hot_reset_release; @@ -1358,7 +1358,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return -ENODEV; if (hdr.count) - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + return vfio_pci_ioctl_pci_hot_reset_files(vdev, &hdr, slot, arg); iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); @@ -2329,16 +2329,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { }; EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_group_info *groups) +static bool vfio_dev_in_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_file_info *info) { unsigned int i; - if (!groups) + if (!info) return false; - for (i = 0; i < groups->count; i++) - if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) + for (i = 0; i < info->count; i++) + if (vfio_file_has_dev(info->files[i], &vdev->vdev)) return true; return false; } @@ -2429,7 +2429,7 @@ static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; @@ -2478,7 +2478,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_files(cur_vma, info) && !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && (dev_set->device_count > 1)) { ret = -EINVAL; From patchwork Mon Mar 27 09:34:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C338C76195 for ; Mon, 27 Mar 2023 09:35:13 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id AE49810E495; Mon, 27 Mar 2023 09:35:12 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5A75010E3A7; Mon, 27 Mar 2023 09:35:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909709; x=1711445709; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=v0D3R5+aGDWyOemoEsVYiHjHAhjBokxdDCY6e1VuJQQ=; b=m+kwdCNJe2CwWWMmVaNd+oU55DQDv/YxXAVlgHAkI0ywZAkmgnYBO19X AQrUyHvrAZZ8n7y6cBjvDNQ/7g6PE8xc7Bqf2hvuUT878Iu94fMbJD0J6 kjWRW4n6NlfCOnIV6l+sK7mq02x7e1tOFGJotv/Eiwlh3LkQMoPXeYcyn bVOxj6PvsV6sBXcsNlFmw3xgy1VsjNreoJSvA2QQ+2UuFfJ3o2CsgmXYP z+yu+M8YmllXaXL93ESofP405YvEWmJ+FKOk77/iH1dTrd9PlcdFGvOKU oJjiNmkJGGXORa+YItJTMNl3Bil1pXd1czdmXHWHUjQvJB2rqV53e3crt Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879592" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879592" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554680" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554680" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:08 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:57 -0700 Message-Id: <20230327093458.44939-10-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 09/10] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages. Clarify it in uAPI. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 9 +++++---- include/uapi/linux/vfio.h | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index da6325008872..19f5b075d70a 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1289,7 +1289,7 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, return -ENOMEM; } - if (copy_from_user(fds, arg->group_fds, + if (copy_from_user(fds, arg->fds, hdr->count * sizeof(*fds))) { kfree(fds); kfree(files); @@ -1297,8 +1297,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, } /* - * Get the group file for each fd to ensure the group held across - * the reset + * Get the file for each fd to ensure the group/device file + * is held across the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(fds[file_idx]); @@ -2469,7 +2469,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * cannot race being opened by another user simultaneously. * * Otherwise all opened devices in the dev_set must be - * contained by the set of groups provided by the user. + * contained by the set of groups/devices provided by + * the user. * * If user provides a zero-length array, then all the * opened devices must be bound to a same iommufd_ctx. diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 17aa5d09db41..25432ef213ee 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -681,6 +681,7 @@ struct vfio_pci_hot_reset_info { * * The ownership can be proved by: * - An array of group fds + * - An array of device fds * - A zero-length array * * In the last case all affected devices which are opened by this user @@ -694,7 +695,7 @@ struct vfio_pci_hot_reset { __u32 argsz; __u32 flags; __u32 count; - __s32 group_fds[]; + __s32 fds[]; }; #define VFIO_DEVICE_PCI_HOT_RESET _IO(VFIO_TYPE, VFIO_BASE + 13) From patchwork Mon Mar 27 09:34:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0EA9C77B62 for ; Mon, 27 Mar 2023 09:35:18 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 05D9D10E3C8; Mon, 27 Mar 2023 09:35:18 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6573110E3C0; Mon, 27 Mar 2023 09:35:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909711; x=1711445711; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8gWbAJJlIuV/z8+WOO6T7u0CyD5t7kZGlKBcfpHfaUQ=; b=hbKHSdalqRF3snvsFYI1XnEHZLnlqKRIPvXDEU462SJ3ebrJ8LyZp9HC Be30glrdAzEbX2+YEmgopu5Sfrscul5UhiKBJjaD3b8jK4Tu9uYPrwmqm QgIIuH/a7bx1xVgNNx/pL/DXb1iWR70uQe+0R3/CoERaZec53OW0cFOi0 sT2UCIeFrsFswvT8ssB6LSgxsuRVLNGzK2rUzMPNEqMhtV6O54ePCLFev xY3OOoHJ/5em+9L6V8+9Tq7ZShDua+wXCoyQBY00TDAKyIeoGpegMvHWd ZaOCP03uY8PBG4HI18JVmD4OyV+VMvATZBviCVpWXwp3dZrRMiT97GAjj w==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879608" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879608" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554687" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554687" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:10 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Mon, 27 Mar 2023 02:34:58 -0700 Message-Id: <20230327093458.44939-11-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required. This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report. This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. Signed-off-by: Yi Liu Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 47 +++++++++++++++ 2 files changed, 145 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 19f5b075d70a..45edf4e9b98b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info( + struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset_group_info __user *arg) +{ + unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); + struct vfio_pci_hot_reset_group_info hdr; + struct iommufd_ctx *iommufd, *cur_iommufd; + u32 count = 0, index = 0, *devices = NULL; + struct vfio_pci_core_device *cur; + bool slot = false; + int ret = 0; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz) + return -EINVAL; + + hdr.flags = 0; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + mutex_lock(&vdev->vdev.dev_set->lock); + if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) { + ret = -EPERM; + goto out_unlock; + } + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + if (!iommufd) { + ret = -EPERM; + goto out_unlock; + } + + /* How many devices are affected? */ + ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_count_devs, + &count, slot); + if (ret) + goto out_unlock; + + WARN_ON(!count); /* Should always be at least one */ + + /* + * If there's enough space, fill it now, otherwise return -ENOSPC and + * the number of devices affected. + */ + if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) { + ret = -ENOSPC; + hdr.count = count; + goto reset_info_exit; + } + + devices = kcalloc(count, sizeof(*devices), GFP_KERNEL); + if (!devices) { + ret = -ENOMEM; + goto reset_info_exit; + } + + list_for_each_entry(cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) { + cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev); + if (cur->vdev.open_count) { + if (cur_iommufd != iommufd) { + ret = -EPERM; + break; + } + ret = vfio_iommufd_physical_devid(&cur->vdev, &devices[index]); + if (ret) + break; + index++; + } + } + +reset_info_exit: + if (copy_to_user(arg, &hdr, minsz)) + ret = -EFAULT; + + if (!ret) { + if (copy_to_user(&arg->devices, devices, + hdr.count * sizeof(*devices))) + ret = -EFAULT; + } + + kfree(devices); +out_unlock: + mutex_unlock(&vdev->vdev.dev_set->lock); + return ret; +} + static int vfio_pci_ioctl_get_pci_hot_reset_info( struct vfio_pci_core_device *vdev, struct vfio_pci_hot_reset_info __user *arg) @@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, return vfio_pci_ioctl_get_irq_info(vdev, uarg); case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO: return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg); + case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO: + return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev, uarg); case VFIO_DEVICE_GET_REGION_INFO: return vfio_pci_ioctl_get_region_info(vdev, uarg); case VFIO_DEVICE_IOEVENTFD: diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 25432ef213ee..61b801dfd40b 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info { #define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12) +/** + * VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, + * struct vfio_pci_hot_reset_group_info) + * + * This is used in the vfio device cdev mode. It returns the list of + * affected devices (represented by iommufd dev_id) when hot reset is + * issued on the current device with which this ioctl is invoked. It + * only includes the devices that are opened by the current user in the + * time of this command is invoked. This list may change when user opens + * new device or close opened device, hence user should re-invoke it + * after open/close devices. This command has no guarantee on the result + * of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can + * be by other users in the window between the two ioctls. If the affected + * devices are opened by multiple users, the VFIO_DEVICE_PCI_HOT_RESET + * shall fail, detail can check the description of VFIO_DEVICE_PCI_HOT_RESET. + * + * For the users that open vfio group/container, this ioctl is useless as + * they have no idea about the iommufd dev_id returned by this ioctl. For + * the users of the traditional mode vfio group/container, this ioctl will + * fail as this mode does not use iommufd hence no dev_id to report back. + * For the users of the iommufd compat mode vfio group/container, this ioctl + * would succeed as this mode uses iommufd as container fd. But such users + * still have no idea about the iommufd dev_id as the dev_id is only stored + * in kernel in this mode. For the users of the vfio group/container, the + * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset + * affected devices. + * + * Return: 0 on success, -errno on failure: + * -enospc = insufficient buffer; + * -enodev = unsupported for device; + * -eperm = no permission for device, this error comes: + * - when there are affected devices that are opened but + * not bound to the same iommufd with the current device + * with which this ioctl is invoked, + * - there are affected devices that are not bound to vfio + * driver yet. + * - no valid iommufd is bound (e.g. noiommu mode) + */ +struct vfio_pci_hot_reset_group_info { + __u32 argsz; + __u32 flags; + __u32 count; + __u32 devices[]; +}; + +#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO _IO(VFIO_TYPE, VFIO_BASE + 18) + /** * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset)