From patchwork Sat Apr 1 14:44:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D535AC6FD1D for ; Sat, 1 Apr 2023 14:44:43 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A977710E227; Sat, 1 Apr 2023 14:44:37 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 37D7710E0F6; Sat, 1 Apr 2023 14:44:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360272; x=1711896272; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IPobf8cBYDdoCsj9nkBcWaL+xVq1fXMUUNMJHvpfpYY=; b=MjW9aR8aHfndUMcNqpP4CyrKHO7uhfnRI3YzoG2KYxTHTNcdnJ5E3nCO bEGDSJxlu4ozowvT5vi9/7y76M+lsSAk9AibQ9kz1Mf0jlWt/c4GMezNf v8wP69wcFIVegBVsTUbs9stOEEst5basPML7tzmmX3Pjv1qyJFTRRzclL FSUzfrGLCw0v1o0M6RLM0Yfg5dnTaGguS6/v8Q2NCZPBDCb9Pn99beS2M RDwAqk7pHY1sSkzquSlABpYPHD34LMBFc75yNWjvEkVtKuiG4vbF0knel iBl8NGSB2jRq4Yr30lMOx6AxLLsHBn3AIHyQT9Xf7TR7IXHS9rULa1Oei w==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385078" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385078" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705808" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705808" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:31 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:18 -0700 Message-Id: <20230401144429.88673-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 01/12] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" this suits more on what the code does. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Eric Auger --- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a5ab416cf476..65bbef562268 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } /* - * For each group_fd, get the group through the vfio external user - * interface and store the group and iommu ID. This ensures the group - * is held across the reset. + * Get the group file for each fd to ensure the group held across + * the reset */ for (file_idx = 0; file_idx < hdr.count; file_idx++) { struct file *file = fget(group_fds[file_idx]); From patchwork Sat Apr 1 14:44:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E28C3C6FD1D for ; Sat, 1 Apr 2023 14:44:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 75D9F10E113; Sat, 1 Apr 2023 14:44:36 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 415EA10E105; Sat, 1 Apr 2023 14:44:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360273; x=1711896273; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7k2A4DGuW0zc8hE6tthl+NEC6pCTmIh+4U8pGHD+swY=; b=BcfRgSNq7q/B9/DKCRmtxVVCHIX7ncJ2YpsPyQsCAR3WsfiKHk/Y4eRZ 4W05+KAfa/fIya1U9Kn46jpclXu8nORSknZSgMZLFu57B/KKmToZfLOLF fCOO0MrM6urGpx51VPZVM0MG+vWwS9iRZNdadqlojHN7onAN1AGf0/y3S N9erGY70Ix2Nd/s/PTZyprUdc00XpAAM8bIBKcMhVdMZwdmjm6WE4OSCh OONCJ9vavY70aSlCj8Mq/1N+nzQkJ69vdTb0ij7RVbWla598mqxvKNFRF mgDYsVpOEmn+u/rckBe4snqzP1g7QC+OyhGx8p/f6ipz5Ve/KsJETke5s w==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385089" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385089" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705813" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705813" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:32 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:19 -0700 Message-Id: <20230401144429.88673-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 02/12] vfio/pci: Only check ownership of opened devices in hot reset X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" If the affected device is not opened by any user, it's safe to reset it given it's not in use. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++++--- include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 65bbef562268..5d745c9abf05 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2429,10 +2429,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) { /* - * Test whether all the affected devices are contained by the - * set of groups provided by the user. + * Test whether all the affected devices can be reset by the + * user. + * + * Resetting an unused device (not opened) is safe, because + * dev_set->lock is held in hot reset path so this device + * cannot race being opened by another user simultaneously. + * + * Otherwise all opened devices in the dev_set must be + * contained by the set of groups provided by the user. */ - if (!vfio_dev_in_groups(cur_vma, groups)) { + if (cur_vma->vdev.open_count && + !vfio_dev_in_groups(cur_vma, groups)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0552e8dcf0cb..f96e5689cffc 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -673,6 +673,14 @@ struct vfio_pci_hot_reset_info { * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * + * Userspace requests hot reset for the devices it uses. Due to the + * underlying topology, multiple devices can be affected in the reset + * while some might be opened by another user. To avoid interference + * the calling user must ensure all affected devices, if opened, are + * owned by itself. + * + * The ownership is proved by an array of group fds. + * * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset { From patchwork Sat Apr 1 14:44:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9C22C77B62 for ; Sat, 1 Apr 2023 14:44:45 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1C2C610E24B; Sat, 1 Apr 2023 14:44:39 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id D56EC10E05B; Sat, 1 Apr 2023 14:44:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360273; x=1711896273; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Dgj5ZUV5FzyshZBmCtEuPGjWqH1I8a/k7CA8N5o/j5U=; b=A+5cd5eDt4b0fHbwT4TwW5+/jqwzdpx2U+nVR9bePJJSQzFFfYCysvAV YNgved0CnmWZiOMU4b2UaeuqXQEVkyzrVsdfPrwQ2+JfNdpvH7OhY4HKk i/R8TbmcmCb0AZEb5nVF9za2JZ4t8WluIDQOXCraODJDkfdzK5O6NiIxX TJFrioc92hkkw72Qn3P9GfsmeYzfpPLppZVj6WpCOs/7Q8GD+Fp7s97tv GdIMvCCvpbXC9WdE/zy6LuWxwqpldNkErH/uh1oLKKOVaS3M7jl8Cv6jr i37CUhFP2paY4EMw2juI7bwNqszJMD+qJWmfIWlJnne/Kqv/n2fjyM8JX A==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385098" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385098" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705821" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705821" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:33 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:20 -0700 Message-Id: <20230401144429.88673-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 03/12] vfio/pci: Move the existing hot reset logic to be a helper X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups(). No functional change is intended. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Eric Auger --- drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++------------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 5d745c9abf05..3696b8e58445 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1255,29 +1255,17 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return ret; } -static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); - struct vfio_pci_hot_reset hdr; int32_t *group_fds; struct file **files; struct vfio_pci_group_info info; - bool slot = false; int file_idx, count = 0, ret = 0; - if (copy_from_user(&hdr, arg, minsz)) - return -EFAULT; - - if (hdr.argsz < minsz || hdr.flags) - return -EINVAL; - - /* Can we do a slot or bus reset or neither? */ - if (!pci_probe_reset_slot(vdev->pdev->slot)) - slot = true; - else if (pci_probe_reset_bus(vdev->pdev->bus)) - return -ENODEV; - /* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have @@ -1289,11 +1277,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr.count || hdr.count > count) + if (!hdr->count || hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL); - files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL); + group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); if (!group_fds || !files) { kfree(group_fds); kfree(files); @@ -1301,7 +1289,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } if (copy_from_user(group_fds, arg->group_fds, - hdr.count * sizeof(*group_fds))) { + hdr->count * sizeof(*group_fds))) { kfree(group_fds); kfree(files); return -EFAULT; @@ -1311,7 +1299,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, * Get the group file for each fd to ensure the group held across * the reset */ - for (file_idx = 0; file_idx < hdr.count; file_idx++) { + for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(group_fds[file_idx]); if (!file) { @@ -1335,7 +1323,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = hdr->count; info.files = files; ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); @@ -1348,6 +1336,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); + struct vfio_pci_hot_reset hdr; + bool slot = false; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz || hdr.flags) + return -EINVAL; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); +} + static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, struct vfio_device_ioeventfd __user *arg) { From patchwork Sat Apr 1 14:44:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E54BC77B6D for ; Sat, 1 Apr 2023 14:44:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C760F10E23B; Sat, 1 Apr 2023 14:44:41 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id C2BC210E0F6; Sat, 1 Apr 2023 14:44:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360275; x=1711896275; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=z99ZnPiTmiMZImt4QS5DbgD7eZm064znhj0YZmEXafY=; b=NgiJmcJnDT4HeWv6o4l4XPJ/9Ml0oME94Ov9peuThCD3+7ur0GJ8T4kw gk62yd76sKgdgzFVsszYEOJgPDucXb7hlqvDcaCo/enO8nXQzKx6yPOFy sscnDiA7ke2xHI9OKsUF8SeKKHnhda7kyA/vq15AMGFvcQqKan+nXGkJr 6W7rwFEqjzTPbHWcqw4DmYj8UDf2OSV3jSzlFtmi081MQVgp6jma/frV6 N2DVGy6L4TTWUXBvssc8jpJU2tSNqEB1bJwFnxfiyS+l9N55iz/preSBs /NuJ64nlX5U7zW/4QfsqZlkMhBq3S/RO7245kv7IaQU9Z4qH3PSy2sQOU Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385107" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385107" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705827" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705827" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:33 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:21 -0700 Message-Id: <20230401144429.88673-5-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 04/12] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This is needed by the vfio-pci driver to report affected devices in the hot reset for a given device. Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Eric Auger --- drivers/iommu/iommufd/device.c | 12 ++++++++++++ drivers/vfio/iommufd.c | 14 ++++++++++++++ include/linux/iommufd.h | 3 +++ include/linux/vfio.h | 13 +++++++++++++ 4 files changed, 42 insertions(+) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 25115d401d8f..04a57aa1ae2c 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -131,6 +131,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) } EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev) +{ + return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD); + +u32 iommufd_device_to_id(struct iommufd_device *idev) +{ + return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD); + static int iommufd_device_setup_msi(struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt, phys_addr_t sw_msi_start) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 88b00c501015..809f2dd73b9e 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -66,6 +66,20 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd(vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + if (!vdev->iommufd_device) + return NULL; + return iommufd_device_to_ictx(vdev->iommufd_device); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx); + +void vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) +{ + if (vdev->iommufd_device) + *id = iommufd_device_to_id(vdev->iommufd_device); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_devid); /* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 1129a36a74c4..ac96df406833 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev); int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); void iommufd_device_detach(struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); +u32 iommufd_device_to_id(struct iommufd_device *idev); + struct iommufd_access_ops { u8 needs_pin_pages : 1; void (*unmap)(void *data, unsigned long iova, unsigned long length); diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 3188d8a374bd..97a1174b922f 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -113,6 +113,8 @@ struct vfio_device_ops { }; #if IS_ENABLED(CONFIG_IOMMUFD) +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev); +void vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id); int vfio_iommufd_physical_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_physical_unbind(struct vfio_device *vdev); @@ -122,6 +124,17 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev, void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); #else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + return NULL; +} + +static inline void +vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) +{ +} + #define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL) From patchwork Sat Apr 1 14:44:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197018 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49BFDC77B71 for ; Sat, 1 Apr 2023 14:44:53 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1B37D10E2D1; Sat, 1 Apr 2023 14:44:50 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9489F10E05B; Sat, 1 Apr 2023 14:44:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360275; x=1711896275; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BWa6T6hOjcr7bNpakDPjbbVXSouryNJ0Y/EIjnK17EQ=; b=SqhrCg64n7NY51/jFSqbGzD218fFdDgcQWI8gIG4qFxbcWxrOT0qFq3J zzdAlKejfxo8/ERs6+6iDAJ479CWHvejekFw0bYB7AjW/2hDJBqJAPfA8 KSr4bBKM1viS5Uw+VlK0cTlf18eXVyo4LtBNGmSdF8m/SooGzqMZs1VQr vIld9Mt4JGIfIkvo84qPzeNSo7lt9aA/s07Ma9lvfthKIpk+p8YpsChuk zy0n7ho48kUrdRI1otTwmHksrDcas632CiIHjvaBmoZT8w6i7ukQ3Bx0B ibDRXtv2rnC5c2zdwr7e/7EZKGA+zB0VQPCQRPL6GxS9sIWBHJo5HIboI g==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385116" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385116" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705833" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705833" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:34 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:22 -0700 Message-Id: <20230401144429.88673-6-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 05/12] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds. a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++----- include/uapi/linux/vfio.h | 9 ++++++- 2 files changed, 44 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 3696b8e58445..b68fcba67a4b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) struct vfio_pci_group_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups); + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx); /* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND @@ -1277,7 +1278,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr->count || hdr->count > count) + if (hdr->count > count) return -EINVAL; group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); @@ -1326,7 +1327,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = hdr->count; info.files = files; - ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); + ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL); hot_reset_release: for (file_idx--; file_idx >= 0; file_idx--) @@ -1341,6 +1342,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, { unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); struct vfio_pci_hot_reset hdr; + struct iommufd_ctx *iommufd; bool slot = false; if (copy_from_user(&hdr, arg, minsz)) @@ -1355,7 +1357,12 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, else if (pci_probe_reset_bus(vdev->pdev->bus)) return -ENODEV; - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + if (hdr.count) + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, iommufd); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, @@ -2327,6 +2334,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, { unsigned int i; + if (!groups) + return false; + for (i = 0; i < groups->count; i++) if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) return true; @@ -2402,13 +2412,25 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) return ret; } +static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, + struct iommufd_ctx *iommufd_ctx) +{ + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + if (!iommufd) + return false; + + return iommufd == iommufd_ctx; +} + /* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups) + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; struct vfio_pci_core_device *cur_vma; @@ -2448,9 +2470,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * * Otherwise all opened devices in the dev_set must be * contained by the set of groups provided by the user. + * + * If user provides a zero-length array, then all the + * opened devices must be bound to a same iommufd_ctx. + * + * If all above checks are failed, reset is allowed only if + * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups)) { + !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && + (dev_set->device_count > 1)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index f96e5689cffc..17aa5d09db41 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -679,7 +679,14 @@ struct vfio_pci_hot_reset_info { * the calling user must ensure all affected devices, if opened, are * owned by itself. * - * The ownership is proved by an array of group fds. + * The ownership can be proved by: + * - An array of group fds + * - A zero-length array + * + * In the last case all affected devices which are opened by this user + * must have been bound to a same iommufd. If the calling device is in + * noiommu mode (no valid iommufd) then it can be reset only if the reset + * doesn't affect other devices. * * Return: 0 on success, -errno on failure. */ From patchwork Sat Apr 1 14:44:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BDE11C6FD1D for ; Sat, 1 Apr 2023 14:44:48 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A84D110E24F; Sat, 1 Apr 2023 14:44:45 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 46FE810E05B; Sat, 1 Apr 2023 14:44:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360276; x=1711896276; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YnqwCmUnJONgJFdKsiHKUFDBA391yTcmtY6vXjL8rkQ=; b=do/TFGCButuEbNH1P+6buzg+ezIftyLw7y1EH+Aqn/FdR9KcLFSB/eas /qHr8cOJiQg+TPuZ8CCeE/0KL6lE8kCwPtDytCw5zXJQ0KZeKMT5A41IA EcDPrd9JpOpH19rfh+Yf3TwPocJ2XoGlgMpqTVcpSEU2HzNI2w9FaMchX zRatX43KhJmB00aobolPzzZuL/tKp9WvShhMkS/kXRvFKmLJ0//8a8poJ ST6wq0gQ7lL2fgq5aBL/pbYwSqSur4LftkUOhZJ4pMV3/bzeZxDU8x/O9 jY4RGimPtt+Zt+jTEXo9/B92J+h1E9ypnsbLRGyR2vtQXOltkI7v4w105 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385126" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385126" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705837" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705837" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:35 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:23 -0700 Message-Id: <20230401144429.88673-7-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 06/12] vfio: Refine vfio file kAPIs for vfio PCI hot reset X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/group.c | 32 ++++++++++++++------------------ drivers/vfio/pci/vfio_pci_core.c | 4 ++-- drivers/vfio/vfio.h | 2 ++ drivers/vfio/vfio_main.c | 29 +++++++++++++++++++++++++++++ include/linux/vfio.h | 1 + 5 files changed, 48 insertions(+), 20 deletions(-) diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c index 27d5ba7cf9dc..d0c95d033605 100644 --- a/drivers/vfio/group.c +++ b/drivers/vfio/group.c @@ -745,6 +745,15 @@ bool vfio_device_has_container(struct vfio_device *device) return device->group->container; } +struct vfio_group *vfio_group_from_file(struct file *file) +{ + struct vfio_group *group = file->private_data; + + if (file->f_op != &vfio_group_fops) + return NULL; + return group; +} + /** * vfio_file_iommu_group - Return the struct iommu_group for the vfio group file * @file: VFIO group file @@ -755,13 +764,13 @@ bool vfio_device_has_container(struct vfio_device *device) */ struct iommu_group *vfio_file_iommu_group(struct file *file) { - struct vfio_group *group = file->private_data; + struct vfio_group *group = vfio_group_from_file(file); struct iommu_group *iommu_group = NULL; if (!IS_ENABLED(CONFIG_SPAPR_TCE_IOMMU)) return NULL; - if (!vfio_file_is_group(file)) + if (!group) return NULL; mutex_lock(&group->group_lock); @@ -775,12 +784,12 @@ struct iommu_group *vfio_file_iommu_group(struct file *file) EXPORT_SYMBOL_GPL(vfio_file_iommu_group); /** - * vfio_file_is_group - True if the file is usable with VFIO aPIS + * vfio_file_is_group - True if the file is a vfio group file * @file: VFIO group file */ bool vfio_file_is_group(struct file *file) { - return file->f_op == &vfio_group_fops; + return vfio_group_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_group); @@ -842,23 +851,10 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm) } EXPORT_SYMBOL_GPL(vfio_file_set_kvm); -/** - * vfio_file_has_dev - True if the VFIO file is a handle for device - * @file: VFIO file to check - * @device: Device that must be part of the file - * - * Returns true if given file has permission to manipulate the given device. - */ -bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device) { - struct vfio_group *group = file->private_data; - - if (!vfio_file_is_group(file)) - return false; - return group == device->group; } -EXPORT_SYMBOL_GPL(vfio_file_has_dev); static char *vfio_devnode(const struct device *dev, umode_t *mode) { diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b68fcba67a4b..2a510b71edcb 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,8 +1308,8 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, break; } - /* Ensure the FD is a vfio group FD.*/ - if (!vfio_file_is_group(file)) { + /* Ensure the FD is a vfio FD. vfio group or vfio device */ + if (!vfio_file_is_valid(file)) { fput(file); ret = -EINVAL; break; diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h index 7b19c621e0e6..c0aeea24fbd6 100644 --- a/drivers/vfio/vfio.h +++ b/drivers/vfio/vfio.h @@ -84,6 +84,8 @@ void vfio_device_group_unregister(struct vfio_device *device); int vfio_device_group_use_iommu(struct vfio_device *device); void vfio_device_group_unuse_iommu(struct vfio_device *device); void vfio_device_group_close(struct vfio_device *device); +struct vfio_group *vfio_group_from_file(struct file *file); +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device); bool vfio_device_has_container(struct vfio_device *device); int __init vfio_group_init(void); void vfio_group_cleanup(void); diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 89497c933490..fe7446805afd 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,6 +1154,35 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +/** + * vfio_file_is_valid - True if the file is valid vfio file + * @file: VFIO group file or VFIO device file + */ +bool vfio_file_is_valid(struct file *file) +{ + return vfio_group_from_file(file); +} +EXPORT_SYMBOL_GPL(vfio_file_is_valid); + +/** + * vfio_file_has_dev - True if the VFIO file is a handle for device + * @file: VFIO file to check + * @device: Device that must be part of the file + * + * Returns true if given file has permission to manipulate the given device. + */ +bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +{ + struct vfio_group *group; + + group = vfio_group_from_file(file); + if (!group) + return false; + + return vfio_group_has_dev(group, device); +} +EXPORT_SYMBOL_GPL(vfio_file_has_dev); + /* * Sub-module support */ diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 97a1174b922f..f8fb9ab25188 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -258,6 +258,7 @@ int vfio_mig_get_next_state(struct vfio_device *device, */ struct iommu_group *vfio_file_iommu_group(struct file *file); bool vfio_file_is_group(struct file *file); +bool vfio_file_is_valid(struct file *file); bool vfio_file_enforced_coherent(struct file *file); void vfio_file_set_kvm(struct file *file, struct kvm *kvm); bool vfio_file_has_dev(struct file *file, struct vfio_device *device); From patchwork Sat Apr 1 14:44:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15C68C76196 for ; Sat, 1 Apr 2023 14:44:50 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 782B110E1BE; Sat, 1 Apr 2023 14:44:45 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6AEBB10E224; Sat, 1 Apr 2023 14:44:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360277; x=1711896277; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hl2dkQNmXDpUh7X8DM3A4uL+zEobUvRXjuLA5s2BKLk=; b=COfd50x+LqZGnlZjsmaA7CO2kE3icabD4NoN0W9SHhQRAFSax5Wmax42 b8zYUTbKcMO5AorxI8YqqPzwWpSP5VdatvEdEW7QA7PKYz2BNs33WT9zR nVaIRwmqds9Tm2mOXTwSfBIsqJnwPkSyD/pFBcra/Xf1gmZvELaxfk+5M AfuJMXgCDAWl7O5ZHgFd4RP+cWW5DpIQw8yA7h24PiOg7ta6kBLY7PE3E EwvX5WYrXKKsQkrrY0tlIckBf6SOvOZclRMXJjG+Z2wVqpvI0xSLv2j68 ZVL80PxDSYGmJqSZ6G7ToLYfpJD1K2t1CphvbUQIzsQg0ztrn2KlJj0qx Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385137" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385137" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705841" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705841" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:36 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:24 -0700 Message-Id: <20230401144429.88673-8-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 07/12] vfio: Accpet device file from vfio PCI hot reset path X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Eric Auger --- drivers/vfio/vfio_main.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index fe7446805afd..ebbb6b91a498 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,13 +1154,23 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +static struct vfio_device *vfio_device_from_file(struct file *file) +{ + struct vfio_device *device = file->private_data; + + if (file->f_op != &vfio_device_fops) + return NULL; + return device; +} + /** * vfio_file_is_valid - True if the file is valid vfio file * @file: VFIO group file or VFIO device file */ bool vfio_file_is_valid(struct file *file) { - return vfio_group_from_file(file); + return vfio_group_from_file(file) || + vfio_device_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_valid); @@ -1174,12 +1184,17 @@ EXPORT_SYMBOL_GPL(vfio_file_is_valid); bool vfio_file_has_dev(struct file *file, struct vfio_device *device) { struct vfio_group *group; + struct vfio_device *vdev; group = vfio_group_from_file(file); - if (!group) - return false; + if (group) + return vfio_group_has_dev(group, device); + + vdev = vfio_device_from_file(file); + if (vdev) + return vdev == device; - return vfio_group_has_dev(group, device); + return false; } EXPORT_SYMBOL_GPL(vfio_file_has_dev); From patchwork Sat Apr 1 14:44:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197020 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55245C77B73 for ; Sat, 1 Apr 2023 14:44:55 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8E06F10E105; Sat, 1 Apr 2023 14:44:54 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 528D510E05B; Sat, 1 Apr 2023 14:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360278; x=1711896278; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1njmBfLyEIbN6KIP5rpN8/ygwQRUQXPg4sAwefM7UhA=; b=GHpkaQlcX/EHyOS/C1QY2DI22MgfBNfLxPZe5dYhmmBo4qWDyDKwSKzl FyjyxvddldnKRxF9Lc70vK6BiIQelOHL5ngtxOqS5NrsJ3w1iH8b5PO+f AqogmEmcJZbzEDv50o4he+Jb9P/PuMr7G4SrzkhwWzPvxVxHDL7DIMIz5 dQSeHI+RFbhBqs5oZ8Y0CMpDdUufe/OoE2fkAFZLPEibpD9UL4X9fv5iq 2zEfM7AjYboow6ucPPjM2p3hGCx9tcAZxzDGWdl0XrirPdYi0AneasSmj Rk7J8EedWEwN9SWZqe4A9cH5shKlLBlwVRJH1nPY8YBge3AMU/Nn3IDUu A==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385150" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385150" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705844" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705844" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:37 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:25 -0700 Message-Id: <20230401144429.88673-9-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 08/12] vfio/pci: Renaming for accepting device fd in hot reset path X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" No functional change is intended. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Eric Auger --- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++---------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 2a510b71edcb..da6325008872 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -177,10 +177,10 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) } } -struct vfio_pci_group_info; +struct vfio_pci_file_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx); /* @@ -800,7 +800,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) return 0; } -struct vfio_pci_group_info { +struct vfio_pci_file_info { int count; struct file **files; }; @@ -1257,14 +1257,14 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( } static int -vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset *hdr, - bool slot, - struct vfio_pci_hot_reset __user *arg) +vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - int32_t *group_fds; + int32_t *fds; struct file **files; - struct vfio_pci_group_info info; + struct vfio_pci_file_info info; int file_idx, count = 0, ret = 0; /* @@ -1281,17 +1281,17 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, if (hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + fds = kcalloc(hdr->count, sizeof(*fds), GFP_KERNEL); files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); - if (!group_fds || !files) { - kfree(group_fds); + if (!fds || !files) { + kfree(fds); kfree(files); return -ENOMEM; } - if (copy_from_user(group_fds, arg->group_fds, - hdr->count * sizeof(*group_fds))) { - kfree(group_fds); + if (copy_from_user(fds, arg->group_fds, + hdr->count * sizeof(*fds))) { + kfree(fds); kfree(files); return -EFAULT; } @@ -1301,7 +1301,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, * the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { - struct file *file = fget(group_fds[file_idx]); + struct file *file = fget(fds[file_idx]); if (!file) { ret = -EBADF; @@ -1318,9 +1318,9 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, files[file_idx] = file; } - kfree(group_fds); + kfree(fds); - /* release reference to groups on error */ + /* release reference to fds on error */ if (ret) goto hot_reset_release; @@ -1358,7 +1358,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return -ENODEV; if (hdr.count) - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + return vfio_pci_ioctl_pci_hot_reset_files(vdev, &hdr, slot, arg); iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); @@ -2329,16 +2329,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { }; EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_group_info *groups) +static bool vfio_dev_in_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_file_info *info) { unsigned int i; - if (!groups) + if (!info) return false; - for (i = 0; i < groups->count; i++) - if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) + for (i = 0; i < info->count; i++) + if (vfio_file_has_dev(info->files[i], &vdev->vdev)) return true; return false; } @@ -2429,7 +2429,7 @@ static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; @@ -2478,7 +2478,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_files(cur_vma, info) && !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && (dev_set->device_count > 1)) { ret = -EINVAL; From patchwork Sat Apr 1 14:44:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DBA96C77B62 for ; Sat, 1 Apr 2023 14:44:56 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1997D10E0F6; Sat, 1 Apr 2023 14:44:56 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id DEC0510E23B; Sat, 1 Apr 2023 14:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360278; x=1711896278; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=o+4hvtO01Hxfljc0ef73oUe7c+gXDNaaBG7dxXvC5xY=; b=D4ZC829FWPr6+n6xqee8C76wvTP+2G6TINiXaJLdcK/IH99yiiTXs1NE bSSI0njp9DY8eTJU/vmr8zQXKJpqRAizow4wrbpKHfrnoF7awMwrXyJDK Lk6IVs2HUHQMLgXs9KcHaGEh4swgg9IoaRy5CbNcDsAtxR8DfVaAy69tw 3uN+YjWCX75F62rd/oLpLa3Fp3t+TJ2LSrJWo25P0ZoukcpSZxRPcHOAL GYJu/UnupesFzk9O0lOYKu+SQ1xWRJIVKAWMy2IiQj8LiJCD8wnb8qfGG +wr3rktFmPU8mE2TaI3yJF28sa9L+3+wYPAFATAzpMeL3/Iz9oFHpKArG Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385160" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385160" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705847" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705847" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:38 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:26 -0700 Message-Id: <20230401144429.88673-10-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 09/12] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages. Clarify it in uAPI. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Eric Auger --- drivers/vfio/pci/vfio_pci_core.c | 9 +++++---- include/uapi/linux/vfio.h | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index da6325008872..19f5b075d70a 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1289,7 +1289,7 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, return -ENOMEM; } - if (copy_from_user(fds, arg->group_fds, + if (copy_from_user(fds, arg->fds, hdr->count * sizeof(*fds))) { kfree(fds); kfree(files); @@ -1297,8 +1297,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, } /* - * Get the group file for each fd to ensure the group held across - * the reset + * Get the file for each fd to ensure the group/device file + * is held across the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(fds[file_idx]); @@ -2469,7 +2469,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * cannot race being opened by another user simultaneously. * * Otherwise all opened devices in the dev_set must be - * contained by the set of groups provided by the user. + * contained by the set of groups/devices provided by + * the user. * * If user provides a zero-length array, then all the * opened devices must be bound to a same iommufd_ctx. diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 17aa5d09db41..25432ef213ee 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -681,6 +681,7 @@ struct vfio_pci_hot_reset_info { * * The ownership can be proved by: * - An array of group fds + * - An array of device fds * - A zero-length array * * In the last case all affected devices which are opened by this user @@ -694,7 +695,7 @@ struct vfio_pci_hot_reset { __u32 argsz; __u32 flags; __u32 count; - __s32 group_fds[]; + __s32 fds[]; }; #define VFIO_DEVICE_PCI_HOT_RESET _IO(VFIO_TYPE, VFIO_BASE + 13) From patchwork Sat Apr 1 14:44:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5AF76C6FD1D for ; Sat, 1 Apr 2023 14:44:51 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4207710E26C; Sat, 1 Apr 2023 14:44:46 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id A637A10E24F; Sat, 1 Apr 2023 14:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360279; x=1711896279; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=r+71o8G6Wo9sr6JKZnAb0/po7K9Eq9HOWBlcp5zMc44=; b=ZBFu/Ap+M6dMT9a0dGOSGmmVfbIY7x8Fl16M3gZYnrP9g3Ko1j7SLOw6 6aDDnoU0BMOtGrLY0bTiChlG8IjT8VAZQCJscZzZdGoEsxp4b7geNsPNA FUIOTc0dtUUbkoRkCHHmRBUKJhDB4q+kx6ehTg6P5v5CpKK95/dV7yyLA 2Igjo8X9SHq4/hN/JoiMy2icEWIO5oEj/AT/rrKkmktoh0G3pxkTfQZgg e4NT2tOguVfBmGKjX4YI3NC7V8Xv0d+zwxCDfieddnfXIkjpFwJdCCwYN fLuei1lO+m3Xh9V6YGjOQpi35VM2Wjx/Q6C9Ku2ME60aK1Mck4F6gM7z8 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385170" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385170" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705855" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705855" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:38 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:27 -0700 Message-Id: <20230401144429.88673-11-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 10/12] vfio: Mark cdev usage in vfio_device X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" There are users that need to check if vfio_device is opened as cdev. e.g. vfio-pci. This adds a flag in vfio_device, it will be set in the cdev path when device is opened. This is not used at this moment, but a preparation for vfio device cdev support. Signed-off-by: Yi Liu --- include/linux/vfio.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index f8fb9ab25188..d9a0770e5fc1 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -62,6 +62,7 @@ struct vfio_device { struct iommufd_device *iommufd_device; bool iommufd_attached; #endif + bool cdev_opened; }; /** @@ -151,6 +152,12 @@ vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) #endif +static inline bool vfio_device_cdev_opened(struct vfio_device *device) +{ + lockdep_assert_held(&device->dev_set->lock); + return device->cdev_opened; +} + /** * @migration_set_state: Optional callback to change the migration state for * devices that support migration. It's mandatory for From patchwork Sat Apr 1 14:44:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CE57C77B6D for ; Sat, 1 Apr 2023 14:44:52 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0A3BB10E286; Sat, 1 Apr 2023 14:44:47 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 44F9810E05B; Sat, 1 Apr 2023 14:44:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360280; x=1711896280; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=y2zlxKZ9NOOslsssQl+DjQN7OV6B46QdrinsLbD5orE=; b=CCOMdBGrP7PsKfATceXEu+I6dCP66P8OzWohzY34P1Rec3u8FyjgViVq Iu5/rrL1ZyanJBIB0hzBh4L9DONJShGo8vDhmvbtFa/qNkJZVR7FEUOKj EvjBn2KLgqoD4zIweWVIE+LcIkkox0qdcA5k+ZClShhEI8WCGfp2bGhkI HbjX7s0KXDaJ8iYzTQZ1D/OJubhnRzVKeqV8OUJIld1bwkII41qjF1hKI yqz8qeXkXbm+4elanZJcj8XaPG/7ees+RdHiL5GvccoBpEsv+sfhBwLXv 1DwBJPhu6c+u5lTLyQ4xu2JWtbgfN0lVMNgVaKIZzySrbymzv+dluBCEe g==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385180" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385180" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705859" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705859" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:39 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:28 -0700 Message-Id: <20230401144429.88673-12-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 11/12] iommufd: Define IOMMUFD_INVALID_ID in uapi X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" as there are IOMMUFD users that want to know check if an ID generated by IOMMUFD is valid or not. e.g. vfio-pci optionaly returns invalid dev_id to user in the VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl. User needs to check if the ID is valid or not. IOMMUFD_INVALID_ID is defined as 0 since the IDs generated by IOMMUFD starts from 0. Signed-off-by: Yi Liu --- include/uapi/linux/iommufd.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index 98ebba80cfa1..aeae73a93833 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -9,6 +9,9 @@ #define IOMMUFD_TYPE (';') +/* IDs allocated by IOMMUFD starts from 0 */ +#define IOMMUFD_INVALID_ID 0 + /** * DOC: General ioctl format * From patchwork Sat Apr 1 14:44:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13197019 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6BAC4C6FD1D for ; Sat, 1 Apr 2023 14:44:54 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A072810E26F; Sat, 1 Apr 2023 14:44:51 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1A3CA10E24F; Sat, 1 Apr 2023 14:44:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680360282; x=1711896282; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=p64YHQtdhIkzhEK1tO8XIf7bxcuE3M+6sbnj/z7LnJY=; b=Moz9bcFjycKXRjviIikYL3tnXc/cGepdR0uSUzi/kGj+Dk4mn/DH4Dvl 12muE6X627JfZSC3qtbhmWFh+qgFyP8zhleCrcA4YEsGTl2gPYmIQafnE cTnSSPVkX0T0y/CJBtbZ3FfAO2TatHtxVt36OYxAyQO1OZn447LxvvJRE RSJtgy/JGw55he1Sl/ppd6fxP1vBUVdOhdxQ/5PCcwrYrzCRxyfmBvuii XKcSbdZadNKKg6dTb7uWXT/30TJoHx2JLFPi4FTJwCzy9TsiTEp1Q79VO uGtcGE0aVTWsACJEld1dqvOK3yKmFFIvuLqytLfjjDDMnXr4mwNPWd6lh A==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="340385190" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="340385190" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 07:44:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662705864" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662705864" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 07:44:41 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Sat, 1 Apr 2023 07:44:29 -0700 Message-Id: <20230401144429.88673-13-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230401144429.88673-1-yi.l.liu@intel.com> References: <20230401144429.88673-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 12/12] vfio/pci: Report dev_id in VFIO_DEVICE_GET_PCI_HOT_RESET_INFO X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" for the users that accept device fds passed from management stacks to be able to figure out the host reset affected devices among the devices opened by the user. This is needed as such users do not have BDF (bus, devfn) knowledge about the devices it has opened, hence unable to use the information reported by existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to figure out the affected devices. Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 58 ++++++++++++++++++++++++++++---- include/uapi/linux/vfio.h | 24 ++++++++++++- 2 files changed, 74 insertions(+), 8 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 19f5b075d70a..a5a7e148dce1 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -30,6 +30,7 @@ #if IS_ENABLED(CONFIG_EEH) #include #endif +#include #include "vfio_pci_priv.h" @@ -767,6 +768,20 @@ static int vfio_pci_get_irq_count(struct vfio_pci_core_device *vdev, int irq_typ return 0; } +static struct vfio_device * +vfio_pci_find_device_in_devset(struct vfio_device_set *dev_set, + struct pci_dev *pdev) +{ + struct vfio_device *cur; + + lockdep_assert_held(&dev_set->lock); + + list_for_each_entry(cur, &dev_set->device_list, dev_set_list) + if (cur->dev == &pdev->dev) + return cur; + return NULL; +} + static int vfio_pci_count_devs(struct pci_dev *pdev, void *data) { (*(int *)data)++; @@ -776,13 +791,20 @@ static int vfio_pci_count_devs(struct pci_dev *pdev, void *data) struct vfio_pci_fill_info { int max; int cur; + bool require_devid; + struct iommufd_ctx *iommufd; + struct vfio_device_set *dev_set; struct vfio_pci_dependent_device *devices; }; static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) { struct vfio_pci_fill_info *fill = data; + struct vfio_device_set *dev_set = fill->dev_set; struct iommu_group *iommu_group; + struct vfio_device *vdev; + + lockdep_assert_held(&dev_set->lock); if (fill->cur == fill->max) return -EAGAIN; /* Something changed, try again */ @@ -791,7 +813,21 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) if (!iommu_group) return -EPERM; /* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id = iommu_group_id(iommu_group); + if (fill->require_devid) { + /* + * Report dev_id of the devices that are opened as cdev + * and have the same iommufd with the fill->iommufd. + * Otherwise, just fill IOMMUFD_INVALID_ID. + */ + vdev = vfio_pci_find_device_in_devset(dev_set, pdev); + if (vdev && vfio_device_cdev_opened(vdev) && + fill->iommufd == vfio_iommufd_physical_ictx(vdev)) + vfio_iommufd_physical_devid(vdev, &fill->devices[fill->cur].dev_id); + else + fill->devices[fill->cur].dev_id = IOMMUFD_INVALID_ID; + } else { + fill->devices[fill->cur].group_id = iommu_group_id(iommu_group); + } fill->devices[fill->cur].segment = pci_domain_nr(pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn; @@ -1230,17 +1266,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return -ENOMEM; fill.devices = devices; + fill.dev_set = vdev->vdev.dev_set; + mutex_lock(&vdev->vdev.dev_set->lock); + if (vfio_device_cdev_opened(&vdev->vdev)) { + fill.require_devid = true; + fill.iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + } ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_fill_devs, &fill, slot); + mutex_unlock(&vdev->vdev.dev_set->lock); /* * If a device was removed between counting and filling, we may come up * short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ - if (!ret) + if (!ret) { hdr.count = fill.cur; + if (fill.require_devid) + hdr.flags = VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID; + } reset_info_exit: if (copy_to_user(arg, &hdr, minsz)) @@ -2346,12 +2392,10 @@ static bool vfio_dev_in_files(struct vfio_pci_core_device *vdev, static int vfio_pci_is_device_in_set(struct pci_dev *pdev, void *data) { struct vfio_device_set *dev_set = data; - struct vfio_device *cur; - list_for_each_entry(cur, &dev_set->device_list, dev_set_list) - if (cur->dev == &pdev->dev) - return 0; - return -EBUSY; + lockdep_assert_held(&dev_set->lock); + + return vfio_pci_find_device_in_devset(dev_set, pdev) ? 0 : -EBUSY; } /* diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 25432ef213ee..5a34364e3b94 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -650,11 +650,32 @@ enum { * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * + * This command is used to query the affected devices in the hot reset for + * a given device. User could use the information reported by this command + * to figure out the affected devices among the devices it has opened. + * This command always reports the segment, bus and devfn information for + * each affected device, and selectively report the group_id or the dev_id + * per the way how the device being queried is opened. + * - If the device is opened via the traditional group/container manner, + * this command reports the group_id for each affected device. + * + * - If the device is opened as a cdev, this command needs to report + * dev_id for each affected device and set the + * VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID flag. For the affected + * devices that are not opened as cdev or bound to different iommufds + * with the device that is queried, report an invalid dev_id to avoid + * potential dev_id conflict as dev_id is local to iommufd. For such + * affected devices, user shall fall back to use the segment, bus and + * devfn info to map it to opened device. + * * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { - __u32 group_id; + union { + __u32 group_id; + __u32 dev_id; + }; __u16 segment; __u8 bus; __u8 devfn; /* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +684,7 @@ struct vfio_pci_dependent_device { struct vfio_pci_hot_reset_info { __u32 argsz; __u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID (1 << 0) __u32 count; struct vfio_pci_dependent_device devices[]; };