Message ID | 20231026024930.382898-1-baolu.lu@linux.intel.com (mailing list archive) |
---|---|
Headers | show |
Series | IOMMUFD: Deliver IO page faults to user space | expand |
On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > Hi folks, > > This series implements the functionality of delivering IO page faults to > user space through the IOMMUFD framework for nested translation. Nested > translation is a hardware feature that supports two-stage translation > tables for IOMMU. The second-stage translation table is managed by the > host VMM, while the first-stage translation table is owned by user > space. This allows user space to control the IOMMU mappings for its > devices. Having now looked more closely at the ARM requirements it seems we will need generic events, not just page fault events to have a complete emulation. So I'd like to see this generalized into a channel to carry any events.. > User space indicates its capability of handling IO page faults by > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > for page fault delivery. On a successful return of HWPT allocation, the > user can retrieve and respond to page faults by reading and writing to > the file descriptor (FD) returned in out_fault_fd. This is the right way to approach it, and more broadly this shouldn't be an iommufd specific thing. Kernel drivers will also need to create fault capable PAGING iommu domains. Jason
> From: Jason Gunthorpe <jgg@ziepe.ca> > Sent: Thursday, November 2, 2023 8:48 PM > > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > > Hi folks, > > > > This series implements the functionality of delivering IO page faults to > > user space through the IOMMUFD framework for nested translation. > Nested > > translation is a hardware feature that supports two-stage translation > > tables for IOMMU. The second-stage translation table is managed by the > > host VMM, while the first-stage translation table is owned by user > > space. This allows user space to control the IOMMU mappings for its > > devices. > > Having now looked more closely at the ARM requirements it seems we > will need generic events, not just page fault events to have a > complete emulation. Can you elaborate? > > So I'd like to see this generalized into a channel to carry any > events.. > > > User space indicates its capability of handling IO page faults by > > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > > for page fault delivery. On a successful return of HWPT allocation, the > > user can retrieve and respond to page faults by reading and writing to > > the file descriptor (FD) returned in out_fault_fd. > > This is the right way to approach it, and more broadly this shouldn't > be an iommufd specific thing. Kernel drivers will also need to create > fault capable PAGING iommu domains. > Are you suggesting a common interface used by both iommufd and kernel drivers? but I didn't get the last piece. If those domains are created by kernel drivers why would they require a uAPI for userspace to specify fault capable?
On Tue, Nov 07, 2023 at 08:35:10AM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe <jgg@ziepe.ca> > > Sent: Thursday, November 2, 2023 8:48 PM > > > > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > > > Hi folks, > > > > > > This series implements the functionality of delivering IO page faults to > > > user space through the IOMMUFD framework for nested translation. > > Nested > > > translation is a hardware feature that supports two-stage translation > > > tables for IOMMU. The second-stage translation table is managed by the > > > host VMM, while the first-stage translation table is owned by user > > > space. This allows user space to control the IOMMU mappings for its > > > devices. > > > > Having now looked more closely at the ARM requirements it seems we > > will need generic events, not just page fault events to have a > > complete emulation. > > Can you elaborate? There are many events related to object in guest memory or controlled by the guest, eg C_BAD_CD and C_BAD_STE. These should be relayed or the emulation is not working well. > > > User space indicates its capability of handling IO page faults by > > > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > > > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > > > for page fault delivery. On a successful return of HWPT allocation, the > > > user can retrieve and respond to page faults by reading and writing to > > > the file descriptor (FD) returned in out_fault_fd. > > > > This is the right way to approach it, and more broadly this shouldn't > > be an iommufd specific thing. Kernel drivers will also need to create > > fault capable PAGING iommu domains. > > Are you suggesting a common interface used by both iommufd and > kernel drivers? Yes > but I didn't get the last piece. If those domains are created by kernel > drivers why would they require a uAPI for userspace to specify fault > capable? Not to userspace, but a kapi to request a fault capable domain and to supply the fault handler. Eg: iommu_domain_alloc_faultable(dev, handler); Jason
> From: Jason Gunthorpe <jgg@ziepe.ca> > Sent: Wednesday, November 8, 2023 1:54 AM > > On Tue, Nov 07, 2023 at 08:35:10AM +0000, Tian, Kevin wrote: > > > From: Jason Gunthorpe <jgg@ziepe.ca> > > > Sent: Thursday, November 2, 2023 8:48 PM > > > > > > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > > > > Hi folks, > > > > > > > > This series implements the functionality of delivering IO page faults to > > > > user space through the IOMMUFD framework for nested translation. > > > Nested > > > > translation is a hardware feature that supports two-stage translation > > > > tables for IOMMU. The second-stage translation table is managed by > the > > > > host VMM, while the first-stage translation table is owned by user > > > > space. This allows user space to control the IOMMU mappings for its > > > > devices. > > > > > > Having now looked more closely at the ARM requirements it seems we > > > will need generic events, not just page fault events to have a > > > complete emulation. > > > > Can you elaborate? > > There are many events related to object in guest memory or controlled > by the guest, eg C_BAD_CD and C_BAD_STE. These should be relayed or > the emulation is not working well. so that's the category of unrecoverable faults? btw I can understand C_BAD_CD given it's walked by the physical SMMU in nested configuration. But presumably STE is created by the smmu driver itself then why would there be an error to be relayed for guest STE? > > > > > User space indicates its capability of handling IO page faults by > > > > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > > > > hardware page table (HWPT). IOMMUFD will then set up its > infrastructure > > > > for page fault delivery. On a successful return of HWPT allocation, the > > > > user can retrieve and respond to page faults by reading and writing to > > > > the file descriptor (FD) returned in out_fault_fd. > > > > > > This is the right way to approach it, and more broadly this shouldn't > > > be an iommufd specific thing. Kernel drivers will also need to create > > > fault capable PAGING iommu domains. > > > > Are you suggesting a common interface used by both iommufd and > > kernel drivers? > > Yes > > > but I didn't get the last piece. If those domains are created by kernel > > drivers why would they require a uAPI for userspace to specify fault > > capable? > > Not to userspace, but a kapi to request a fault capable domain and to > supply the fault handler. Eg: > > iommu_domain_alloc_faultable(dev, handler); > Does it affect SVA too?
On Wed, Nov 08, 2023 at 08:53:00AM +0000, Tian, Kevin wrote: > > There are many events related to object in guest memory or controlled > > by the guest, eg C_BAD_CD and C_BAD_STE. These should be relayed or > > the emulation is not working well. > > so that's the category of unrecoverable faults? I haven't looked exhaustively but I do have the impression that the only recoverable fault is the 'page not present' one. > btw I can understand C_BAD_CD given it's walked by the physical SMMU > in nested configuration. But presumably STE is created by the smmu > driver itself then why would there be an error to be relayed for > guest STE? If the guest programs a bad STE it should still generate a C_BAD_STE even if the mediation SW could theoretically sanitize it (but sanitize it to what? BLOCKED?). Since we have to forward things like C_BAD_CD and others we may as well just drop an invalid STE and forward the event like real HW. > > > but I didn't get the last piece. If those domains are created by kernel > > > drivers why would they require a uAPI for userspace to specify fault > > > capable? > > > > Not to userspace, but a kapi to request a fault capable domain and to > > supply the fault handler. Eg: > > > > iommu_domain_alloc_faultable(dev, handler); > > Does it affect SVA too? Inside the driver the SVA should be constructed out of the same fault handling infrastructure, but a SVA domain allocation should have a different allocation function. Jason
On Wed, Nov 15, 2023 at 01:17:06PM +0800, Liu, Jing2 wrote: > This is the right way to approach it, > > I learned that there was discussion about using io_uring to get the > page fault without > > eventfd notification in [1], and I am new at io_uring and studying the > man page of > > liburing, but there're questions in my mind on how can QEMU get the > coming page fault > > with a good performance. > > Since both QEMU and Kernel don't know when comes faults, after QEMU > submits one > > read task to io_uring, we want kernel pending until fault comes. While > based on > > hwpt_fault_fops_read() in [patch v2 4/6], it just returns 0 since > there's now no fault, > > thus this round of read completes to CQ but it's not what we want. So > I'm wondering > > how kernel pending on the read until fault comes. Does fops callback > need special work to Implement a fops with poll support that triggers when a new event is pushed and everything will be fine. There are many examples in the kernel. The ones in the mlx5 vfio driver spring to mind as a scheme I recently looked at. Jason
Hi Jason, On 11/15/2023 9:58 PM, Jason Gunthorpe wrote: > On Wed, Nov 15, 2023 at 01:17:06PM +0800, Liu, Jing2 wrote: > >> This is the right way to approach it, >> >> I learned that there was discussion about using io_uring to get the >> page fault without >> >> eventfd notification in [1], and I am new at io_uring and studying the >> man page of >> >> liburing, but there're questions in my mind on how can QEMU get the >> coming page fault >> >> with a good performance. >> >> Since both QEMU and Kernel don't know when comes faults, after QEMU >> submits one >> >> read task to io_uring, we want kernel pending until fault comes. While >> based on >> >> hwpt_fault_fops_read() in [patch v2 4/6], it just returns 0 since >> there's now no fault, >> >> thus this round of read completes to CQ but it's not what we want. So >> I'm wondering >> >> how kernel pending on the read until fault comes. Does fops callback >> need special work to > Implement a fops with poll support that triggers when a new event is > pushed and everything will be fine. Does userspace need also setup a POLL flag to let io_uring go into poll, or io_uring always try to poll? > There are many examples in the > kernel. The ones in the mlx5 vfio driver spring to mind as a scheme I > recently looked at. Thank you very much for guiding the way. We will study the example to understand more. BRs, Jing > Jason
On Thu, Nov 16, 2023 at 09:42:23AM +0800, Liu, Jing2 wrote: > Hi Jason, > > On 11/15/2023 9:58 PM, Jason Gunthorpe wrote: > > On Wed, Nov 15, 2023 at 01:17:06PM +0800, Liu, Jing2 wrote: > > > > > This is the right way to approach it, > > > > > > I learned that there was discussion about using io_uring to get the > > > page fault without > > > > > > eventfd notification in [1], and I am new at io_uring and studying the > > > man page of > > > > > > liburing, but there're questions in my mind on how can QEMU get the > > > coming page fault > > > > > > with a good performance. > > > > > > Since both QEMU and Kernel don't know when comes faults, after QEMU > > > submits one > > > > > > read task to io_uring, we want kernel pending until fault comes. While > > > based on > > > > > > hwpt_fault_fops_read() in [patch v2 4/6], it just returns 0 since > > > there's now no fault, > > > > > > thus this round of read completes to CQ but it's not what we want. So > > > I'm wondering > > > > > > how kernel pending on the read until fault comes. Does fops callback > > > need special work to > > Implement a fops with poll support that triggers when a new event is > > pushed and everything will be fine. > > Does userspace need also setup a POLL flag to let io_uring go into poll, or > io_uring always try to poll? io_uring can trigger poll and use other approaches, it is flexible the driver can scale this in different ways. Jason
> -----Original Message----- > From: Lu Baolu [mailto:baolu.lu@linux.intel.com] > Sent: 26 October 2023 03:49 > To: Jason Gunthorpe <jgg@ziepe.ca>; Kevin Tian <kevin.tian@intel.com>; > Joerg Roedel <joro@8bytes.org>; Will Deacon <will@kernel.org>; Robin > Murphy <robin.murphy@arm.com>; Jean-Philippe Brucker > <jean-philippe@linaro.org>; Nicolin Chen <nicolinc@nvidia.com>; Yi Liu > <yi.l.liu@intel.com>; Jacob Pan <jacob.jun.pan@linux.intel.com> > Cc: iommu@lists.linux.dev; linux-kselftest@vger.kernel.org; > virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org; Lu > Baolu <baolu.lu@linux.intel.com> > Subject: [PATCH v2 0/6] IOMMUFD: Deliver IO page faults to user space > > Hi folks, > > This series implements the functionality of delivering IO page faults to > user space through the IOMMUFD framework for nested translation. Nested > translation is a hardware feature that supports two-stage translation > tables for IOMMU. The second-stage translation table is managed by the > host VMM, while the first-stage translation table is owned by user > space. This allows user space to control the IOMMU mappings for its > devices. > > When an IO page fault occurs on the first-stage translation table, the > IOMMU hardware can deliver the page fault to user space through the > IOMMUFD framework. User space can then handle the page fault and > respond > to the device top-down through the IOMMUFD. This allows user space to > implement its own IO page fault handling policies. > > User space indicates its capability of handling IO page faults by > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > for page fault delivery. On a successful return of HWPT allocation, the > user can retrieve and respond to page faults by reading and writing to > the file descriptor (FD) returned in out_fault_fd. > > The iommu selftest framework has been updated to test the IO page fault > delivery and response functionality. > > This series is based on the latest implementation of nested translation > under discussion [1] and the page fault handling framework refactoring in > the IOMMU core [2]. > > The series and related patches are available on GitHub: [3] > > [1] > https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@int > el.com/ > [2] > https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@li > nux.intel.com/ > [3] > https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-deliv > ery-v2 Hi Baolu, Do you have a corresponding Qemu git to share? I could give it a spin on our ARM platform. Please let me know. Thanks, Shameer
On 2023/11/29 17:08, Shameerali Kolothum Thodi wrote: > > >> -----Original Message----- >> From: Lu Baolu [mailto:baolu.lu@linux.intel.com] >> Sent: 26 October 2023 03:49 >> To: Jason Gunthorpe <jgg@ziepe.ca>; Kevin Tian <kevin.tian@intel.com>; >> Joerg Roedel <joro@8bytes.org>; Will Deacon <will@kernel.org>; Robin >> Murphy <robin.murphy@arm.com>; Jean-Philippe Brucker >> <jean-philippe@linaro.org>; Nicolin Chen <nicolinc@nvidia.com>; Yi Liu >> <yi.l.liu@intel.com>; Jacob Pan <jacob.jun.pan@linux.intel.com> >> Cc: iommu@lists.linux.dev; linux-kselftest@vger.kernel.org; >> virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org; Lu >> Baolu <baolu.lu@linux.intel.com> >> Subject: [PATCH v2 0/6] IOMMUFD: Deliver IO page faults to user space >> >> Hi folks, >> >> This series implements the functionality of delivering IO page faults to >> user space through the IOMMUFD framework for nested translation. Nested >> translation is a hardware feature that supports two-stage translation >> tables for IOMMU. The second-stage translation table is managed by the >> host VMM, while the first-stage translation table is owned by user >> space. This allows user space to control the IOMMU mappings for its >> devices. >> >> When an IO page fault occurs on the first-stage translation table, the >> IOMMU hardware can deliver the page fault to user space through the >> IOMMUFD framework. User space can then handle the page fault and >> respond >> to the device top-down through the IOMMUFD. This allows user space to >> implement its own IO page fault handling policies. >> >> User space indicates its capability of handling IO page faults by >> setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a >> hardware page table (HWPT). IOMMUFD will then set up its infrastructure >> for page fault delivery. On a successful return of HWPT allocation, the >> user can retrieve and respond to page faults by reading and writing to >> the file descriptor (FD) returned in out_fault_fd. >> >> The iommu selftest framework has been updated to test the IO page fault >> delivery and response functionality. >> >> This series is based on the latest implementation of nested translation >> under discussion [1] and the page fault handling framework refactoring in >> the IOMMU core [2]. >> >> The series and related patches are available on GitHub: [3] >> >> [1] >> https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@int >> el.com/ >> [2] >> https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@li >> nux.intel.com/ >> [3] >> https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-deliv >> ery-v2 > > Hi Baolu, Hi Shameer, > > Do you have a corresponding Qemu git to share? I could give it a spin on our ARM > platform. Please let me know. This version of the series is tested by the iommufd selftest. We are in process of developing the QEMU code. I will provide the repo link after we complete it. Best regards, baolu
On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > Hi folks, > > This series implements the functionality of delivering IO page faults to > user space through the IOMMUFD framework for nested translation. Nested > translation is a hardware feature that supports two-stage translation > tables for IOMMU. The second-stage translation table is managed by the > host VMM, while the first-stage translation table is owned by user > space. This allows user space to control the IOMMU mappings for its > devices. > > When an IO page fault occurs on the first-stage translation table, the > IOMMU hardware can deliver the page fault to user space through the > IOMMUFD framework. User space can then handle the page fault and respond > to the device top-down through the IOMMUFD. This allows user space to > implement its own IO page fault handling policies. > > User space indicates its capability of handling IO page faults by > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > for page fault delivery. On a successful return of HWPT allocation, the > user can retrieve and respond to page faults by reading and writing to > the file descriptor (FD) returned in out_fault_fd. This is probably backwards, userspace should allocate the FD with a dedicated ioctl and provide it during domain allocation. If the userspace wants a fd per domain then it should do that. If it wants to share fds between domains that should work too. Jason
On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > Hi folks, > > This series implements the functionality of delivering IO page faults to > user space through the IOMMUFD framework for nested translation. Nested Does this mean the IOPF_CAPABLE HWPT needs to be parented by a HWPT created with IOMMU_HWPT_ALLOC_NEST_PARENT set? > translation is a hardware feature that supports two-stage translation > tables for IOMMU. The second-stage translation table is managed by the > host VMM, while the first-stage translation table is owned by user > space. This allows user space to control the IOMMU mappings for its > devices. > > When an IO page fault occurs on the first-stage translation table, the > IOMMU hardware can deliver the page fault to user space through the > IOMMUFD framework. User space can then handle the page fault and respond > to the device top-down through the IOMMUFD. This allows user space to > implement its own IO page fault handling policies. > > User space indicates its capability of handling IO page faults by > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > for page fault delivery. On a successful return of HWPT allocation, the > user can retrieve and respond to page faults by reading and writing to > the file descriptor (FD) returned in out_fault_fd. > > The iommu selftest framework has been updated to test the IO page fault > delivery and response functionality. > > This series is based on the latest implementation of nested translation > under discussion [1] and the page fault handling framework refactoring in > the IOMMU core [2]. > > The series and related patches are available on GitHub: [3] > > [1] https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@intel.com/ > [2] https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ > [3] https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-delivery-v2 > > Best regards, > baolu > > Change log: > v2: > - Move all iommu refactoring patches into a sparated series and discuss > it in a different thread. The latest patch series [v6] is available at > https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ > - We discussed the timeout of the pending page fault messages. We > agreed that we shouldn't apply any timeout policy for the page fault > handling in user space. > https://lore.kernel.org/linux-iommu/20230616113232.GA84678@myrica/ > - Jason suggested that we adopt a simple file descriptor interface for > reading and responding to I/O page requests, so that user space > applications can improve performance using io_uring. > https://lore.kernel.org/linux-iommu/ZJWjD1ajeem6pK3I@ziepe.ca/ > > v1: https://lore.kernel.org/linux-iommu/20230530053724.232765-1-baolu.lu@linux.intel.com/ > > Lu Baolu (6): > iommu: Add iommu page fault cookie helpers > iommufd: Add iommu page fault uapi data > iommufd: Initializing and releasing IO page fault data > iommufd: Deliver fault messages to user space > iommufd/selftest: Add IOMMU_TEST_OP_TRIGGER_IOPF test support > iommufd/selftest: Add coverage for IOMMU_TEST_OP_TRIGGER_IOPF > > include/linux/iommu.h | 9 + > drivers/iommu/iommu-priv.h | 15 + > drivers/iommu/iommufd/iommufd_private.h | 12 + > drivers/iommu/iommufd/iommufd_test.h | 8 + > include/uapi/linux/iommufd.h | 65 +++++ > tools/testing/selftests/iommu/iommufd_utils.h | 66 ++++- > drivers/iommu/io-pgfault.c | 50 ++++ > drivers/iommu/iommufd/device.c | 69 ++++- > drivers/iommu/iommufd/hw_pagetable.c | 260 +++++++++++++++++- > drivers/iommu/iommufd/selftest.c | 56 ++++ > tools/testing/selftests/iommu/iommufd.c | 24 +- > .../selftests/iommu/iommufd_fail_nth.c | 2 +- > 12 files changed, 620 insertions(+), 16 deletions(-) > > -- > 2.34.1 >
On Mon, Dec 04, 2023 at 04:07:44PM +0100, Joel Granados wrote: > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > > Hi folks, > > > > This series implements the functionality of delivering IO page faults to > > user space through the IOMMUFD framework for nested translation. Nested > Does this mean the IOPF_CAPABLE HWPT needs to be parented by a HWPT > created with IOMMU_HWPT_ALLOC_NEST_PARENT set? I would expect no. Both a nested and un-nested configuration should work. Jason
On 12/4/23 11:07 PM, Joel Granados wrote: > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: >> Hi folks, >> >> This series implements the functionality of delivering IO page faults to >> user space through the IOMMUFD framework for nested translation. Nested > Does this mean the IOPF_CAPABLE HWPT needs to be parented by a HWPT > created with IOMMU_HWPT_ALLOC_NEST_PARENT set? No. It's generic, nested translation is simply a use case that is currently feasible. Best regards, baolu
On 12/1/23 10:24 PM, Jason Gunthorpe wrote: > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: >> Hi folks, >> >> This series implements the functionality of delivering IO page faults to >> user space through the IOMMUFD framework for nested translation. Nested >> translation is a hardware feature that supports two-stage translation >> tables for IOMMU. The second-stage translation table is managed by the >> host VMM, while the first-stage translation table is owned by user >> space. This allows user space to control the IOMMU mappings for its >> devices. >> >> When an IO page fault occurs on the first-stage translation table, the >> IOMMU hardware can deliver the page fault to user space through the >> IOMMUFD framework. User space can then handle the page fault and respond >> to the device top-down through the IOMMUFD. This allows user space to >> implement its own IO page fault handling policies. >> >> User space indicates its capability of handling IO page faults by >> setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a >> hardware page table (HWPT). IOMMUFD will then set up its infrastructure >> for page fault delivery. On a successful return of HWPT allocation, the >> user can retrieve and respond to page faults by reading and writing to >> the file descriptor (FD) returned in out_fault_fd. > > This is probably backwards, userspace should allocate the FD with a > dedicated ioctl and provide it during domain allocation. Introducing a dedicated fault FD for fault handling seems promising. It decouples the fault handling from any specific domain. I suppose we need different fault fd for recoverable faults (a.k.a. IO page fault) and unrecoverable faults. Do I understand you correctly? > If the userspace wants a fd per domain then it should do that. If it > wants to share fds between domains that should work too. Yes, it's more flexible. The fault message contains the hwpt obj id, so user space can recognize the hwpt on which the fault happened. Best regards, baolu
On Fri, Dec 08, 2023 at 01:57:26PM +0800, Baolu Lu wrote: > On 12/1/23 10:24 PM, Jason Gunthorpe wrote: > > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > > > Hi folks, > > > > > > This series implements the functionality of delivering IO page faults to > > > user space through the IOMMUFD framework for nested translation. Nested > > > translation is a hardware feature that supports two-stage translation > > > tables for IOMMU. The second-stage translation table is managed by the > > > host VMM, while the first-stage translation table is owned by user > > > space. This allows user space to control the IOMMU mappings for its > > > devices. > > > > > > When an IO page fault occurs on the first-stage translation table, the > > > IOMMU hardware can deliver the page fault to user space through the > > > IOMMUFD framework. User space can then handle the page fault and respond > > > to the device top-down through the IOMMUFD. This allows user space to > > > implement its own IO page fault handling policies. > > > > > > User space indicates its capability of handling IO page faults by > > > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > > > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > > > for page fault delivery. On a successful return of HWPT allocation, the > > > user can retrieve and respond to page faults by reading and writing to > > > the file descriptor (FD) returned in out_fault_fd. > > > > This is probably backwards, userspace should allocate the FD with a > > dedicated ioctl and provide it during domain allocation. > > Introducing a dedicated fault FD for fault handling seems promising. It > decouples the fault handling from any specific domain. I suppose we need > different fault fd for recoverable faults (a.k.a. IO page fault) and > unrecoverable faults. Do I understand you correctly? I haven't thought that far ahead :) Once you have a generic fault FD concept it can be sliced in different ways. If there is a technical need to seperate recoverable/unrecoverable then the FD flavour should be specified during FD creation. Otherwise just let userspace do whatever it wants. Jason
On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > Hi folks, > > This series implements the functionality of delivering IO page faults to > user space through the IOMMUFD framework for nested translation. Nested > translation is a hardware feature that supports two-stage translation > tables for IOMMU. The second-stage translation table is managed by the > host VMM, while the first-stage translation table is owned by user > space. This allows user space to control the IOMMU mappings for its > devices. > > When an IO page fault occurs on the first-stage translation table, the > IOMMU hardware can deliver the page fault to user space through the > IOMMUFD framework. User space can then handle the page fault and respond > to the device top-down through the IOMMUFD. This allows user space to > implement its own IO page fault handling policies. > > User space indicates its capability of handling IO page faults by > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > hardware page table (HWPT). IOMMUFD will then set up its infrastructure > for page fault delivery. On a successful return of HWPT allocation, the > user can retrieve and respond to page faults by reading and writing to > the file descriptor (FD) returned in out_fault_fd. > > The iommu selftest framework has been updated to test the IO page fault > delivery and response functionality. > > This series is based on the latest implementation of nested translation > under discussion [1] and the page fault handling framework refactoring in > the IOMMU core [2]. > > The series and related patches are available on GitHub: [3] > > [1] https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@intel.com/ > [2] https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ > [3] https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-delivery-v2 I was working with this branch that included Yi Liu's wip/iommufd_nesting branch. Now Yi Lui has updated his work in this post https://lore.kernel.org/all/20240102143834.146165-1-yi.l.liu@intel.com. Is there an updated version of the page fault work that is rebased on top of Liu's new version? Thx in advance Best > > Best regards, > baolu > > Change log: > v2: > - Move all iommu refactoring patches into a sparated series and discuss > it in a different thread. The latest patch series [v6] is available at > https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ > - We discussed the timeout of the pending page fault messages. We > agreed that we shouldn't apply any timeout policy for the page fault > handling in user space. > https://lore.kernel.org/linux-iommu/20230616113232.GA84678@myrica/ > - Jason suggested that we adopt a simple file descriptor interface for > reading and responding to I/O page requests, so that user space > applications can improve performance using io_uring. > https://lore.kernel.org/linux-iommu/ZJWjD1ajeem6pK3I@ziepe.ca/ > > v1: https://lore.kernel.org/linux-iommu/20230530053724.232765-1-baolu.lu@linux.intel.com/ > > Lu Baolu (6): > iommu: Add iommu page fault cookie helpers > iommufd: Add iommu page fault uapi data > iommufd: Initializing and releasing IO page fault data > iommufd: Deliver fault messages to user space > iommufd/selftest: Add IOMMU_TEST_OP_TRIGGER_IOPF test support > iommufd/selftest: Add coverage for IOMMU_TEST_OP_TRIGGER_IOPF > > include/linux/iommu.h | 9 + > drivers/iommu/iommu-priv.h | 15 + > drivers/iommu/iommufd/iommufd_private.h | 12 + > drivers/iommu/iommufd/iommufd_test.h | 8 + > include/uapi/linux/iommufd.h | 65 +++++ > tools/testing/selftests/iommu/iommufd_utils.h | 66 ++++- > drivers/iommu/io-pgfault.c | 50 ++++ > drivers/iommu/iommufd/device.c | 69 ++++- > drivers/iommu/iommufd/hw_pagetable.c | 260 +++++++++++++++++- > drivers/iommu/iommufd/selftest.c | 56 ++++ > tools/testing/selftests/iommu/iommufd.c | 24 +- > .../selftests/iommu/iommufd_fail_nth.c | 2 +- > 12 files changed, 620 insertions(+), 16 deletions(-) > > -- > 2.34.1 >
On 2024/1/13 5:56, Joel Granados wrote: > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: >> Hi folks, >> >> This series implements the functionality of delivering IO page faults to >> user space through the IOMMUFD framework for nested translation. Nested >> translation is a hardware feature that supports two-stage translation >> tables for IOMMU. The second-stage translation table is managed by the >> host VMM, while the first-stage translation table is owned by user >> space. This allows user space to control the IOMMU mappings for its >> devices. >> >> When an IO page fault occurs on the first-stage translation table, the >> IOMMU hardware can deliver the page fault to user space through the >> IOMMUFD framework. User space can then handle the page fault and respond >> to the device top-down through the IOMMUFD. This allows user space to >> implement its own IO page fault handling policies. >> >> User space indicates its capability of handling IO page faults by >> setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a >> hardware page table (HWPT). IOMMUFD will then set up its infrastructure >> for page fault delivery. On a successful return of HWPT allocation, the >> user can retrieve and respond to page faults by reading and writing to >> the file descriptor (FD) returned in out_fault_fd. >> >> The iommu selftest framework has been updated to test the IO page fault >> delivery and response functionality. >> >> This series is based on the latest implementation of nested translation >> under discussion [1] and the page fault handling framework refactoring in >> the IOMMU core [2]. >> >> The series and related patches are available on GitHub: [3] >> >> [1]https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@intel.com/ >> [2]https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ >> [3]https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-delivery-v2 > I was working with this branch that included Yi Liu's > wip/iommufd_nesting branch. Now Yi Lui has updated his work in this post > https://lore.kernel.org/all/20240102143834.146165-1-yi.l.liu@intel.com. > Is there an updated version of the page fault work that is rebased on > top of Liu's new version? Yes. I am preparing the new version and will post it for discussion after the merge window. Best regards, baolu
On Sun, Jan 14, 2024 at 09:13:19PM +0800, Baolu Lu wrote: > On 2024/1/13 5:56, Joel Granados wrote: > > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: > >> Hi folks, > >> > >> This series implements the functionality of delivering IO page faults to > >> user space through the IOMMUFD framework for nested translation. Nested > >> translation is a hardware feature that supports two-stage translation > >> tables for IOMMU. The second-stage translation table is managed by the > >> host VMM, while the first-stage translation table is owned by user > >> space. This allows user space to control the IOMMU mappings for its > >> devices. > >> > >> When an IO page fault occurs on the first-stage translation table, the > >> IOMMU hardware can deliver the page fault to user space through the > >> IOMMUFD framework. User space can then handle the page fault and respond > >> to the device top-down through the IOMMUFD. This allows user space to > >> implement its own IO page fault handling policies. > >> > >> User space indicates its capability of handling IO page faults by > >> setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a > >> hardware page table (HWPT). IOMMUFD will then set up its infrastructure > >> for page fault delivery. On a successful return of HWPT allocation, the > >> user can retrieve and respond to page faults by reading and writing to > >> the file descriptor (FD) returned in out_fault_fd. > >> > >> The iommu selftest framework has been updated to test the IO page fault > >> delivery and response functionality. > >> > >> This series is based on the latest implementation of nested translation > >> under discussion [1] and the page fault handling framework refactoring in > >> the IOMMU core [2]. > >> > >> The series and related patches are available on GitHub: [3] > >> > >> [1]https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@intel.com/ > >> [2]https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ > >> [3]https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-delivery-v2 > > I was working with this branch that included Yi Liu's > > wip/iommufd_nesting branch. Now Yi Lui has updated his work in this post > > https://lore.kernel.org/all/20240102143834.146165-1-yi.l.liu@intel.com. > > Is there an updated version of the page fault work that is rebased on > > top of Liu's new version? > > Yes. I am preparing the new version and will post it for discussion > after the merge window. Great to hear and thx for getting back to me. I'll be on the look out for your post. Would it be possible for you to add me to the CC when you send it? Best
On 1/15/24 1:18 AM, Joel Granados wrote: > On Sun, Jan 14, 2024 at 09:13:19PM +0800, Baolu Lu wrote: >> On 2024/1/13 5:56, Joel Granados wrote: >>> On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote: >>>> Hi folks, >>>> >>>> This series implements the functionality of delivering IO page faults to >>>> user space through the IOMMUFD framework for nested translation. Nested >>>> translation is a hardware feature that supports two-stage translation >>>> tables for IOMMU. The second-stage translation table is managed by the >>>> host VMM, while the first-stage translation table is owned by user >>>> space. This allows user space to control the IOMMU mappings for its >>>> devices. >>>> >>>> When an IO page fault occurs on the first-stage translation table, the >>>> IOMMU hardware can deliver the page fault to user space through the >>>> IOMMUFD framework. User space can then handle the page fault and respond >>>> to the device top-down through the IOMMUFD. This allows user space to >>>> implement its own IO page fault handling policies. >>>> >>>> User space indicates its capability of handling IO page faults by >>>> setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a >>>> hardware page table (HWPT). IOMMUFD will then set up its infrastructure >>>> for page fault delivery. On a successful return of HWPT allocation, the >>>> user can retrieve and respond to page faults by reading and writing to >>>> the file descriptor (FD) returned in out_fault_fd. >>>> >>>> The iommu selftest framework has been updated to test the IO page fault >>>> delivery and response functionality. >>>> >>>> This series is based on the latest implementation of nested translation >>>> under discussion [1] and the page fault handling framework refactoring in >>>> the IOMMU core [2]. >>>> >>>> The series and related patches are available on GitHub: [3] >>>> >>>> [1]https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@intel.com/ >>>> [2]https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ >>>> [3]https://github.com/LuBaolu/intel-iommu/commits/iommufd-io-pgfault-delivery-v2 >>> I was working with this branch that included Yi Liu's >>> wip/iommufd_nesting branch. Now Yi Lui has updated his work in this post >>> https://lore.kernel.org/all/20240102143834.146165-1-yi.l.liu@intel.com. >>> Is there an updated version of the page fault work that is rebased on >>> top of Liu's new version? >> Yes. I am preparing the new version and will post it for discussion >> after the merge window. > Great to hear and thx for getting back to me. > > I'll be on the look out for your post. Would it be possible for you to > add me to the CC when you send it? Sure. Best regards, baolu