| Message ID | 1571214873-27359-2-git-send-email-zhangfei.gao@linaro.org (mailing list archive) |
|---|---|
| State | Not Applicable |
| Delegated to: | Herbert Xu |
| Headers | show |
| Series | Add uacce module for Accelerator | expand |
Hi, I already commented on the interface in patch 2/3, so I just have a few additional comments on the documentation itself. On Wed, Oct 16, 2019 at 04:34:31PM +0800, Zhangfei Gao wrote: > +The user API > +------------ > + > +We adopt a polling style interface in the user space: :: > + > + int wd_request_queue(struct wd_queue *q); > + void wd_release_queue(struct wd_queue *q); > + int wd_send(struct wd_queue *q, void *req); > + int wd_recv(struct wd_queue *q, void **req); > + int wd_recv_sync(struct wd_queue *q, void **req); > + void wd_flush(struct wd_queue *q); > + > +wd_recv_sync() is a wrapper to its non-sync version. It will trap into > +kernel and wait until the queue become available. > + > +If the queue do not support SVA/SVM. The following helper functions > +can be used to create Static Virtual Share Memory: :: > + > + void *wd_reserve_memory(struct wd_queue *q, size_t size); > + int wd_share_reserved_memory(struct wd_queue *q, > + struct wd_queue *target_q); > + > +The user API is not mandatory. It is simply a suggestion and hint what the > +kernel interface is supposed to be. Maybe move this to the beginning of the section, to make it clear that you're describing an example API. On first read I found it odd that we're documenting a userspace library in this document. [...] > + > +The Memory Sharing Model > +------------------------ > +The perfect form of a Uacce device is to support SVM/SVA. We built this upon > +Jean Philippe Brucker's SVA patches. [1] I don't think this belongs in the doc, more on a cover letter. Since the SVA API is now upstream (implementation in progress), you could simply say something like "the uacce device is built around the IOMMU SVA API". > + > +If the hardware support UACCE_DEV_SVA, the user process's page table is > +shared to the opened queue. So the device can access any address in the > +process address space. > +And it can raise a page fault if the physical page is not available yet. > +It can also access the address in the kernel space, which is referred by > +another page table particular to the kernel. Most of IOMMU implementation can > +handle this by a tag on the address request of the device. For example, ARM > +SMMU uses SSV bit to indicate that the address request is for kernel or user > +space. That might be a bit too detailed, you can just say that the device can access multiple address spaces, including the one without PASID. All IOMMU architectures with PASID support this now. > +Queue file regions can be used: > +UACCE_QFRT_MMIO: device mmio region (map to user) > +UACCE_QFRT_DUS: device user share (map to dev and user) > + > +If the device does not support UACCE_DEV_SVA, Uacce allow only one process at > +the same time. DMA API cannot be used as well, since Uacce will create an > +unmanaged iommu_domain for the device. > +Queue file regions can be used: > +UACCE_QFRT_MMIO: device mmio region (map to user) > +UACCE_QFRT_DKO: device kernel-only (map to dev, no user) > +UACCE_QFRT_DUS: device user share (map to dev and user) > +UACCE_QFRT_SS: static shared memory (map to devs and user) > + > + > +The Fork Scenario > +================= > +For a process with allocated queues and shared memory, what happen if it forks > +a child? > + > +The fd of the queue will be duplicated on folk, so the child can send request > +to the same queue as its parent. But the requests which is sent from processes > +except for the one who opens the queue will be blocked. Would it be correct and clearer to say "The fd of the queue is duplicated on fork, but requests sent from the child process are blocked"? > + > +It is recommended to add O_CLOEXEC to the queue file. > + > +The queue mmap space has a VM_DONTCOPY in its VMA. So the child will lose all > +those VMAs. > + > +This is a reason why Uacce does not adopt the mode used in VFIO and > +InfiniBand. Both solutions can set any user pointer for hardware sharing. > +But they cannot support fork when the dma is in process. Or the > +"Copy-On-Write" procedure will make the parent process lost its physical > +pages. > + > + > +Difference to the VFIO and IB framework > +--------------------------------------- > +The essential function of Uacce is to let the device access the user > +address directly. There are many device drivers doing the same in the kernel. > +And both VFIO and IB can provide similar functions in framework level. > + > +But Uacce has a different goal: "share address space". It is > +not taken the request to the accelerator as an enclosure data structure. It > +takes the accelerator as another thread of the same process. So the > +accelerator can refer to any address used by the process. > + > +Both VFIO and IB are taken this as "memory sharing", not "address sharing". > +They care more on sharing the block of memory. But if there is an address > +stored in the block and referring to another memory region. The address may > +not be valid. > + > +By adding more constraints to the VFIO and IB framework, in some sense, we may > +achieve a similar goal. But we gave it up finally. Both VFIO and IB have extra > +assumption which is unnecessary to Uacce. They may hurt each other if we > +try to merge them together. I don't know if this particular rationale belongs here rather than a cover letter, but some of this section can be useful to let users decide if they need uacce or VFIO. For the record I'm still not entirely convinced that a new solution is preferable to vfio-mdev. * Existing userspace drivers such as DPDK may someday benefit from adding SVA support to VFIO. * Patch 2/3 does seem to duplicate a lot of VFIO code for the !SVA mode. I'd rather we avoided !SVA support altogether at first, to make the code simpler. * The issue with fork should be fixed in VFIO anyway, if it's an actual concern for userspace drivers. On the other hand, I do agree with the following paragraph that a lighter solution such as uacce focusing on shared address space and queues could mean less work for device drivers and libraries. It would be interesting to write a device driver prototype that implements both vfio-mdev (with added SVA support) and uacce interfaces and compare them. But since I'm not the one writing this or the corresponding userspace libs, I'll stop advocating vfio-mdev next time and focus on the implementation details :) Thanks, Jean > + > +VFIO manages resource of a hardware as a "virtual device". If a device need to > +serve a separated application. It must isolate the resource as a separate > +virtual device. And the life cycle of the application and virtual device are > +unnecessary unrelated. And most concepts, such as bus, driver, probe and > +so on, to make it as a "device" is unnecessary either. And the logic added to > +VFIO to make address sharing do no help on "creating a virtual device". > + > +IB creates a "verbs" standard for sharing memory region to another remote > +entity. Most of these verbs are to make memory region between entities to be > +synchronized. This is not what accelerator need. Accelerator is in the same > +memory system with the CPU. It refers to the same memory system among CPU and > +devices. So the local memory terms/verbs are good enough for it. Extra "verbs" > +are not necessary. And its queue (like queue pair in IB) is the communication > +channel direct to the accelerator hardware. There is nothing about memory > +itself. > + > +Further, both VFIO and IB use the "pin" (get_user_page) way to lock local > +memory in place. This is flexible. But it can cause other problems. For > +example, if the user process fork a child process. The COW procedure may make > +the parent process lost its pages which are sharing with the device. These may > +be fixed in the future. But is not going to be easy. (There is a discussion > +about this on Linux Plumbers Conference 2018 [2]) > + > +So we choose to build the solution directly on top of IOMMU interface. IOMMU > +is the essential way for device and process to share their page mapping from > +the hardware perspective. It will be safe to create a software solution on > +this assumption. Uacce manages the IOMMU interface for the accelerator > +device, so the device driver can export some of the resources to the user > +space. Uacce than can make sure the device and the process have the same > +address space. > + > + > +References > +========== > +.. [1] http://jpbrucker.net/sva/ > +.. [2] https://lwn.net/Articles/774411/ > -- > 2.7.4 >
Hi Kenneth, On Thu, Oct 17, 2019 at 03:49:07PM +0800, Kenneth Lee wrote: > Dear Jean, > > Please let me answer your question about why we build another subsystem > other than use vfio-mdev. > > I think you might remember that we did build WarpDrive on top of > vfio-mdev from the very beginning. Right thanks for reminding me, I had forgotten about the first RFCs. > Both RFCv1, Share Parent IOMMU mdev, and the RFCv2, Share Domain mdev, > are based on mdev. We got many comments and we finally felt we could not > solve all of them if we continued the mdev directory. > > I think the key problem here is that mdev is a virtual *device*. So, > > 1. As you have said, this creates more logic which is useless for > accelerator. For example, it gives the user the full control of DMA > and irq, it replays the dma mapping for new attach device and it > create VFIO IOMMU drivers... These are necessary to simulate a raw > device to a virtual machine. But it is not necessary to an > accelerator. > > 2. You are forced to separate the resource to a device before use it. > And if the user process crash, we need extra facility to put it back > to the resource pool. I don't understand the difference between vfio-mdev and uacce in this context. An example may help. If you want to give direct access of a bit of hardware to userspace, you necessarily need to isolate any resource associated to that partition from other processes and from the kernel. Namely create a DMA address space (SVA or AUXD), allocate an MMIO frame and an interrupt (although IRQs are still handled by the kernel and could be shared). And then you need to release those resources back into the pool when the process exits or crashes. > 3. Though Alex Williamson argues that vfio is not just used for > virtualisation. But it is indeed used only by virtualisation for the > time being. There is DPDK, that implements userspace drivers for net and crypto using vfio-pci, and will likely gain support for vfio-mdev soon. However similarly to Qemu they are self-contained and can easily deal with the fork problem, unlike a decompression library for example, that could be included by any application. In any case I agree with your point 1. that a simpler user interface might be beneficial. Perhaps DPDK could support uacce as well later. Thanks, Jean > And Jerome Glisse （also from Redhat) said he could > accept some problem from a virtual machine because we can constrain > the behavior of virtual machine program (such as qemu) . But he could > not accept that happened in an general application. For example, if > you pin the memory in a process and then fork, you may lost the > physical page due to COW. We can solve the problem in uacce by > letting go the shared pages in the child. > > So we think we should not continue the mdev direction for uacce. Even we > can merge the logic together. It we become a burden for both vfio and > uacce. Both of them make use of IOMMU, so of course they will have some > code similar. But they go to different direction. VFIO is trying to > export virtual DMA and irq to the user space. While uacce is trying to > let the accelerator share address with the process itself. Many tricks > can be made between the final user interface and the address translation > framework. Merge them together will not be good to both of them. > > Hope this answer some of your question. > > Cheers:)
diff --git a/Documentation/misc-devices/uacce.rst b/Documentation/misc-devices/uacce.rst new file mode 100644 index 0000000..05c1e09 --- /dev/null +++ b/Documentation/misc-devices/uacce.rst @@ -0,0 +1,297 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Introduction of Uacce +========================= + +Uacce (Unified/User-space-access-intended Accelerator Framework) targets to +provide Shared Virtual Addressing (SVA) between accelerators and processes. +So accelerator can access any data structure of the main cpu. +This differs from the data sharing between cpu and io device, which share +data content rather than address. +Because of the unified address, hardware and user space of process can +share the same virtual address in the communication. +Uacce takes the hardware accelerator as a heterogeneous processor, while +IOMMU share the same CPU page tables and as a result the same translation +from va to pa. + + __________________________ __________________________ + | | | | + | User application (CPU) | | Hardware Accelerator | + |__________________________| |__________________________| + + | | + | va | va + V V + __________ __________ + | | | | + | MMU | | IOMMU | + |__________| |__________| + | | + | | + V pa V pa + _______________________________________ + | | + | Memory | + |_______________________________________| + + + +Architecture +------------ + +Uacce is the kernel module, taking charge of iommu and address sharing. +The user drivers and libraries are called WarpDrive. + +A virtual concept, queue, is used for the communication. It provides a +FIFO-like interface. And it maintains a unified address space between the +application and all involved hardware. + + ___________________ ________________ + | | user API | | + | WarpDrive library | ------------> | user driver | + |___________________| |________________| + | | + | | + | queue fd | + | | + | | + v | + ___________________ _________ | + | | | | | mmap memory + | Other framework | | uacce | | r/w interface + | crypto/nic/others | |_________| | + |___________________| | + | | | + | register | register | + | | | + | | | + | _________________ __________ | + | | | | | | + ------------- | Device Driver | | IOMMU | | + |_________________| |__________| | + | | + | V + | ___________________ + | | | + -------------------------- | Device(Hardware) | + |___________________| + + +How does it work +================ + +Uacce uses mmap and IOMMU to play the trick. + +Uacce create a chrdev for every device registered to it. New queue is +created when user application open the chrdev. The file descriptor is used +as the user handle of the queue. +The accelerator device present itself as an Uacce object, which exports as +chrdev to the user space. The user application communicates with the +hardware by ioctl (as control path) or share memory (as data path). + +The control path to the hardware is via file operation, while data path is +via mmap space of the queue fd. + +The queue file address space: + +enum uacce_qfrt { + UACCE_QFRT_MMIO = 0, /* device mmio region */ + UACCE_QFRT_DKO = 1, /* device kernel-only region */ + UACCE_QFRT_DUS = 2, /* device user share region */ + UACCE_QFRT_SS = 3, /* static shared memory (for non-sva devices) */ + UACCE_QFRT_MAX = 16, +}; + +All regions are optional and differ from device type to type. The +communication protocol is wrapped by the user driver. + +The device mmio region is mapped to the hardware mmio space. It is generally +used for doorbell or other notification to the hardware. It is not fast enough +as data channel. + +The device kernel-only region is necessary only if the device IOMMU has no +PASID support or it cannot send kernel-only address request. In this case, if +kernel need to share memory with the device, kernel has to share iova address +space with the user process via mmap, to prevent iova conflict. + +The device user share region is used for share data buffer between user process +and device. It can be merged into other regions. But a separated region can help +on device state management. For example, the device can be started when this +region is mapped. + +The static share virtual memory region is used for share data buffer with the +device and can be shared among queues / devices. +Its size is set according to the application requirement. + + +The user API +------------ + +We adopt a polling style interface in the user space: :: + + int wd_request_queue(struct wd_queue *q); + void wd_release_queue(struct wd_queue *q); + int wd_send(struct wd_queue *q, void *req); + int wd_recv(struct wd_queue *q, void **req); + int wd_recv_sync(struct wd_queue *q, void **req); + void wd_flush(struct wd_queue *q); + +wd_recv_sync() is a wrapper to its non-sync version. It will trap into +kernel and wait until the queue become available. + +If the queue do not support SVA/SVM. The following helper functions +can be used to create Static Virtual Share Memory: :: + + void *wd_reserve_memory(struct wd_queue *q, size_t size); + int wd_share_reserved_memory(struct wd_queue *q, + struct wd_queue *target_q); + +The user API is not mandatory. It is simply a suggestion and hint what the +kernel interface is supposed to be. + + +The user driver +--------------- + +The queue file mmap space will need a user driver to wrap the communication +protocol. Uacce provides some attributes in sysfs for the user driver to +match the right accelerator accordingly. +More details in Documentation/ABI/testing/sysfs-driver-uacce. + + +The Uacce register API +----------------------- +The register API is defined in uacce.h. + +struct uacce_interface { + char name[32]; + unsigned int flags; + struct uacce_ops *ops; +}; + +According to the IOMMU capability, uacce_interface flags can be: + +UACCE_DEV_SVA (0x1) + Support shared virtual address + +UACCE_DEV_SHARE_DOMAIN (0) + This is used for device which does not support pasid. + +struct uacce_device *uacce_register(struct device *parent, + struct uacce_interface *interface); +void uacce_unregister(struct uacce_device *uacce); + +uacce_register results can be: +a. If uacce module is not compiled, ERR_PTR(-ENODEV) +b. Succeed with the desired flags +c. Succeed with the negotiated flags, for example + uacce_interface.flags = UACCE_DEV_SVA but uacce->flags = ~UACCE_DEV_SVA +So user driver need check return value as well as the negotiated uacce->flags. + + +The Memory Sharing Model +------------------------ +The perfect form of a Uacce device is to support SVM/SVA. We built this upon +Jean Philippe Brucker's SVA patches. [1] + +If the hardware support UACCE_DEV_SVA, the user process's page table is +shared to the opened queue. So the device can access any address in the +process address space. +And it can raise a page fault if the physical page is not available yet. +It can also access the address in the kernel space, which is referred by +another page table particular to the kernel. Most of IOMMU implementation can +handle this by a tag on the address request of the device. For example, ARM +SMMU uses SSV bit to indicate that the address request is for kernel or user +space. +Queue file regions can be used: +UACCE_QFRT_MMIO: device mmio region (map to user) +UACCE_QFRT_DUS: device user share (map to dev and user) + +If the device does not support UACCE_DEV_SVA, Uacce allow only one process at +the same time. DMA API cannot be used as well, since Uacce will create an +unmanaged iommu_domain for the device. +Queue file regions can be used: +UACCE_QFRT_MMIO: device mmio region (map to user) +UACCE_QFRT_DKO: device kernel-only (map to dev, no user) +UACCE_QFRT_DUS: device user share (map to dev and user) +UACCE_QFRT_SS: static shared memory (map to devs and user) + + +The Fork Scenario +================= +For a process with allocated queues and shared memory, what happen if it forks +a child? + +The fd of the queue will be duplicated on folk, so the child can send request +to the same queue as its parent. But the requests which is sent from processes +except for the one who opens the queue will be blocked. + +It is recommended to add O_CLOEXEC to the queue file. + +The queue mmap space has a VM_DONTCOPY in its VMA. So the child will lose all +those VMAs. + +This is a reason why Uacce does not adopt the mode used in VFIO and +InfiniBand. Both solutions can set any user pointer for hardware sharing. +But they cannot support fork when the dma is in process. Or the +"Copy-On-Write" procedure will make the parent process lost its physical +pages. + + +Difference to the VFIO and IB framework +--------------------------------------- +The essential function of Uacce is to let the device access the user +address directly. There are many device drivers doing the same in the kernel. +And both VFIO and IB can provide similar functions in framework level. + +But Uacce has a different goal: "share address space". It is +not taken the request to the accelerator as an enclosure data structure. It +takes the accelerator as another thread of the same process. So the +accelerator can refer to any address used by the process. + +Both VFIO and IB are taken this as "memory sharing", not "address sharing". +They care more on sharing the block of memory. But if there is an address +stored in the block and referring to another memory region. The address may +not be valid. + +By adding more constraints to the VFIO and IB framework, in some sense, we may +achieve a similar goal. But we gave it up finally. Both VFIO and IB have extra +assumption which is unnecessary to Uacce. They may hurt each other if we +try to merge them together. + +VFIO manages resource of a hardware as a "virtual device". If a device need to +serve a separated application. It must isolate the resource as a separate +virtual device. And the life cycle of the application and virtual device are +unnecessary unrelated. And most concepts, such as bus, driver, probe and +so on, to make it as a "device" is unnecessary either. And the logic added to +VFIO to make address sharing do no help on "creating a virtual device". + +IB creates a "verbs" standard for sharing memory region to another remote +entity. Most of these verbs are to make memory region between entities to be +synchronized. This is not what accelerator need. Accelerator is in the same +memory system with the CPU. It refers to the same memory system among CPU and +devices. So the local memory terms/verbs are good enough for it. Extra "verbs" +are not necessary. And its queue (like queue pair in IB) is the communication +channel direct to the accelerator hardware. There is nothing about memory +itself. + +Further, both VFIO and IB use the "pin" (get_user_page) way to lock local +memory in place. This is flexible. But it can cause other problems. For +example, if the user process fork a child process. The COW procedure may make +the parent process lost its pages which are sharing with the device. These may +be fixed in the future. But is not going to be easy. (There is a discussion +about this on Linux Plumbers Conference 2018 [2]) + +So we choose to build the solution directly on top of IOMMU interface. IOMMU +is the essential way for device and process to share their page mapping from +the hardware perspective. It will be safe to create a software solution on +this assumption. Uacce manages the IOMMU interface for the accelerator +device, so the device driver can export some of the resources to the user +space. Uacce than can make sure the device and the process have the same +address space. + + +References +========== +.. [1] http://jpbrucker.net/sva/ +.. [2] https://lwn.net/Articles/774411/