| Message ID | 20161017152623.7649-1-punit.agrawal@arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, 17 Oct 2016 16:26:23 +0100 Punit Agrawal <punit.agrawal@arm.com> wrote: > The dma mapping api howto gives the impression that using the > dma_set_mask_and_coherent (and related DMA APIs) will cause the kernel > to check all the components in the path from the device to memory for > addressing restrictions. In systems with address translations between > the device and memory (e.g., when using IOMMU), this implies that a > successful call to set set dma mask has checked the addressing > constraints of the intermediaries as well. > > For the IOMMU drivers in the tree, the check is actually performed while > allocating the DMA buffer rather than when the DMA mask is > configured. For MMUs that do not support the full device addressing > capability, the allocations are made from a reduced address space. > > Update the documentation to clarify that even though the call to > dma_set_mask_and_coherent succeeds, it may not be possible to use the > full addressing capability of the device. OK, so I guess I can buy this. But... > Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> > Cc: Jonathan Corbet <corbet@lwn.net> > --- > Documentation/DMA-API-HOWTO.txt | 39 +++++++++++++++++++++++---------------- > 1 file changed, 23 insertions(+), 16 deletions(-) > > diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt > index 979228b..240d1ee 100644 > --- a/Documentation/DMA-API-HOWTO.txt > +++ b/Documentation/DMA-API-HOWTO.txt > @@ -159,39 +159,46 @@ support 64-bit addressing (DAC) for all transactions. And at least > one platform (SGI SN2) requires 64-bit consistent allocations to > operate correctly when the IO bus is in PCI-X mode. > > -For correct operation, you must interrogate the kernel in your device > -probe routine to see if the DMA controller on the machine can properly > -support the DMA addressing limitation your device has. It is good > +For correct operation, you must inform the kernel in your device probe > +routine to see if the DMA controller on the machine can properly > +support the DMA addressing capabilities your device has. It is good Here it's still saying "to see if the DMA controller on the machine can properly support the DMA addressing capabilities your device has". So you've not really changed the sense of this sentence here. If I understand things correctly, the calls in question are storing the device's limitations; they will only fail if the kernel is entirely unable to work within the indicated range, right? I don't think there's ever been any guarantee that the system as a whole could use the entire range that is addressable by the device. I have no objection to making that more clear, but let's actually make it more clear by saying what the functions are actually doing. Make sense, or am I missing something here? Thanks, jon
On Fri, Oct 21, 2016 at 03:09:16PM -0600, Jonathan Corbet wrote: > On Mon, 17 Oct 2016 16:26:23 +0100 > Punit Agrawal <punit.agrawal@arm.com> wrote: > > > The dma mapping api howto gives the impression that using the > > dma_set_mask_and_coherent (and related DMA APIs) will cause the kernel > > to check all the components in the path from the device to memory for > > addressing restrictions. In systems with address translations between > > the device and memory (e.g., when using IOMMU), this implies that a > > successful call to set set dma mask has checked the addressing > > constraints of the intermediaries as well. This is basically true when you have DMA controllers in the path from device to memory. But it is not true for IOMMUs, because IOMMU drivers are consumers of the dma-masks, they don't really restrict them. An IOMMU driver knows the limitations of IOMMU hardware and counts that in when allocating an address for a dma-buffer. So long story short: Any IOMMU restrictions in address space size don't need to be represented in the dma-mask for a device. Joerg
Joerg Roedel <joro@8bytes.org> writes: > On Fri, Oct 21, 2016 at 03:09:16PM -0600, Jonathan Corbet wrote: >> On Mon, 17 Oct 2016 16:26:23 +0100 >> Punit Agrawal <punit.agrawal@arm.com> wrote: >> >> > The dma mapping api howto gives the impression that using the >> > dma_set_mask_and_coherent (and related DMA APIs) will cause the kernel >> > to check all the components in the path from the device to memory for >> > addressing restrictions. In systems with address translations between >> > the device and memory (e.g., when using IOMMU), this implies that a >> > successful call to set set dma mask has checked the addressing >> > constraints of the intermediaries as well. > > This is basically true when you have DMA controllers in the path from > device to memory. But it is not true for IOMMUs, because IOMMU drivers > are consumers of the dma-masks, they don't really restrict them. An > IOMMU driver knows the limitations of IOMMU hardware and counts that in > when allocating an address for a dma-buffer. Yes, that's what I'd found looking at the IOMMU drivers in the tree. > > So long story short: Any IOMMU restrictions in address space size don't > need to be represented in the dma-mask for a device. That was another rabbit hole I'd spend some time in - whether IOMMU restrictions need to be factored into the dma_mask for devices. As size(dma_mask) > size(iommu supported address size) still works, I came to the conclusion that the documentation can maybe help clarify this. This patch is an attempt to update the documentation to inform the user that even if the dma_set_mask call succeeds, the system may still not use the entire device address space. Thanks for clarifying a few of my doubts. Punit > > > > Joerg
Jonathan Corbet <corbet@lwn.net> writes: > On Mon, 17 Oct 2016 16:26:23 +0100 > Punit Agrawal <punit.agrawal@arm.com> wrote: > >> The dma mapping api howto gives the impression that using the >> dma_set_mask_and_coherent (and related DMA APIs) will cause the kernel >> to check all the components in the path from the device to memory for >> addressing restrictions. In systems with address translations between >> the device and memory (e.g., when using IOMMU), this implies that a >> successful call to set set dma mask has checked the addressing >> constraints of the intermediaries as well. >> >> For the IOMMU drivers in the tree, the check is actually performed while >> allocating the DMA buffer rather than when the DMA mask is >> configured. For MMUs that do not support the full device addressing >> capability, the allocations are made from a reduced address space. >> >> Update the documentation to clarify that even though the call to >> dma_set_mask_and_coherent succeeds, it may not be possible to use the >> full addressing capability of the device. > > OK, so I guess I can buy this. But... > >> Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> >> Cc: Jonathan Corbet <corbet@lwn.net> >> --- >> Documentation/DMA-API-HOWTO.txt | 39 +++++++++++++++++++++++---------------- >> 1 file changed, 23 insertions(+), 16 deletions(-) >> >> diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt >> index 979228b..240d1ee 100644 >> --- a/Documentation/DMA-API-HOWTO.txt >> +++ b/Documentation/DMA-API-HOWTO.txt >> @@ -159,39 +159,46 @@ support 64-bit addressing (DAC) for all transactions. And at least >> one platform (SGI SN2) requires 64-bit consistent allocations to >> operate correctly when the IO bus is in PCI-X mode. >> >> -For correct operation, you must interrogate the kernel in your device >> -probe routine to see if the DMA controller on the machine can properly >> -support the DMA addressing limitation your device has. It is good >> +For correct operation, you must inform the kernel in your device probe >> +routine to see if the DMA controller on the machine can properly >> +support the DMA addressing capabilities your device has. It is good > > Here it's still saying "to see if the DMA controller on the machine can > properly support the DMA addressing capabilities your device has". So > you've not really changed the sense of this sentence here. You're right - the changes don't go far enough. How about dropping the bit so that it now reads - "For correct operation, in your device probe routine, you must inform the DMA addressing capabilities of your device to the kernel." > > If I understand things correctly, the calls in question are storing the > device's limitations; they will only fail if the kernel is entirely > unable to work within the indicated range, right? I don't think there's > ever been any guarantee that the system as a whole could use the entire > range that is addressable by the device. That matches my understanding as well. I was just trying to make it more explicit with this patch. > I have no objection to making > that more clear, but let's actually make it more clear by saying what the > functions are actually doing. > > Make sense, or am I missing something here? Let me know if the change I suggest is an improvement and I'll send out a v2. Thanks, Punit > > Thanks, > > jon
On Friday, October 21, 2016 3:09:16 PM CEST Jonathan Corbet wrote: > On Mon, 17 Oct 2016 16:26:23 +0100 > Punit Agrawal <punit.agrawal@arm.com> wrote: > > > The dma mapping api howto gives the impression that using the > > dma_set_mask_and_coherent (and related DMA APIs) will cause the kernel > > to check all the components in the path from the device to memory for > > addressing restrictions. In systems with address translations between > > the device and memory (e.g., when using IOMMU), this implies that a > > successful call to set set dma mask has checked the addressing > > constraints of the intermediaries as well. > > > > For the IOMMU drivers in the tree, the check is actually performed while > > allocating the DMA buffer rather than when the DMA mask is > > configured. For MMUs that do not support the full device addressing > > capability, the allocations are made from a reduced address space. > > > > Update the documentation to clarify that even though the call to > > dma_set_mask_and_coherent succeeds, it may not be possible to use the > > full addressing capability of the device. > > OK, so I guess I can buy this. But... > > > Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> > > Cc: Jonathan Corbet <corbet@lwn.net> > > --- > > Documentation/DMA-API-HOWTO.txt | 39 +++++++++++++++++++++++---------------- > > 1 file changed, 23 insertions(+), 16 deletions(-) > > > > diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt > > index 979228b..240d1ee 100644 > > --- a/Documentation/DMA-API-HOWTO.txt > > +++ b/Documentation/DMA-API-HOWTO.txt > > @@ -159,39 +159,46 @@ support 64-bit addressing (DAC) for all transactions. And at least > > one platform (SGI SN2) requires 64-bit consistent allocations to > > operate correctly when the IO bus is in PCI-X mode. > > > > -For correct operation, you must interrogate the kernel in your device > > -probe routine to see if the DMA controller on the machine can properly > > -support the DMA addressing limitation your device has. It is good > > +For correct operation, you must inform the kernel in your device probe > > +routine to see if the DMA controller on the machine can properly > > +support the DMA addressing capabilities your device has. It is good > > Here it's still saying "to see if the DMA controller on the machine can > properly support the DMA addressing capabilities your device has". So > you've not really changed the sense of this sentence here. > > If I understand things correctly, the calls in question are storing the > device's limitations; they will only fail if the kernel is entirely > unable to work within the indicated range, right? I don't think there's > ever been any guarantee that the system as a whole could use the entire > range that is addressable by the device. I have no objection to making > that more clear, but let's actually make it more clear by saying what the > functions are actually doing. > > Make sense, or am I missing something here? The call is a two-way interface, and the existing text tries to convey that already: The device tells the kernel whether it is limited (< 32 bit mask) or if it can support extended addresses (> 32 bit mask), or just handles the default 32bit mask, and the kernel should come back saying whether that mask allows a correct operation of the device on the given platform, as well as set it up correctly that way. What exactly happens in dma_set_mask() and the related interfaces is highly platform specific, including: - if the mask is smaller than the smallest memory zone and the swiotlb bounce buffers (if any) don't fit inside it, it has to fail - if the device claims to support larger mask, but the bus it connects to does not (e.g. a 32-bit PCI host), it may also fail (or succeed if there is no RAM outside of the intersection of the two masks) - if the mask is large enough to cover all RAM, we can bypass the IOMMU and use a direct mapping - if swiotlb is enabled or an IOMMU is present, any mask that includes the bounce buffer area (or the virtual address space of the IOMMU) should succeed. Arnd
diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt index 979228b..240d1ee 100644 --- a/Documentation/DMA-API-HOWTO.txt +++ b/Documentation/DMA-API-HOWTO.txt @@ -159,39 +159,46 @@ support 64-bit addressing (DAC) for all transactions. And at least one platform (SGI SN2) requires 64-bit consistent allocations to operate correctly when the IO bus is in PCI-X mode. -For correct operation, you must interrogate the kernel in your device -probe routine to see if the DMA controller on the machine can properly -support the DMA addressing limitation your device has. It is good +For correct operation, you must inform the kernel in your device probe +routine to see if the DMA controller on the machine can properly +support the DMA addressing capabilities your device has. It is good style to do this even if your device holds the default setting, because this shows that you did think about these issues wrt. your device. -The query is performed via a call to dma_set_mask_and_coherent(): +The call to inform the kernel is performed via a call to +dma_set_mask_and_coherent(): int dma_set_mask_and_coherent(struct device *dev, u64 mask); -which will query the mask for both streaming and coherent APIs together. -If you have some special requirements, then the following two separate -queries can be used instead: +which will set the mask for both streaming and coherent APIs together. +If there are some special requirements, then the following two +separate functions can be used instead: - The query for streaming mappings is performed via a call to - dma_set_mask(): + The configuration for streaming mappings is performed via a + call to dma_set_mask(): int dma_set_mask(struct device *dev, u64 mask); - The query for consistent allocations is performed via a call - to dma_set_coherent_mask(): + The configuration for consistent allocations is performed via + a call to dma_set_coherent_mask(): int dma_set_coherent_mask(struct device *dev, u64 mask); Here, dev is a pointer to the device struct of your device, and mask is a bit mask describing which bits of an address your device supports. It returns zero if your card can perform DMA properly on -the machine given the address mask you provided. In general, the -device struct of your device is embedded in the bus-specific device -struct of your device. For example, &pdev->dev is a pointer to the -device struct of a PCI device (pdev is a pointer to the PCI device -struct of your device). +the machine given the address mask you provided. Subsequent to +calling the above apis, DMA allocations will be made from address +space that conforms to the mask. The DMA allocation space maybe +further restricted if devices along the patch to memory have stricter +addressing requirements than the device performing the DMA, e.g., +IOMMU. + +In general, the device struct of your device is embedded in the +bus-specific device struct of your device. For example, &pdev->dev is +a pointer to the device struct of a PCI device (pdev is a pointer to +the PCI device struct of your device). If it returns non-zero, your device cannot perform DMA properly on this platform, and attempting to do so will result in undefined
The dma mapping api howto gives the impression that using the dma_set_mask_and_coherent (and related DMA APIs) will cause the kernel to check all the components in the path from the device to memory for addressing restrictions. In systems with address translations between the device and memory (e.g., when using IOMMU), this implies that a successful call to set set dma mask has checked the addressing constraints of the intermediaries as well. For the IOMMU drivers in the tree, the check is actually performed while allocating the DMA buffer rather than when the DMA mask is configured. For MMUs that do not support the full device addressing capability, the allocations are made from a reduced address space. Update the documentation to clarify that even though the call to dma_set_mask_and_coherent succeeds, it may not be possible to use the full addressing capability of the device. Signed-off-by: Punit Agrawal <punit.agrawal@arm.com> Cc: Jonathan Corbet <corbet@lwn.net> --- Documentation/DMA-API-HOWTO.txt | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-)