| Message ID | 20170817141449.23958-4-wsa+renesas@sang-engineering.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Em Thu, 17 Aug 2017 16:14:46 +0200 Wolfram Sang <wsa+renesas@sang-engineering.com> escreveu: > Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com> > --- > Documentation/i2c/DMA-considerations | 50 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 50 insertions(+) > create mode 100644 Documentation/i2c/DMA-considerations > > diff --git a/Documentation/i2c/DMA-considerations b/Documentation/i2c/DMA-considerations > new file mode 100644 > index 00000000000000..a4b4a107102452 > --- /dev/null > +++ b/Documentation/i2c/DMA-considerations > @@ -0,0 +1,50 @@ > +Linux I2C and DMA > +----------------- I would use, instead: ================= Linux I2C and DMA ================= As this is the way we're starting document titles, after converted to ReST. So, better to have it already using the right format, as one day someone will convert i2c documentation to ReST. So, it would be really cool if this document could be just renamed without needing to patch it during such conversion :-) There are also a couple of things here that Sphinx would complain. So, it could be worth to rename it to *.rst, while you're writing it, and see what: make htmldocs will complain and how it will look in html. > + > +Given that I2C is a low-speed bus where largely small messages are transferred, > +it is not considered a prime user of DMA access. At this time of writing, only > +10% of I2C bus master drivers have DMA support implemented. Are you sure about that? I'd say that, on media, at least half of the drivers use DMA for I2C bus access, as the I2C bus is on a remote board that talks with CPU via USB, using DMA, and all communication with USB should be DMA-safe. I guess what you really wanted to say on most of this section is about SoC (and other CPUs) where the I2C bus master is is at the mainboard, and not on some peripheral. > And the vast > +majority of transactions are so small that setting up DMA for it will likely > +add more overhead than a plain PIO transfer. > + > +Therefore, it is *not* mandatory that the buffer of an I2C message is DMA safe. Again, that may not be true on media boards. The code that implements the I2C transfers there, on most boards, have to be DMA safe, as it won't otherwise send/receive commands from the chips that are after the USB bridge. > +It does not seem reasonable to apply additional burdens when the feature is so > +rarely used. However, it is recommended to use a DMA-safe buffer if your > +message size is likely applicable for DMA. Most drivers have this threshold > +around 8 bytes (as of today, this is mostly an educated guess, however). For > +any message of 16 byte or larger, it is probably a really good idea. > + > +If you use such a buffer in a i2c_msg, set the I2C_M_DMA_SAFE flag with it. > +Then, the I2C core and drivers know they can safely operate DMA on it. Note > +that setting this flag makes only sense in kernel space. User space data is > +copied into kernel space anyhow. The I2C core makes sure the destination > +buffers in kernel space are always DMA capable. > + > +FIXME: Need to implement i2c_master_{send|receive}_dma and proper buffers for i2c_smbus_xfer_emulated. > + > +Drivers wishing to implement DMA can use helper functions from the I2C core. > +One gives you a DMA-safe buffer for a given i2c_msg as long as a certain > +threshold is met. > + > + dma_buf = i2c_get_dma_safe_msg_buf(msg, threshold_in_byte); I'm concerned about the new bits added by this call. Right now, USB drivers just use kalloc() for transfer buffers used to send and receive URB control messages for both setting the main device and for I2C messages. Before this changeset, buffers allocated this way are DMA save and have been working for years. When you add some flags that would make the I2C subsystem aware that a buffer is now DMA safe, I guess you could be breaking those drivers, as a DMA safe buffer will now be considered as DMA-unsafe. So, you'll likely need to patch all media USB drivers to fix it, or come up with some other solution. > + > +If a buffer is returned, it either msg->buf for the I2C_M_DMA_SAFE case or a > +bounce buffer. But you don't need to care about that detail. If NULL is > +returned, the threshold was not met or a bounce buffer could not be allocated. > +Fall back to PIO in that case. > + > +In any case, a buffer obtained from above needs to be released. It ensures data > +is copied back to the message and a potentially used bounce buffer is freed. > + > + i2c_release_dma_safe_msg_buf(msg, dma_buf); > + > +The bounce buffer handling from the core is generic and simple. It will always > +allocate a new bounce buffer. If you want a more sophisticated handling (e.g. > +reusing pre-allocated buffers), you are free to implement your own. > + > +Please also check the in-kernel documentation for details. The i2c-sh_mobile > +driver can be used as a reference example how to use the above helpers. > + > +Final note: If you plan to use DMA with I2C (or with anything else, actually) > +make sure you have CONFIG_DMA_API_DEBUG enabled during development. It can help > +you find various issues which can be complex to debug otherwise. Thanks, Mauro
Hi Mauro, thanks for your comments. Much appreciated! > There are also a couple of things here that Sphinx would complain. > So, it could be worth to rename it to *.rst, while you're writing > it, and see what: > make htmldocs > will complain and how it will look in html. OK, I'll check that. > > +Given that I2C is a low-speed bus where largely small messages are transferred, > > +it is not considered a prime user of DMA access. At this time of writing, only > > +10% of I2C bus master drivers have DMA support implemented. > > Are you sure about that? I'd say that, on media, at least half of the > drivers use DMA for I2C bus access, as the I2C bus is on a remote > board that talks with CPU via USB, using DMA, and all communication > with USB should be DMA-safe. Well, the DMA-safe requirement comes then from the USB subsystem, doesn't it? Or do you really use DMA on the remote board to transfer data via I2C to an I2C client? > I guess what you really wanted to say on most of this section is > about SoC (and other CPUs) where the I2C bus master is is at the > mainboard, and not on some peripheral. I might be biased to that, yes. So, it is good talking about it. > > And the vast > > +majority of transactions are so small that setting up DMA for it will likely > > +add more overhead than a plain PIO transfer. > > + > > +Therefore, it is *not* mandatory that the buffer of an I2C message is DMA safe. > > Again, that may not be true on media boards. The code that implements the > I2C transfers there, on most boards, have to be DMA safe, as it won't > otherwise send/receive commands from the chips that are after the USB > bridge. That still sounds to me like the DMA-safe requirement comes from USB (which is fine, of course.). In any case, a sentence like "Other subsystem you might use for bridging I2C might impose other DMA requirements" sounds like to nice to have. > > +Drivers wishing to implement DMA can use helper functions from the I2C core. > > +One gives you a DMA-safe buffer for a given i2c_msg as long as a certain > > +threshold is met. > > + > > + dma_buf = i2c_get_dma_safe_msg_buf(msg, threshold_in_byte); > > I'm concerned about the new bits added by this call. Right now, > USB drivers just use kalloc() for transfer buffers used to send and > receive URB control messages for both setting the main device and > for I2C messages. Before this changeset, buffers allocated this > way are DMA save and have been working for years. Can you give me a pointer to a driver doing this? I glimpsed around in drivers/media/usb and found that most drivers are having their i2c_msg buffers on the stack. Which is clearly not DMA-safe. > When you add some flags that would make the I2C subsystem aware > that a buffer is now DMA safe, I guess you could be breaking > those drivers, as a DMA safe buffer will now be considered as > DMA-unsafe. Well, this flag is only relevant for i2c master drivers wishing to do DMA. So, grepping in the above directory grep dma $(grep -rl i2c_add_adapter *) only gives one driver which is irrelevant because the i2c master it registers is not doing any DMA? Am I missing something? We are clearly not aligned yet... Regards, Wolfram
Em Fri, 8 Sep 2017 10:56:40 +0200 Wolfram Sang <wsa@the-dreams.de> escreveu: > Hi Mauro, > > thanks for your comments. Much appreciated! > > > There are also a couple of things here that Sphinx would complain. > > So, it could be worth to rename it to *.rst, while you're writing > > it, and see what: > > make htmldocs > > will complain and how it will look in html. > > OK, I'll check that. Ok. > > > +Given that I2C is a low-speed bus where largely small messages are transferred, > > > +it is not considered a prime user of DMA access. At this time of writing, only > > > +10% of I2C bus master drivers have DMA support implemented. > > > > Are you sure about that? I'd say that, on media, at least half of the > > drivers use DMA for I2C bus access, as the I2C bus is on a remote > > board that talks with CPU via USB, using DMA, and all communication > > with USB should be DMA-safe. > > Well, the DMA-safe requirement comes then from the USB subsystem, > doesn't it? Yes, but the statistics that 10% of I2C bus master drivers are DMA-safe is not true, if you take those into account ;-) Perhaps you could write it as (or something similar): At this time of writing, only +10% of I2C bus master drivers for non-remote boards have DMA support implemented. > Or do you really use DMA on the remote board to transfer > data via I2C to an I2C client? > > > I guess what you really wanted to say on most of this section is > > about SoC (and other CPUs) where the I2C bus master is is at the > > mainboard, and not on some peripheral. > > I might be biased to that, yes. So, it is good talking about it. > > > > And the vast > > > +majority of transactions are so small that setting up DMA for it will likely > > > +add more overhead than a plain PIO transfer. > > > + > > > +Therefore, it is *not* mandatory that the buffer of an I2C message is DMA safe. > > > > Again, that may not be true on media boards. The code that implements the > > I2C transfers there, on most boards, have to be DMA safe, as it won't > > otherwise send/receive commands from the chips that are after the USB > > bridge. > > That still sounds to me like the DMA-safe requirement comes from USB > (which is fine, of course.). In any case, a sentence like "Other > subsystem you might use for bridging I2C might impose other DMA > requirements" sounds like to nice to have. Agreed. > > > > +Drivers wishing to implement DMA can use helper functions from the I2C core. > > > +One gives you a DMA-safe buffer for a given i2c_msg as long as a certain > > > +threshold is met. > > > + > > > + dma_buf = i2c_get_dma_safe_msg_buf(msg, threshold_in_byte); > > > > I'm concerned about the new bits added by this call. Right now, > > USB drivers just use kalloc() for transfer buffers used to send and > > receive URB control messages for both setting the main device and > > for I2C messages. Before this changeset, buffers allocated this > > way are DMA save and have been working for years. > > Can you give me a pointer to a driver doing this? I glimpsed around in > drivers/media/usb and found that most drivers are having their i2c_msg > buffers on the stack. Which is clearly not DMA-safe. The way it is implemented depends on the driver, and usually envolves double-buffering, e. g., on some place of the driver, a buffer that may not be DMA-save is copied into a DMA safe buffer. On most cases, like on this driver: drivers/media/usb/dvb-usb-v2/az6007.c The i2c_xfer logic (or the read/write functions) is the one responsible for double-buffering. In this specific example, the DVB usbv2 core allocates the device's "state" struct using kmalloc (it uses .size_of_priv field to know the size of the "state" buffer). On struct az6007_device_state, there is a "data" buffer with 4096 bytes. At the i2c transfer function, it retrieves and use it: struct az6007_device_state *st = d_to_priv(d) ... ret = __az6007_read(d->udev, req, value, index, st->data, length); ... ret = __az6007_write(d->udev, req, value, index, st->data, length); In the past, on lots of drivers, the i2c_xfer logic just used to assume that the I2C client driver allocated a DMA-safe buffer, as it just used to pass whatever buffer it receives directly to USB core. We did an effort to change it, as it can be messy, but I'm not sure if this is solved everywhere. The __az6007_read() and __az6007_write() indirectly do DMA (for most USB host drivers), when they call usb_control_msg(). > > > When you add some flags that would make the I2C subsystem aware > > that a buffer is now DMA safe, I guess you could be breaking > > those drivers, as a DMA safe buffer will now be considered as > > DMA-unsafe. > > Well, this flag is only relevant for i2c master drivers wishing to do > DMA. So, grepping in the above directory > > grep dma $(grep -rl i2c_add_adapter *) The usage of I2C at the media subsystem predates the I2C subsystem. at V4L2 drivers, a great effort was done to port it to fully use the I2C subsystem when it was added upstream, but there were some problems a the initial implementation of the i2c subsystem that prevented its usage for the DVB drivers. By the time such restrictions got removed, it was too late, as there were already a large amount of drivers relying on i2c "low level" functions. So, almost all DVB drivers don't use i2c_add_adapter(). Instead, the I2C clients just call directly the I2C transfer functions: drivers/media/dvb-frontends/au8522_common.c: ret = i2c_transfer(state->i2c, msg, 2); > only gives one driver which is irrelevant because the i2c master it > registers is not doing any DMA? Even on the drivers that use i2c_add_adapter(), the usage of DMA can't be get by the above grep, as the DMA usage is actually at drivers/usb. For example, the em28xx driver uses i2c_add_adapter(): $ git grep i2c_add_adapter drivers/media/usb/em28xx/ drivers/media/usb/em28xx/em28xx-i2c.c: retval = i2c_add_adapter(&dev->i2c_adap[bus]); drivers/media/usb/em28xx/em28xx-i2c.c: "%s: i2c_add_adapter failed! retval [%d]\n", And the actual data transfer happens via usb_control_msg(): $ git grep usb_control_msg drivers/media/usb/em28xx/ drivers/media/usb/em28xx/em28xx-core.c: ret = usb_control_msg(udev, pipe, req, drivers/media/usb/em28xx/em28xx-core.c: ret = usb_control_msg(udev, pipe, req, If you modify your grep function, you'll see that usb_control_msg() is DMA-dependent. Try: $ grep dma $(grep -rl usb_control_msg) drivers/usb/core/ > Am I missing something? We are clearly not aligned yet... > > Regards, > > Wolfram > Thanks, Mauro
> Yes, but the statistics that 10% of I2C bus master drivers > are DMA-safe is not true, if you take those into account ;-) > > Perhaps you could write it as (or something similar): > > At this time of writing, only +10% of I2C bus master > drivers for non-remote boards have DMA support implemented. No, this is confusing IMO. Being remote has nothing to with DMA. What has to do with DMA is that these are using USB as communication channel. And USB clearly needs DMA-safe buffers. The encapsulation needs DMA-safe buffers. So, I think the new sentence "other subsystems might impose" mentions that. Let me recap what this series is for: a) It makes clear that I2C subsystem does not require DMA-safety because for the reasons given in the textfile. If I2C is encapsulated over USB, then DMA-safety is of course required, but this comes from USB. So, those USB-I2C master drivers need to do something to make sure DMA-safety is guaranteed because i2c_msg buffers don't need to be DMA safe because of a). They could use the newly introduced features, but they don't have to. b) a flag for DMA safe i2c_msgs is added So, for messages we know to be DMA safe, we can flag that. Drivers could check that and use a bounce buffer if the flag is not set. However, this is all optional. Your drivers can still choose to not check the flag and everything will stay as before. Check patch 5 for a use case. c) helper functions for bounce buffers are added These are *helper* functions for drivers wishing to do DMA. A super simple bounce buffer support. Check patch 4 for a use case. Again, this is optional. Drivers can implement their own bounce buffer support. Or, as in your case, if you know that your buffers are good, then don't use any of this and things will keep going. This all is to allow bus master drivers to opt-in for DMA safety if they want to do DMA directly. Because currently, with a lot of i2c_msgs on stack, this is more or less working accidently. And, yes, I know I have to add this new flag to a few central places in client drivers. Otherwise, master drivers checking for DMA safety will initially have a performance regression. This is scheduled for V5 and mentioned in this series. > In the past, on lots of drivers, the i2c_xfer logic just used to assume > that the I2C client driver allocated a DMA-safe buffer, as it just used to > pass whatever buffer it receives directly to USB core. We did an effort to > change it, as it can be messy, but I'm not sure if this is solved everywhere. Good, I can imagine this being some effort. But again, this is because USB needs the DMA-safety, not I2C. AFAICS, most i2c_msgs are register accesses and thus, small messages. > The usage of I2C at the media subsystem predates the I2C subsystem. > at V4L2 drivers, a great effort was done to port it to fully use the > I2C subsystem when it was added upstream, but there were some problems > a the initial implementation of the i2c subsystem that prevented its > usage for the DVB drivers. By the time such restrictions got removed, > it was too late, as there were already a large amount of drivers relying > on i2c "low level" functions. Didn't know that, but this is good to know! Thanks. > Even on the drivers that use i2c_add_adapter(), the usage of DMA can't > be get by the above grep, as the DMA usage is actually at drivers/usb. Ok. But as said before, what works now will continue to work. This series is about clarifying that i2c_msg buffers need not to be DMA safe and offering some helpers for those bus master drivers wanting to opt-in to be sure. Clearer now? Regards, Wolfram
Em Sat, 9 Sep 2017 17:27:06 +0200 Wolfram Sang <wsa@the-dreams.de> escreveu: > > Yes, but the statistics that 10% of I2C bus master drivers > > are DMA-safe is not true, if you take those into account ;-) > > > > Perhaps you could write it as (or something similar): > > > > At this time of writing, only +10% of I2C bus master > > drivers for non-remote boards have DMA support implemented. > > No, this is confusing IMO. Being remote has nothing to with DMA. What > has to do with DMA is that these are using USB as communication channel. > And USB clearly needs DMA-safe buffers. The encapsulation needs DMA-safe > buffers. So, I think the new sentence "other subsystems might impose" > mentions that. > > Let me recap what this series is for: > > a) It makes clear that I2C subsystem does not require DMA-safety because > for the reasons given in the textfile. > > If I2C is encapsulated over USB, then DMA-safety is of course required, > but this comes from USB. So, those USB-I2C master drivers need to do > something to make sure DMA-safety is guaranteed because i2c_msg buffers > don't need to be DMA safe because of a). They could use the newly > introduced features, but they don't have to. > > b) a flag for DMA safe i2c_msgs is added > > So, for messages we know to be DMA safe, we can flag that. Drivers > could check that and use a bounce buffer if the flag is not set. > However, this is all optional. Your drivers can still choose to not > check the flag and everything will stay as before. Check patch 5 for a > use case. > > c) helper functions for bounce buffers are added > > These are *helper* functions for drivers wishing to do DMA. A super > simple bounce buffer support. Check patch 4 for a use case. Again, this > is optional. Drivers can implement their own bounce buffer support. Or, > as in your case, if you know that your buffers are good, then don't use > any of this and things will keep going. > > > This all is to allow bus master drivers to opt-in for DMA safety if they > want to do DMA directly. Because currently, with a lot of i2c_msgs on > stack, this is more or less working accidently. > > And, yes, I know I have to add this new flag to a few central places in > client drivers. Otherwise, master drivers checking for DMA safety will > initially have a performance regression. This is scheduled for V5 and > mentioned in this series. > > > In the past, on lots of drivers, the i2c_xfer logic just used to assume > > that the I2C client driver allocated a DMA-safe buffer, as it just used to > > pass whatever buffer it receives directly to USB core. We did an effort to > > change it, as it can be messy, but I'm not sure if this is solved everywhere. > > Good, I can imagine this being some effort. But again, this is because > USB needs the DMA-safety, not I2C. AFAICS, most i2c_msgs are register > accesses and thus, small messages. > > > The usage of I2C at the media subsystem predates the I2C subsystem. > > at V4L2 drivers, a great effort was done to port it to fully use the > > I2C subsystem when it was added upstream, but there were some problems > > a the initial implementation of the i2c subsystem that prevented its > > usage for the DVB drivers. By the time such restrictions got removed, > > it was too late, as there were already a large amount of drivers relying > > on i2c "low level" functions. > > Didn't know that, but this is good to know! Thanks. > > > Even on the drivers that use i2c_add_adapter(), the usage of DMA can't > > be get by the above grep, as the DMA usage is actually at drivers/usb. > > Ok. But as said before, what works now will continue to work. OK, good! > This series is about clarifying that i2c_msg buffers need not to be DMA safe > and offering some helpers for those bus master drivers wanting to opt-in > to be sure. > > Clearer now? Yeah, it is clearer for me. Thanks! Anyway, it doesn't hurt if you could improve the documentation patch to mention the above somewhow, as I want to avoid patches trying to make existing media USB drivers to use the new flag without changing the already existing DMA-safe logic there (causing double-buffering), or, even worse, making the I2C bus DMA-unsafe because someone misread this document. Regards, Mauro
Hi Mauro, > > +Linux I2C and DMA > > +----------------- > > I would use, instead: > > ================= > Linux I2C and DMA > ================= > > As this is the way we're starting document titles, after converted to > ReST. So, better to have it already using the right format, as one day I did this. > There are also a couple of things here that Sphinx would complain. The only complaint I got was WARNING: document isn't included in any toctree which makes sense because I renamed it only temporarily to *.rst > So, it could be worth to rename it to *.rst, while you're writing > it, and see what: > make htmldocs > will complain and how it will look in html. So, no complaints from Sphinx and the HTML output looks good IMO. Was there anything specific you had in mind when saying that Sphinx would complain? Regards, Wolfram
> In order to avoid that, and to place them into a box using monotonic fonts, > I usually add "::" at the preceding line, e. g.: Just in time: I added the '::' and will resubmit the new version in a minute. Thanks for the pointers!
diff --git a/Documentation/i2c/DMA-considerations b/Documentation/i2c/DMA-considerations new file mode 100644 index 00000000000000..a4b4a107102452 --- /dev/null +++ b/Documentation/i2c/DMA-considerations @@ -0,0 +1,50 @@ +Linux I2C and DMA +----------------- + +Given that I2C is a low-speed bus where largely small messages are transferred, +it is not considered a prime user of DMA access. At this time of writing, only +10% of I2C bus master drivers have DMA support implemented. And the vast +majority of transactions are so small that setting up DMA for it will likely +add more overhead than a plain PIO transfer. + +Therefore, it is *not* mandatory that the buffer of an I2C message is DMA safe. +It does not seem reasonable to apply additional burdens when the feature is so +rarely used. However, it is recommended to use a DMA-safe buffer if your +message size is likely applicable for DMA. Most drivers have this threshold +around 8 bytes (as of today, this is mostly an educated guess, however). For +any message of 16 byte or larger, it is probably a really good idea. + +If you use such a buffer in a i2c_msg, set the I2C_M_DMA_SAFE flag with it. +Then, the I2C core and drivers know they can safely operate DMA on it. Note +that setting this flag makes only sense in kernel space. User space data is +copied into kernel space anyhow. The I2C core makes sure the destination +buffers in kernel space are always DMA capable. + +FIXME: Need to implement i2c_master_{send|receive}_dma and proper buffers for i2c_smbus_xfer_emulated. + +Drivers wishing to implement DMA can use helper functions from the I2C core. +One gives you a DMA-safe buffer for a given i2c_msg as long as a certain +threshold is met. + + dma_buf = i2c_get_dma_safe_msg_buf(msg, threshold_in_byte); + +If a buffer is returned, it either msg->buf for the I2C_M_DMA_SAFE case or a +bounce buffer. But you don't need to care about that detail. If NULL is +returned, the threshold was not met or a bounce buffer could not be allocated. +Fall back to PIO in that case. + +In any case, a buffer obtained from above needs to be released. It ensures data +is copied back to the message and a potentially used bounce buffer is freed. + + i2c_release_dma_safe_msg_buf(msg, dma_buf); + +The bounce buffer handling from the core is generic and simple. It will always +allocate a new bounce buffer. If you want a more sophisticated handling (e.g. +reusing pre-allocated buffers), you are free to implement your own. + +Please also check the in-kernel documentation for details. The i2c-sh_mobile +driver can be used as a reference example how to use the above helpers. + +Final note: If you plan to use DMA with I2C (or with anything else, actually) +make sure you have CONFIG_DMA_API_DEBUG enabled during development. It can help +you find various issues which can be complex to debug otherwise.
Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com> --- Documentation/i2c/DMA-considerations | 50 ++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 Documentation/i2c/DMA-considerations