| Message ID | 20190531035348.7194-1-elder@linaro.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | net: introduce Qualcomm IPA driver | expand |
On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > This series presents the driver for the Qualcomm IP Accelerator > (IPA). > > This is version 2 of the series. This version has addressed almost > all of the feedback received in the first version: > > https://lore.kernel.org/lkml/20190512012508.10608-1-elder@linaro.org/ > More detail is included in the individual patches, but here is a > high-level summary of what's changed since then: > - Two spinlocks have been removed. > - The code for enabling and disabling endpoint interrupts has > been simplified considerably, and the spinlock is no longer > required > - A spinlock used when updating ring buffer pointers is no > longer needed. Integers indexing the ring are used instead > (and they don't even have to be atomic). > - One spinlock remains to protect list updates, but it is always > acquired using spin_lock_bh() (no more irqsave). > - Information about the queueing and completion of messages is now > supplied to the network stack in batches rather than one at a > time. > - I/O completion handling has been simplified, with the IRQ > handler now consisting mainly of disabling the interrupt and > calling napi_schedule(). > - Some comments have been updated and improved througout. > > What follows is the introduction supplied with v1 of the series. > > ----- > > The IPA is a component present in some Qualcomm SoCs that allows > network functions such as aggregation, filtering, routing, and NAT > to be performed without active involvement of the main application > processor (AP). > > Initially, these advanced features are disabled; the IPA driver > simply provides a network interface that makes the modem's LTE > network available to the AP. In addition, only support for the > IPA found in the Qualcomm SDM845 SoC is provided. My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC. Is the firmware that the driver loads already in linux-firmware or going to be there soon? How does the driver support multiple PDNs (eg PDP or EPS contexts) that are enabled through the control plane via QMI messages? I couldn't quite find that out. Thanks, Dan > This code is derived from a driver developed internally by Qualcomm. > A version of the original source can be seen here: > https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree > in the "drivers/platform/msm/ipa" directory. Many were involved in > developing this, but the following individuals deserve explicit > acknowledgement for their substantial contributions: > > Abhishek Choubey > Ady Abraham > Chaitanya Pratapa > David Arinzon > Ghanim Fodi > Gidon Studinski > Ravi Gummadidala > Shihuan Liu > Skylar Chang > > A version of this code was posted in November 2018 as an RFC. > > https://lore.kernel.org/lkml/20181107003250.5832-1-elder@linaro.org/ > All feedback received was addressed. The code has undergone > considerable further rework since that time, and most of the > "future work" described then has now been completed. > > This code is available in buildable form here, based on kernel > v5.2-rc1: > remote: ssh://git@git.linaro.org/people/alex.elder/linux.git > branch: ipa-v2_kernel-v5.2-rc2 > 75adf2ac1266 arm64: defconfig: enable build of IPA code > > The branch depends on a commit now found in in net-next. It has > been cherry-picked, and (in this branch) has this commit ID: > 13c627b5a078 net: qualcomm: rmnet: Move common struct definitions > to include > by > > -Alex > > Alex Elder (17): > bitfield.h: add FIELD_MAX() and field_max() > dt-bindings: soc: qcom: add IPA bindings > soc: qcom: ipa: main code > soc: qcom: ipa: configuration data > soc: qcom: ipa: clocking, interrupts, and memory > soc: qcom: ipa: GSI headers > soc: qcom: ipa: the generic software interface > soc: qcom: ipa: GSI transactions > soc: qcom: ipa: IPA interface to GSI > soc: qcom: ipa: IPA endpoints > soc: qcom: ipa: immediate commands > soc: qcom: ipa: IPA network device and microcontroller > soc: qcom: ipa: AP/modem communications > soc: qcom: ipa: support build of IPA code > MAINTAINERS: add entry for the Qualcomm IPA driver > arm64: dts: sdm845: add IPA information > arm64: defconfig: enable build of IPA code > > .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++ > MAINTAINERS | 6 + > arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 + > arch/arm64/configs/defconfig | 1 + > drivers/net/Kconfig | 2 + > drivers/net/Makefile | 1 + > drivers/net/ipa/Kconfig | 16 + > drivers/net/ipa/Makefile | 7 + > drivers/net/ipa/gsi.c | 1635 > +++++++++++++++++ > drivers/net/ipa/gsi.h | 246 +++ > drivers/net/ipa/gsi_private.h | 148 ++ > drivers/net/ipa/gsi_reg.h | 376 ++++ > drivers/net/ipa/gsi_trans.c | 624 +++++++ > drivers/net/ipa/gsi_trans.h | 116 ++ > drivers/net/ipa/ipa.h | 131 ++ > drivers/net/ipa/ipa_clock.c | 297 +++ > drivers/net/ipa/ipa_clock.h | 52 + > drivers/net/ipa/ipa_cmd.c | 377 ++++ > drivers/net/ipa/ipa_cmd.h | 116 ++ > drivers/net/ipa/ipa_data-sdm845.c | 245 +++ > drivers/net/ipa/ipa_data.h | 267 +++ > drivers/net/ipa/ipa_endpoint.c | 1283 +++++++++++++ > drivers/net/ipa/ipa_endpoint.h | 97 + > drivers/net/ipa/ipa_gsi.c | 48 + > drivers/net/ipa/ipa_gsi.h | 49 + > drivers/net/ipa/ipa_interrupt.c | 279 +++ > drivers/net/ipa/ipa_interrupt.h | 53 + > drivers/net/ipa/ipa_main.c | 921 ++++++++++ > drivers/net/ipa/ipa_mem.c | 234 +++ > drivers/net/ipa/ipa_mem.h | 83 + > drivers/net/ipa/ipa_netdev.c | 251 +++ > drivers/net/ipa/ipa_netdev.h | 24 + > drivers/net/ipa/ipa_qmi.c | 402 ++++ > drivers/net/ipa/ipa_qmi.h | 35 + > drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++ > drivers/net/ipa/ipa_qmi_msg.h | 238 +++ > drivers/net/ipa/ipa_reg.h | 279 +++ > drivers/net/ipa/ipa_smp2p.c | 304 +++ > drivers/net/ipa/ipa_smp2p.h | 47 + > drivers/net/ipa/ipa_uc.c | 208 +++ > drivers/net/ipa/ipa_uc.h | 32 + > include/linux/bitfield.h | 14 + > 42 files changed, 10358 insertions(+) > create mode 100644 > Documentation/devicetree/bindings/net/qcom,ipa.yaml > create mode 100644 drivers/net/ipa/Kconfig > create mode 100644 drivers/net/ipa/Makefile > create mode 100644 drivers/net/ipa/gsi.c > create mode 100644 drivers/net/ipa/gsi.h > create mode 100644 drivers/net/ipa/gsi_private.h > create mode 100644 drivers/net/ipa/gsi_reg.h > create mode 100644 drivers/net/ipa/gsi_trans.c > create mode 100644 drivers/net/ipa/gsi_trans.h > create mode 100644 drivers/net/ipa/ipa.h > create mode 100644 drivers/net/ipa/ipa_clock.c > create mode 100644 drivers/net/ipa/ipa_clock.h > create mode 100644 drivers/net/ipa/ipa_cmd.c > create mode 100644 drivers/net/ipa/ipa_cmd.h > create mode 100644 drivers/net/ipa/ipa_data-sdm845.c > create mode 100644 drivers/net/ipa/ipa_data.h > create mode 100644 drivers/net/ipa/ipa_endpoint.c > create mode 100644 drivers/net/ipa/ipa_endpoint.h > create mode 100644 drivers/net/ipa/ipa_gsi.c > create mode 100644 drivers/net/ipa/ipa_gsi.h > create mode 100644 drivers/net/ipa/ipa_interrupt.c > create mode 100644 drivers/net/ipa/ipa_interrupt.h > create mode 100644 drivers/net/ipa/ipa_main.c > create mode 100644 drivers/net/ipa/ipa_mem.c > create mode 100644 drivers/net/ipa/ipa_mem.h > create mode 100644 drivers/net/ipa/ipa_netdev.c > create mode 100644 drivers/net/ipa/ipa_netdev.h > create mode 100644 drivers/net/ipa/ipa_qmi.c > create mode 100644 drivers/net/ipa/ipa_qmi.h > create mode 100644 drivers/net/ipa/ipa_qmi_msg.c > create mode 100644 drivers/net/ipa/ipa_qmi_msg.h > create mode 100644 drivers/net/ipa/ipa_reg.h > create mode 100644 drivers/net/ipa/ipa_smp2p.c > create mode 100644 drivers/net/ipa/ipa_smp2p.h > create mode 100644 drivers/net/ipa/ipa_uc.c > create mode 100644 drivers/net/ipa/ipa_uc.h >
On 5/31/19 9:58 AM, Dan Williams wrote: > On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: >> This series presents the driver for the Qualcomm IP Accelerator >> (IPA). >> >> This is version 2 of the series. This version has addressed almost >> all of the feedback received in the first version: >> >> https://lore.kernel.org/lkml/20190512012508.10608-1-elder@linaro.org/ >> More detail is included in the individual patches, but here is a >> high-level summary of what's changed since then: >> - Two spinlocks have been removed. >> - The code for enabling and disabling endpoint interrupts has >> been simplified considerably, and the spinlock is no longer >> required >> - A spinlock used when updating ring buffer pointers is no >> longer needed. Integers indexing the ring are used instead >> (and they don't even have to be atomic). >> - One spinlock remains to protect list updates, but it is always >> acquired using spin_lock_bh() (no more irqsave). >> - Information about the queueing and completion of messages is now >> supplied to the network stack in batches rather than one at a >> time. >> - I/O completion handling has been simplified, with the IRQ >> handler now consisting mainly of disabling the interrupt and >> calling napi_schedule(). >> - Some comments have been updated and improved througout. >> >> What follows is the introduction supplied with v1 of the series. >> >> ----- >> >> The IPA is a component present in some Qualcomm SoCs that allows >> network functions such as aggregation, filtering, routing, and NAT >> to be performed without active involvement of the main application >> processor (AP). >> >> Initially, these advanced features are disabled; the IPA driver >> simply provides a network interface that makes the modem's LTE >> network available to the AP. In addition, only support for the >> IPA found in the Qualcomm SDM845 SoC is provided. > > My question from the Nov 2018 IPA rmnet driver still stands; how does > this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is > really just a netdev talking to the IPA itself and unrelated to > net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- > culting rmnet around just because it happens to be a net driver for a > QC SoC. First, the relationship between the IPA driver and the rmnet driver is that the IPA driver is assumed to sit between the rmnet driver and the hardware. Currently the modem is assumed to use QMAP protocol. This means each packet is prefixed by a (struct rmnet_map_header) structure that allows the IPA connection to be multiplexed for several logical connections. The rmnet driver parses such messages and implements the multiplexed network interfaces. QMAP protocol can also be used for aggregating many small packets into a larger message. The rmnet driver implements de-aggregation of such messages (and could probably aggregate them for TX as well). Finally, the IPA can support checksum offload, and the rmnet driver handles providing a prepended header (for TX) and interpreting the appended trailer (for RX) if these features are enabled. So basically, the purpose of the rmnet driver is to handle QMAP protocol connections, and right now that's what the modem provides. > Is the firmware that the driver loads already in linux-firmware or > going to be there soon? It is not right now, and I have no information on when it can be available. The AP *can* load the firmware but right now we rely on the modem doing it (until we can make the firmware available). > How does the driver support multiple PDNs (eg PDP or EPS contexts) that > are enabled through the control plane via QMI messages? I couldn't > quite find that out. To be honest, I don't know the answer to this. All of my work has been on this transport driver and I believe these things are handled by user space. But I really don't know details. -Alex > Thanks, > Dan > >> This code is derived from a driver developed internally by Qualcomm. >> A version of the original source can be seen here: >> https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree >> in the "drivers/platform/msm/ipa" directory. Many were involved in >> developing this, but the following individuals deserve explicit >> acknowledgement for their substantial contributions: >> >> Abhishek Choubey >> Ady Abraham >> Chaitanya Pratapa >> David Arinzon >> Ghanim Fodi >> Gidon Studinski >> Ravi Gummadidala >> Shihuan Liu >> Skylar Chang >> >> A version of this code was posted in November 2018 as an RFC. >> >> https://lore.kernel.org/lkml/20181107003250.5832-1-elder@linaro.org/ >> All feedback received was addressed. The code has undergone >> considerable further rework since that time, and most of the >> "future work" described then has now been completed. >> >> This code is available in buildable form here, based on kernel >> v5.2-rc1: >> remote: ssh://git@git.linaro.org/people/alex.elder/linux.git >> branch: ipa-v2_kernel-v5.2-rc2 >> 75adf2ac1266 arm64: defconfig: enable build of IPA code >> >> The branch depends on a commit now found in in net-next. It has >> been cherry-picked, and (in this branch) has this commit ID: >> 13c627b5a078 net: qualcomm: rmnet: Move common struct definitions >> to include >> by >> >> -Alex >> >> Alex Elder (17): >> bitfield.h: add FIELD_MAX() and field_max() >> dt-bindings: soc: qcom: add IPA bindings >> soc: qcom: ipa: main code >> soc: qcom: ipa: configuration data >> soc: qcom: ipa: clocking, interrupts, and memory >> soc: qcom: ipa: GSI headers >> soc: qcom: ipa: the generic software interface >> soc: qcom: ipa: GSI transactions >> soc: qcom: ipa: IPA interface to GSI >> soc: qcom: ipa: IPA endpoints >> soc: qcom: ipa: immediate commands >> soc: qcom: ipa: IPA network device and microcontroller >> soc: qcom: ipa: AP/modem communications >> soc: qcom: ipa: support build of IPA code >> MAINTAINERS: add entry for the Qualcomm IPA driver >> arm64: dts: sdm845: add IPA information >> arm64: defconfig: enable build of IPA code >> >> .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++ >> MAINTAINERS | 6 + >> arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 + >> arch/arm64/configs/defconfig | 1 + >> drivers/net/Kconfig | 2 + >> drivers/net/Makefile | 1 + >> drivers/net/ipa/Kconfig | 16 + >> drivers/net/ipa/Makefile | 7 + >> drivers/net/ipa/gsi.c | 1635 >> +++++++++++++++++ >> drivers/net/ipa/gsi.h | 246 +++ >> drivers/net/ipa/gsi_private.h | 148 ++ >> drivers/net/ipa/gsi_reg.h | 376 ++++ >> drivers/net/ipa/gsi_trans.c | 624 +++++++ >> drivers/net/ipa/gsi_trans.h | 116 ++ >> drivers/net/ipa/ipa.h | 131 ++ >> drivers/net/ipa/ipa_clock.c | 297 +++ >> drivers/net/ipa/ipa_clock.h | 52 + >> drivers/net/ipa/ipa_cmd.c | 377 ++++ >> drivers/net/ipa/ipa_cmd.h | 116 ++ >> drivers/net/ipa/ipa_data-sdm845.c | 245 +++ >> drivers/net/ipa/ipa_data.h | 267 +++ >> drivers/net/ipa/ipa_endpoint.c | 1283 +++++++++++++ >> drivers/net/ipa/ipa_endpoint.h | 97 + >> drivers/net/ipa/ipa_gsi.c | 48 + >> drivers/net/ipa/ipa_gsi.h | 49 + >> drivers/net/ipa/ipa_interrupt.c | 279 +++ >> drivers/net/ipa/ipa_interrupt.h | 53 + >> drivers/net/ipa/ipa_main.c | 921 ++++++++++ >> drivers/net/ipa/ipa_mem.c | 234 +++ >> drivers/net/ipa/ipa_mem.h | 83 + >> drivers/net/ipa/ipa_netdev.c | 251 +++ >> drivers/net/ipa/ipa_netdev.h | 24 + >> drivers/net/ipa/ipa_qmi.c | 402 ++++ >> drivers/net/ipa/ipa_qmi.h | 35 + >> drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++ >> drivers/net/ipa/ipa_qmi_msg.h | 238 +++ >> drivers/net/ipa/ipa_reg.h | 279 +++ >> drivers/net/ipa/ipa_smp2p.c | 304 +++ >> drivers/net/ipa/ipa_smp2p.h | 47 + >> drivers/net/ipa/ipa_uc.c | 208 +++ >> drivers/net/ipa/ipa_uc.h | 32 + >> include/linux/bitfield.h | 14 + >> 42 files changed, 10358 insertions(+) >> create mode 100644 >> Documentation/devicetree/bindings/net/qcom,ipa.yaml >> create mode 100644 drivers/net/ipa/Kconfig >> create mode 100644 drivers/net/ipa/Makefile >> create mode 100644 drivers/net/ipa/gsi.c >> create mode 100644 drivers/net/ipa/gsi.h >> create mode 100644 drivers/net/ipa/gsi_private.h >> create mode 100644 drivers/net/ipa/gsi_reg.h >> create mode 100644 drivers/net/ipa/gsi_trans.c >> create mode 100644 drivers/net/ipa/gsi_trans.h >> create mode 100644 drivers/net/ipa/ipa.h >> create mode 100644 drivers/net/ipa/ipa_clock.c >> create mode 100644 drivers/net/ipa/ipa_clock.h >> create mode 100644 drivers/net/ipa/ipa_cmd.c >> create mode 100644 drivers/net/ipa/ipa_cmd.h >> create mode 100644 drivers/net/ipa/ipa_data-sdm845.c >> create mode 100644 drivers/net/ipa/ipa_data.h >> create mode 100644 drivers/net/ipa/ipa_endpoint.c >> create mode 100644 drivers/net/ipa/ipa_endpoint.h >> create mode 100644 drivers/net/ipa/ipa_gsi.c >> create mode 100644 drivers/net/ipa/ipa_gsi.h >> create mode 100644 drivers/net/ipa/ipa_interrupt.c >> create mode 100644 drivers/net/ipa/ipa_interrupt.h >> create mode 100644 drivers/net/ipa/ipa_main.c >> create mode 100644 drivers/net/ipa/ipa_mem.c >> create mode 100644 drivers/net/ipa/ipa_mem.h >> create mode 100644 drivers/net/ipa/ipa_netdev.c >> create mode 100644 drivers/net/ipa/ipa_netdev.h >> create mode 100644 drivers/net/ipa/ipa_qmi.c >> create mode 100644 drivers/net/ipa/ipa_qmi.h >> create mode 100644 drivers/net/ipa/ipa_qmi_msg.c >> create mode 100644 drivers/net/ipa/ipa_qmi_msg.h >> create mode 100644 drivers/net/ipa/ipa_reg.h >> create mode 100644 drivers/net/ipa/ipa_smp2p.c >> create mode 100644 drivers/net/ipa/ipa_smp2p.h >> create mode 100644 drivers/net/ipa/ipa_uc.c >> create mode 100644 drivers/net/ipa/ipa_uc.h >> >
On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: > On 5/31/19 9:58 AM, Dan Williams wrote: > > On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > > > > My question from the Nov 2018 IPA rmnet driver still stands; how does > > this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is > > really just a netdev talking to the IPA itself and unrelated to > > net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- > > culting rmnet around just because it happens to be a net driver for a > > QC SoC. > > First, the relationship between the IPA driver and the rmnet driver > is that the IPA driver is assumed to sit between the rmnet driver > and the hardware. Does this mean that IPA can only be used to back rmnet, and rmnet can only be used on top of IPA, or can or both of them be combined with another driver to talk to instead? > Currently the modem is assumed to use QMAP protocol. This means > each packet is prefixed by a (struct rmnet_map_header) structure > that allows the IPA connection to be multiplexed for several logical > connections. The rmnet driver parses such messages and implements > the multiplexed network interfaces. > > QMAP protocol can also be used for aggregating many small packets > into a larger message. The rmnet driver implements de-aggregation > of such messages (and could probably aggregate them for TX as well). > > Finally, the IPA can support checksum offload, and the rmnet > driver handles providing a prepended header (for TX) and > interpreting the appended trailer (for RX) if these features > are enabled. > > So basically, the purpose of the rmnet driver is to handle QMAP > protocol connections, and right now that's what the modem provides. Do you have any idea why this particular design was picked? My best guess is that it evolved organically with multiple generations of hardware and software, rather than being thought out as a nice abstraction layer. If the two are tightly connected, this might mean that what we actually want here is to reintegrate the two components into a single driver with a much simpler RX and TX path that handles the checksumming and aggregation of data packets directly as it passes them from the network stack into the hardware. Always passing data from one netdev to another both ways sounds like it introduces both direct CPU overhead, and problems with flow control when data gets buffered inbetween. The intermediate buffer here acts like a router that must pass data along or randomly drop packets when the consumer can't keep up with the producer. Arnd
On 5/31/19 2:19 PM, Arnd Bergmann wrote: > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: >> On 5/31/19 9:58 AM, Dan Williams wrote: >>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: >>> >>> My question from the Nov 2018 IPA rmnet driver still stands; how does >>> this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is >>> really just a netdev talking to the IPA itself and unrelated to >>> net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- >>> culting rmnet around just because it happens to be a net driver for a >>> QC SoC. >> >> First, the relationship between the IPA driver and the rmnet driver >> is that the IPA driver is assumed to sit between the rmnet driver >> and the hardware. > > Does this mean that IPA can only be used to back rmnet, and rmnet > can only be used on top of IPA, or can or both of them be combined > with another driver to talk to instead? No it does not mean that. As I understand it, one reason for the rmnet layer was to abstract the back end, which would allow using a modem, or using something else (a LAN?), without exposing certain details of the hardware. (Perhaps to support multiplexing, etc. without duplicating that logic in two "back-end" drivers?) To be perfectly honest, at first I thought having IPA use rmnet was a cargo cult thing like Dan suggested, because I didn't see the benefit. I now see why one would use that pass-through layer to handle the QMAP features. But back to your question. The other thing is that I see no reason the IPA couldn't present a "normal" (non QMAP) interface for a modem. It's something I'd really like to be able to do, but I can't do it without having the modem firmware change its configuration for these endpoints. My access to the people who implement the modem firmware has been very limited (something I hope to improve), and unless and until I can get corresponding changes on the modem side to implement connections that don't use QMAP, I can't implement such a thing. >> Currently the modem is assumed to use QMAP protocol. This means >> each packet is prefixed by a (struct rmnet_map_header) structure >> that allows the IPA connection to be multiplexed for several logical >> connections. The rmnet driver parses such messages and implements >> the multiplexed network interfaces. >> >> QMAP protocol can also be used for aggregating many small packets >> into a larger message. The rmnet driver implements de-aggregation >> of such messages (and could probably aggregate them for TX as well). >> >> Finally, the IPA can support checksum offload, and the rmnet >> driver handles providing a prepended header (for TX) and >> interpreting the appended trailer (for RX) if these features >> are enabled. >> >> So basically, the purpose of the rmnet driver is to handle QMAP >> protocol connections, and right now that's what the modem provides. > > Do you have any idea why this particular design was picked? I don't really. I inherited it. Early on, when I asked about the need for QMAP I was told it was important because it offered certain features, but at that time I was somewhat new to the code and didn't have the insight to judge the merits of the design. Since then I've mostly just accepted it and concentrated on improving the IPA driver. > My best guess is that it evolved organically with multiple > generations of hardware and software, rather than being thought > out as a nice abstraction layer. If the two are tightly connected, > this might mean that what we actually want here is to reintegrate > the two components into a single driver with a much simpler > RX and TX path that handles the checksumming and aggregation > of data packets directly as it passes them from the network > stack into the hardware. In general, I agree. And Dan suggested combining the rmnet and IPA drivers into a single driver when I posted the RFC code last year. There's still the notion of switching back ends that I mentioned earlier; if that's indeed an important feature it might argue for keeping rmnet as a shim layer. But I'm really not the person to comment on this. Someone (Subash?) from Qualcomm might be able to provide better answers. > Always passing data from one netdev to another both ways > sounds like it introduces both direct CPU overhead, and > problems with flow control when data gets buffered inbetween. My impression is the rmnet driver is a pretty thin layer, so the CPU overhead is probably not that great (though deaggregating a message is expensive). I agree with you on the flow control. > The intermediate buffer here acts like a router that must > pass data along or randomly drop packets when the consumer > can't keep up with the producer. I haven't reviewed the rmnet code in any detail, but you may be right. -Alex > > Arnd >
On Fri, May 31, 2019 at 10:47 PM Alex Elder <elder@linaro.org> wrote: > On 5/31/19 2:19 PM, Arnd Bergmann wrote: > > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: > >> On 5/31/19 9:58 AM, Dan Williams wrote: > >>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > > > > Does this mean that IPA can only be used to back rmnet, and rmnet > > can only be used on top of IPA, or can or both of them be combined > > with another driver to talk to instead? > > No it does not mean that. > > As I understand it, one reason for the rmnet layer was to abstract > the back end, which would allow using a modem, or using something > else (a LAN?), without exposing certain details of the hardware. > (Perhaps to support multiplexing, etc. without duplicating that > logic in two "back-end" drivers?) > > To be perfectly honest, at first I thought having IPA use rmnet > was a cargo cult thing like Dan suggested, because I didn't see > the benefit. I now see why one would use that pass-through layer > to handle the QMAP features. > > But back to your question. The other thing is that I see no > reason the IPA couldn't present a "normal" (non QMAP) interface > for a modem. It's something I'd really like to be able to do, > but I can't do it without having the modem firmware change its > configuration for these endpoints. My access to the people who > implement the modem firmware has been very limited (something > I hope to improve), and unless and until I can get corresponding > changes on the modem side to implement connections that don't > use QMAP, I can't implement such a thing. Why would that require firmware changes? What I was thinking here is to turn the bits of the rmnet driver that actually do anything interesting on the headers into a library module (or a header file with inline functions) that can be called directly by the ipa driver, keeping the protocol unchanged. > > Always passing data from one netdev to another both ways > > sounds like it introduces both direct CPU overhead, and > > problems with flow control when data gets buffered inbetween. > > My impression is the rmnet driver is a pretty thin layer, > so the CPU overhead is probably not that great (though > deaggregating a message is expensive). I agree with you > on the flow control. The CPU overhead I mean is not from executing code in the rmnet driver, but from passing packets through the network stack between the two drivers, i.e. adding each frame to a queue and taking it back out. I'm not sure how this ends up working in reality but from a first look it seems like we might bounce in an out of the softirq handler inbetween. Arnd
On 5/31/19 4:12 PM, Arnd Bergmann wrote: > On Fri, May 31, 2019 at 10:47 PM Alex Elder <elder@linaro.org> wrote: >> On 5/31/19 2:19 PM, Arnd Bergmann wrote: >>> On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: >>>> On 5/31/19 9:58 AM, Dan Williams wrote: >>>>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: >>> >>> Does this mean that IPA can only be used to back rmnet, and rmnet >>> can only be used on top of IPA, or can or both of them be combined >>> with another driver to talk to instead? >> >> No it does not mean that. >> >> As I understand it, one reason for the rmnet layer was to abstract >> the back end, which would allow using a modem, or using something >> else (a LAN?), without exposing certain details of the hardware. >> (Perhaps to support multiplexing, etc. without duplicating that >> logic in two "back-end" drivers?) >> >> To be perfectly honest, at first I thought having IPA use rmnet >> was a cargo cult thing like Dan suggested, because I didn't see >> the benefit. I now see why one would use that pass-through layer >> to handle the QMAP features. >> >> But back to your question. The other thing is that I see no >> reason the IPA couldn't present a "normal" (non QMAP) interface >> for a modem. It's something I'd really like to be able to do, >> but I can't do it without having the modem firmware change its >> configuration for these endpoints. My access to the people who >> implement the modem firmware has been very limited (something >> I hope to improve), and unless and until I can get corresponding >> changes on the modem side to implement connections that don't >> use QMAP, I can't implement such a thing. > > Why would that require firmware changes? What I was thinking > here is to turn the bits of the rmnet driver that actually do anything > interesting on the headers into a library module (or a header file > with inline functions) that can be called directly by the ipa driver, > keeping the protocol unchanged. You know, it's possible you're right about not needing firmware changes. But it has always been my impression they would be needed. Here's why. It looks like this: GSI Channel GSI Channel | | ---------- v ------- v ------------- | AP (ep)|=======| IPA |=======|(ep) Modem | ---------- ------- ------------- The AP and Modem each have IPA endpoints (ep), which use GSI channels, to communicate with the IPA. Each endpoint has configuration options (such as checksum offload). I *thought* that the configurations of the two endpoints need to be compatible (e.g., they need to agree on whether they're aggregating). But with your questioning I now think you may be right, that only the local endpoint's configuration matters. I will inquire further on this. I *know* that the AP and modem exchange some information about IPA configuration, but looking more closely that looks like it's all about the configuration of shared IPA resources, not endpoints. That said, the broader design (including the user space code) surely assumes rmnet, and I don't have any sense of what impact changing that would make. I am sure that changing it would not be well received. -Alex >>> Always passing data from one netdev to another both ways >>> sounds like it introduces both direct CPU overhead, and >>> problems with flow control when data gets buffered inbetween. >> >> My impression is the rmnet driver is a pretty thin layer, >> so the CPU overhead is probably not that great (though >> deaggregating a message is expensive). I agree with you >> on the flow control. > > The CPU overhead I mean is not from executing code in the > rmnet driver, but from passing packets through the network > stack between the two drivers, i.e. adding each frame to > a queue and taking it back out. I'm not sure how this ends > up working in reality but from a first look it seems like > we might bounce in an out of the softirq handler inbetween. > > Arnd >
On Fri 31 May 12:19 PDT 2019, Arnd Bergmann wrote: > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: > > On 5/31/19 9:58 AM, Dan Williams wrote: > > > On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: [..] > > So basically, the purpose of the rmnet driver is to handle QMAP > > protocol connections, and right now that's what the modem provides. > > Do you have any idea why this particular design was picked? > From what I've seen of QMAP it seems like a reasonable design choice to have a software component (rmnet) dealing with this, separate from the transport. And I think IPA is the 4th or 5th mechanism for transporting QMAP packets back and forth to the modem. Downstream rmnet is copyright 2007-, and I know of interest in bringing at least one of the other transports upstream. Regards, Bjorn
On Fri 31 May 13:47 PDT 2019, Alex Elder wrote: > On 5/31/19 2:19 PM, Arnd Bergmann wrote: > > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: > >> On 5/31/19 9:58 AM, Dan Williams wrote: > >>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > >>> > >>> My question from the Nov 2018 IPA rmnet driver still stands; how does > >>> this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is > >>> really just a netdev talking to the IPA itself and unrelated to > >>> net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- > >>> culting rmnet around just because it happens to be a net driver for a > >>> QC SoC. > >> > >> First, the relationship between the IPA driver and the rmnet driver > >> is that the IPA driver is assumed to sit between the rmnet driver > >> and the hardware. > > > > Does this mean that IPA can only be used to back rmnet, and rmnet > > can only be used on top of IPA, or can or both of them be combined > > with another driver to talk to instead? > > No it does not mean that. > > As I understand it, one reason for the rmnet layer was to abstract > the back end, which would allow using a modem, or using something > else (a LAN?), without exposing certain details of the hardware. > (Perhaps to support multiplexing, etc. without duplicating that > logic in two "back-end" drivers?) > > To be perfectly honest, at first I thought having IPA use rmnet > was a cargo cult thing like Dan suggested, because I didn't see > the benefit. I now see why one would use that pass-through layer > to handle the QMAP features. > > But back to your question. The other thing is that I see no > reason the IPA couldn't present a "normal" (non QMAP) interface > for a modem. It's something I'd really like to be able to do, > but I can't do it without having the modem firmware change its > configuration for these endpoints. My access to the people who > implement the modem firmware has been very limited (something > I hope to improve), and unless and until I can get corresponding > changes on the modem side to implement connections that don't > use QMAP, I can't implement such a thing. > But any such changes would either be years into the future or for specific devices and as such not applicable to any/most of devices on the market now or in the coming years. But as Arnd points out, if the software split between IPA and rmnet is suboptimal your are encouraged to fix that. Regards, Bjorn
On 2019-05-31 17:33, Bjorn Andersson wrote: > On Fri 31 May 13:47 PDT 2019, Alex Elder wrote: > >> On 5/31/19 2:19 PM, Arnd Bergmann wrote: >> > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: >> >> On 5/31/19 9:58 AM, Dan Williams wrote: >> >>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: >> >>> >> >>> My question from the Nov 2018 IPA rmnet driver still stands; how does >> >>> this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is >> >>> really just a netdev talking to the IPA itself and unrelated to >> >>> net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- >> >>> culting rmnet around just because it happens to be a net driver for a >> >>> QC SoC. >> >> >> >> First, the relationship between the IPA driver and the rmnet driver >> >> is that the IPA driver is assumed to sit between the rmnet driver >> >> and the hardware. >> > >> > Does this mean that IPA can only be used to back rmnet, and rmnet >> > can only be used on top of IPA, or can or both of them be combined >> > with another driver to talk to instead? >> >> No it does not mean that. >> >> As I understand it, one reason for the rmnet layer was to abstract >> the back end, which would allow using a modem, or using something >> else (a LAN?), without exposing certain details of the hardware. >> (Perhaps to support multiplexing, etc. without duplicating that >> logic in two "back-end" drivers?) >> >> To be perfectly honest, at first I thought having IPA use rmnet >> was a cargo cult thing like Dan suggested, because I didn't see >> the benefit. I now see why one would use that pass-through layer >> to handle the QMAP features. >> >> But back to your question. The other thing is that I see no >> reason the IPA couldn't present a "normal" (non QMAP) interface >> for a modem. It's something I'd really like to be able to do, >> but I can't do it without having the modem firmware change its >> configuration for these endpoints. My access to the people who >> implement the modem firmware has been very limited (something >> I hope to improve), and unless and until I can get corresponding >> changes on the modem side to implement connections that don't >> use QMAP, I can't implement such a thing. >> > > But any such changes would either be years into the future or for > specific devices and as such not applicable to any/most of devices on > the market now or in the coming years. > > > But as Arnd points out, if the software split between IPA and rmnet is > suboptimal your are encouraged to fix that. > > Regards, > Bjorn The split rmnet design was chosen because we could place rmnet over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159) or USB. rmnet registers a rx handler, so the rmnet packet processing itself happens in the same softirq when packets are queued to network stack by IPA.
On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan <subashab@codeaurora.org> wrote: > On 2019-05-31 17:33, Bjorn Andersson wrote: > > On Fri 31 May 13:47 PDT 2019, Alex Elder wrote: > >> On 5/31/19 2:19 PM, Arnd Bergmann wrote: > > But any such changes would either be years into the future or for > > specific devices and as such not applicable to any/most of devices on > > the market now or in the coming years. > > > > > > But as Arnd points out, if the software split between IPA and rmnet is > > suboptimal your are encouraged to fix that. > > The split rmnet design was chosen because we could place rmnet > over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159) > or USB. > > rmnet registers a rx handler, so the rmnet packet processing itself > happens in the same softirq when packets are queued to network stack > by IPA. I've read up on the implementation some more, and concluded that it's mostly a regular protocol wrapper, doing IP over QMAP. There is nothing wrong with the basic concept I think, and as you describe this is an abstraction to keep the common bits in one place, and have them configured consistently. A few observations on more details here: - What I'm worried about most here is the flow control handling on the transmit side. The IPA driver now uses the modern BQL method to control how much data gets submitted to the hardware at any time. The rmnet driver also uses flow control using the rmnet_map_command() function, that blocks tx on the higher level device when the remote side asks us to. I fear that doing flow control for a single physical device on two separate netdev instances is counterproductive and confuses both sides. - I was a little confused by the location of the rmnet driver in drivers/net/ethernet/... More conventionally, I think as a protocol handler it should go into net/qmap/, with the ipa driver going into drivers/net/qmap/ipa/, similar to what we have fo ethernet, wireless, ppp, appletalk, etc. - The rx_handler uses gro_cells, which as I understand is meant for generic tunnelling setups and takes another loop through NAPI to aggregate data from multiple queues, but in case of IPA's single-queue receive calling gro directly would be simpler and more efficient. - I'm still not sure I understand the purpose of the layering with using an rx_handler as opposed to just using EXPORT_SYMBOL(rmnet_rx_handler) and calling that from the hardware driver directly. From the overall design and the rmnet Kconfig description, it appears as though the intention as that rmnet could be a generic wrapper on top of any device, but from the implementation it seems that IPA is not actually usable that way and would always go through IPA. Arnd
On 6/3/19 5:04 AM, Arnd Bergmann wrote: > On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan > <subashab@codeaurora.org> wrote: >> On 2019-05-31 17:33, Bjorn Andersson wrote: >>> On Fri 31 May 13:47 PDT 2019, Alex Elder wrote: >>>> On 5/31/19 2:19 PM, Arnd Bergmann wrote: >>> But any such changes would either be years into the future or for >>> specific devices and as such not applicable to any/most of devices on >>> the market now or in the coming years. >>> >>> >>> But as Arnd points out, if the software split between IPA and rmnet is >>> suboptimal your are encouraged to fix that. >> >> The split rmnet design was chosen because we could place rmnet >> over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159) >> or USB. >> >> rmnet registers a rx handler, so the rmnet packet processing itself >> happens in the same softirq when packets are queued to network stack >> by IPA. > > I've read up on the implementation some more, and concluded that > it's mostly a regular protocol wrapper, doing IP over QMAP. There > is nothing wrong with the basic concept I think, and as you describe > this is an abstraction to keep the common bits in one place, and > have them configured consistently. > > A few observations on more details here: > > - What I'm worried about most here is the flow control handling on the > transmit side. The IPA driver now uses the modern BQL method to > control how much data gets submitted to the hardware at any time. > The rmnet driver also uses flow control using the > rmnet_map_command() function, that blocks tx on the higher > level device when the remote side asks us to. > I fear that doing flow control for a single physical device on two > separate netdev instances is counterproductive and confuses > both sides. I understand what you're saying here, and instinctively I think you're right. But BQL manages the *local* interface's ability to get rid of packets, whereas the QMAP flow control is initiated by the other end of the connection (the modem in this case). With multiplexing, it's possible that one of several logical devices on the modem side has exhausted a resource and must ask the source of the data on the host side to suspend the flow. Meanwhile the other logical devices sharing the physical link might be fine, and should not be delayed by the first one. It is the multiplexing itself that confuses the BQL algorithm. The abstraction obscures the *real* rates at which individual logical connections are able to transmit data. Even if the multiple logical interfaces implemented BQL, they would not get the feedback they need directly from the IPA driver, because transmitting over the physical interface might succeed even if the logical interface on the modem side can't handle more data. So I think the flow control commands may be necessary, given multiplexing. The rmnet driver could use BQL, and could return NETDEV_TX_BUSY for a logical interface when its TX flow has been stopped by a QMAP command. That way the feedback for BQL on the logical interfaces would be provided in the right place. I have no good intuition about the interaction between two layered BQL managed queues though. > - I was a little confused by the location of the rmnet driver in > drivers/net/ethernet/... More conventionally, I think as a protocol > handler it should go into net/qmap/, with the ipa driver going > into drivers/net/qmap/ipa/, similar to what we have fo ethernet, > wireless, ppp, appletalk, etc. > > - The rx_handler uses gro_cells, which as I understand is meant > for generic tunnelling setups and takes another loop through > NAPI to aggregate data from multiple queues, but in case of > IPA's single-queue receive calling gro directly would be simpler > and more efficient. I have been planning to investigate some of the generic GRO stuff for IPA but was going to wait on that until the basic code was upstream. > - I'm still not sure I understand the purpose of the layering with > using an rx_handler as opposed to just using > EXPORT_SYMBOL(rmnet_rx_handler) and calling that from > the hardware driver directly. I think that's a good idea. > From the overall design and the rmnet Kconfig description, it > appears as though the intention as that rmnet could be a > generic wrapper on top of any device, but from the > implementation it seems that IPA is not actually usable that > way and would always go through IPA. As far as I know *nothing* upstream currently uses rmnet; the IPA driver will be the first, but as Bjorn said others seem to be on the way. I'm not sure what you mean by "IPA is not usable that way." Currently the IPA driver assumes a fixed configuration, and that configuration assumes the use of QMAP, and therefore assumes the rmnet driver is layered above it. That doesn't preclude rmnet from using a different back end. And I'll also mention that although QMAP *can* do multiplexed connections over a single physical link, the IPA code I posted currently supports only one logical interface. -Alex > > Arnd >
On Fri, 2019-05-31 at 17:59 -0600, Subash Abhinov Kasiviswanathan wrote: > On 2019-05-31 17:33, Bjorn Andersson wrote: > > On Fri 31 May 13:47 PDT 2019, Alex Elder wrote: > > > > > On 5/31/19 2:19 PM, Arnd Bergmann wrote: > > > > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> > > > > wrote: > > > > > On 5/31/19 9:58 AM, Dan Williams wrote: > > > > > > On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > > > > > > > > > > > > My question from the Nov 2018 IPA rmnet driver still > > > > > > stands; how does > > > > > > this relate to net/ethernet/qualcomm/rmnet/ if at all? And > > > > > > if this is > > > > > > really just a netdev talking to the IPA itself and > > > > > > unrelated to > > > > > > net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop > > > > > > cargo- > > > > > > culting rmnet around just because it happens to be a net > > > > > > driver for a > > > > > > QC SoC. > > > > > > > > > > First, the relationship between the IPA driver and the rmnet > > > > > driver > > > > > is that the IPA driver is assumed to sit between the rmnet > > > > > driver > > > > > and the hardware. > > > > > > > > Does this mean that IPA can only be used to back rmnet, and > > > > rmnet > > > > can only be used on top of IPA, or can or both of them be > > > > combined > > > > with another driver to talk to instead? > > > > > > No it does not mean that. > > > > > > As I understand it, one reason for the rmnet layer was to > > > abstract > > > the back end, which would allow using a modem, or using something > > > else (a LAN?), without exposing certain details of the hardware. > > > (Perhaps to support multiplexing, etc. without duplicating that > > > logic in two "back-end" drivers?) > > > > > > To be perfectly honest, at first I thought having IPA use rmnet > > > was a cargo cult thing like Dan suggested, because I didn't see > > > the benefit. I now see why one would use that pass-through layer > > > to handle the QMAP features. > > > > > > But back to your question. The other thing is that I see no > > > reason the IPA couldn't present a "normal" (non QMAP) interface > > > for a modem. It's something I'd really like to be able to do, > > > but I can't do it without having the modem firmware change its > > > configuration for these endpoints. My access to the people who > > > implement the modem firmware has been very limited (something > > > I hope to improve), and unless and until I can get corresponding > > > changes on the modem side to implement connections that don't > > > use QMAP, I can't implement such a thing. > > > > > > > But any such changes would either be years into the future or for > > specific devices and as such not applicable to any/most of devices > > on > > the market now or in the coming years. > > > > > > But as Arnd points out, if the software split between IPA and rmnet > > is > > suboptimal your are encouraged to fix that. > > > > Regards, > > Bjorn > > The split rmnet design was chosen because we could place rmnet > over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159) > or USB. Yeah, that's what I was looking for clarification on :) Clearly since rmnet can have many transports it should be able to be used by different HW drivers, be that qmi_wwan, IPA, and maybe even rmnet_smd.c? > rmnet registers a rx handler, so the rmnet packet processing itself > happens in the same softirq when packets are queued to network stack > by IPA. This directly relates to the discussion about a WWAN subsystem that Johannes Berg started a couple weeks ago. IPA appears to create a netdev of its own. Is that netdev usable immediately, or does one need to create an rmnet device on top to access the default PDN? Thanks, Dan
On Fri, 2019-05-31 at 15:47 -0500, Alex Elder wrote: > On 5/31/19 2:19 PM, Arnd Bergmann wrote: > > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> > > wrote: > > > On 5/31/19 9:58 AM, Dan Williams wrote: > > > > On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > > > > > > > > My question from the Nov 2018 IPA rmnet driver still stands; > > > > how does > > > > this relate to net/ethernet/qualcomm/rmnet/ if at all? And if > > > > this is > > > > really just a netdev talking to the IPA itself and unrelated to > > > > net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop > > > > cargo- > > > > culting rmnet around just because it happens to be a net driver > > > > for a > > > > QC SoC. > > > > > > First, the relationship between the IPA driver and the rmnet > > > driver > > > is that the IPA driver is assumed to sit between the rmnet driver > > > and the hardware. > > > > Does this mean that IPA can only be used to back rmnet, and rmnet > > can only be used on top of IPA, or can or both of them be combined > > with another driver to talk to instead? > > No it does not mean that. > > As I understand it, one reason for the rmnet layer was to abstract > the back end, which would allow using a modem, or using something > else (a LAN?), without exposing certain details of the hardware. > (Perhaps to support multiplexing, etc. without duplicating that > logic in two "back-end" drivers?) > > To be perfectly honest, at first I thought having IPA use rmnet > was a cargo cult thing like Dan suggested, because I didn't see To be clear I only meant cargo-culting the naming, not any functionality. Clearly IPA/rmnet/QMAP are pretty intimately connected at this point. But this goes back to whether IPA needs a netdev itself or whether you need an rmnet device created on top. If the former then I'd say no cargo-culting, if the later then it's a moot point because the device name will be rmnet%d anyway. Dan > the benefit. I now see why one would use that pass-through layer > to handle the QMAP features. > > But back to your question. The other thing is that I see no > reason the IPA couldn't present a "normal" (non QMAP) interface > for a modem. It's something I'd really like to be able to do, > but I can't do it without having the modem firmware change its > configuration for these endpoints. My access to the people who > implement the modem firmware has been very limited (something > I hope to improve), and unless and until I can get corresponding > changes on the modem side to implement connections that don't > use QMAP, I can't implement such a thing. > > > > Currently the modem is assumed to use QMAP protocol. This means > > > each packet is prefixed by a (struct rmnet_map_header) structure > > > that allows the IPA connection to be multiplexed for several > > > logical > > > connections. The rmnet driver parses such messages and > > > implements > > > the multiplexed network interfaces. > > > > > > QMAP protocol can also be used for aggregating many small packets > > > into a larger message. The rmnet driver implements de- > > > aggregation > > > of such messages (and could probably aggregate them for TX as > > > well). > > > > > > Finally, the IPA can support checksum offload, and the rmnet > > > driver handles providing a prepended header (for TX) and > > > interpreting the appended trailer (for RX) if these features > > > are enabled. > > > > > > So basically, the purpose of the rmnet driver is to handle QMAP > > > protocol connections, and right now that's what the modem > > > provides. > > > > Do you have any idea why this particular design was picked? > > I don't really. I inherited it. Early on, when I asked about > the need for QMAP I was told it was important because it offered > certain features, but at that time I was somewhat new to the code > and didn't have the insight to judge the merits of the design. > Since then I've mostly just accepted it and concentrated on > improving the IPA driver. > > > My best guess is that it evolved organically with multiple > > generations of hardware and software, rather than being thought > > out as a nice abstraction layer. If the two are tightly connected, > > this might mean that what we actually want here is to reintegrate > > the two components into a single driver with a much simpler > > RX and TX path that handles the checksumming and aggregation > > of data packets directly as it passes them from the network > > stack into the hardware. > > In general, I agree. And Dan suggested combining the rmnet > and IPA drivers into a single driver when I posted the RFC > code last year. There's still the notion of switching back > ends that I mentioned earlier; if that's indeed an important > feature it might argue for keeping rmnet as a shim layer. > But I'm really not the person to comment on this. Someone > (Subash?) from Qualcomm might be able to provide better answers. > > > Always passing data from one netdev to another both ways > > sounds like it introduces both direct CPU overhead, and > > problems with flow control when data gets buffered inbetween. > > My impression is the rmnet driver is a pretty thin layer, > so the CPU overhead is probably not that great (though > deaggregating a message is expensive). I agree with you > on the flow control. > > > The intermediate buffer here acts like a router that must > > pass data along or randomly drop packets when the consumer > > can't keep up with the producer. > > I haven't reviewed the rmnet code in any detail, but you > may be right. > > -Alex > > > Arnd > >
On 6/3/19 9:54 AM, Dan Williams wrote: >> To be perfectly honest, at first I thought having IPA use rmnet >> was a cargo cult thing like Dan suggested, because I didn't see > To be clear I only meant cargo-culting the naming, not any > functionality. Clearly IPA/rmnet/QMAP are pretty intimately connected > at this point. But this goes back to whether IPA needs a netdev itself > or whether you need an rmnet device created on top. If the former then > I'd say no cargo-culting, if the later then it's a moot point because > the device name will be rmnet%d anyway. OK I thought you weren't sure why rmnet was a layer at all. As I said, I didn't have a very good understanding of why it was even needed when I first started working on this. I can't (or won't) comment right now on whether IPA needs its own netdev for rmnet to use. The IPA endpoints used for the modem network interfaces are enabled when the netdev is opened and disabled when closed. Outside of that, TX and RX are pretty much immediately passed through to the layer below or above. IPA currently has no other net device operations. -Alex
On Mon, 2019-06-03 at 10:52 -0500, Alex Elder wrote: > On 6/3/19 9:54 AM, Dan Williams wrote: > > > To be perfectly honest, at first I thought having IPA use rmnet > > > was a cargo cult thing like Dan suggested, because I didn't see > > To be clear I only meant cargo-culting the naming, not any > > functionality. Clearly IPA/rmnet/QMAP are pretty intimately > > connected > > at this point. But this goes back to whether IPA needs a netdev > > itself > > or whether you need an rmnet device created on top. If the former > > then > > I'd say no cargo-culting, if the later then it's a moot point > > because > > the device name will be rmnet%d anyway. > > OK I thought you weren't sure why rmnet was a layer at all. As I > said, I didn't have a very good understanding of why it was even > needed when I first started working on this. No problem. > I can't (or won't) comment right now on whether IPA needs its own > netdev for rmnet to use. The IPA endpoints used for the modem > network interfaces are enabled when the netdev is opened and > disabled when closed. Outside of that, TX and RX are pretty > much immediately passed through to the layer below or above. > IPA currently has no other net device operations. I don't really have issues with the patchset underneath the netdev layer. I'm interested in how the various bits present themselves to userspace, which is why I am trying to tie this in with Johannes' conversation about WWAN devices, netdevs, channels, and how the various drivers present API for creating data channels that map to different modem contexts. So let me rephrase. If the control plane has set up the default context and sent a QMI Start Network message (or the network attached the default one) and the resulting IP details are applied to the IPA netdev can things just start sending data? Or do we need to create an rmnet on top to get that working? Dan
>> I can't (or won't) comment right now on whether IPA needs its own >> netdev for rmnet to use. The IPA endpoints used for the modem >> network interfaces are enabled when the netdev is opened and >> disabled when closed. Outside of that, TX and RX are pretty >> much immediately passed through to the layer below or above. >> IPA currently has no other net device operations. > > I don't really have issues with the patchset underneath the netdev > layer. I'm interested in how the various bits present themselves to > userspace, which is why I am trying to tie this in with Johannes' > conversation about WWAN devices, netdevs, channels, and how the various > drivers present API for creating data channels that map to different > modem contexts. > > So let me rephrase. If the control plane has set up the default context > and sent a QMI Start Network message (or the network attached the > default one) and the resulting IP details are applied to the IPA netdev > can things just start sending data? Or do we need to create an rmnet on > top to get that working? > > Dan Hi Dan All data from the hardware will have the MAP headers. We still need to create rmnet devs over the IPA netdev and use it for the data path. The IPA netdev will pass on the packets which it receives from the hardware and queue it to network stack where it will be intercepted by the rmnet rx handler.
On Mon, Jun 3, 2019 at 3:32 PM Alex Elder <elder@linaro.org> wrote: > On 6/3/19 5:04 AM, Arnd Bergmann wrote: > > On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan > > > > - What I'm worried about most here is the flow control handling on the > > transmit side. The IPA driver now uses the modern BQL method to > > control how much data gets submitted to the hardware at any time. > > The rmnet driver also uses flow control using the > > rmnet_map_command() function, that blocks tx on the higher > > level device when the remote side asks us to. > > I fear that doing flow control for a single physical device on two > > separate netdev instances is counterproductive and confuses > > both sides. > > I understand what you're saying here, and instinctively I think > you're right. > > But BQL manages the *local* interface's ability to get rid of > packets, whereas the QMAP flow control is initiated by the other > end of the connection (the modem in this case). > > With multiplexing, it's possible that one of several logical > devices on the modem side has exhausted a resource and must > ask the source of the data on the host side to suspend the > flow. Meanwhile the other logical devices sharing the physical > link might be fine, and should not be delayed by the first one. > > It is the multiplexing itself that confuses the BQL algorithm. > The abstraction obscures the *real* rates at which individual > logical connections are able to transmit data. I would assume that the real rate constantly changes, at least for wireless interfaces that are also shared with other users on the same network. BQL is meant to deal with that, at least when using a modern queuing algorithm. > Even if the multiple logical interfaces implemented BQL, they > would not get the feedback they need directly from the IPA > driver, because transmitting over the physical interface might > succeed even if the logical interface on the modem side can't > handle more data. So I think the flow control commands may be > necessary, given multiplexing. Can you describe what kind of multiplexing is actually going on? I'm still unclear about what we actually use multiple logical interfaces for here, and how they relate to one another. > The rmnet driver could use BQL, and could return NETDEV_TX_BUSY > for a logical interface when its TX flow has been stopped by a > QMAP command. That way the feedback for BQL on the logical > interfaces would be provided in the right place. > > I have no good intuition about the interaction between > two layered BQL managed queues though. Returning NETDEV_TX_BUSY is usually a bad idea as that leads to unnecessary frame drop. I do think that using BQL and the QMAP flow command on the /same/ device would be best, as that throttles the connection when either of the two algorithms wants us to slow down. The question is mainly which of the two devices that should be. Doing it in the ipa driver is probably easier to implement here, but ideally I think we'd only have a single queue visible to the network stack, if we can come up with a way to do that. > > - I was a little confused by the location of the rmnet driver in > > drivers/net/ethernet/... More conventionally, I think as a protocol > > handler it should go into net/qmap/, with the ipa driver going > > into drivers/net/qmap/ipa/, similar to what we have fo ethernet, > > wireless, ppp, appletalk, etc. > > > > - The rx_handler uses gro_cells, which as I understand is meant > > for generic tunnelling setups and takes another loop through > > NAPI to aggregate data from multiple queues, but in case of > > IPA's single-queue receive calling gro directly would be simpler > > and more efficient. > > I have been planning to investigate some of the generic GRO > stuff for IPA but was going to wait on that until the basic > code was upstream. That's ok, that part can easily be changed after the fact, as it does not impact the user interface or the general design. > > From the overall design and the rmnet Kconfig description, it > > appears as though the intention as that rmnet could be a > > generic wrapper on top of any device, but from the > > implementation it seems that IPA is not actually usable that > > way and would always go through IPA. > > As far as I know *nothing* upstream currently uses rmnet; the > IPA driver will be the first, but as Bjorn said others seem to > be on the way. I'm not sure what you mean by "IPA is not > usable that way." Currently the IPA driver assumes a fixed > configuration, and that configuration assumes the use of QMAP, > and therefore assumes the rmnet driver is layered above it. > That doesn't preclude rmnet from using a different back end. Yes, that's what I meant above: IPA can only be used through rmnet (I wrote "through IPA", sorry for the typo), but cannot be used by itself. Arnd
On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: > On Mon, Jun 3, 2019 at 3:32 PM Alex Elder <elder@linaro.org> wrote: > > On 6/3/19 5:04 AM, Arnd Bergmann wrote: > > > On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan > > > > > > - What I'm worried about most here is the flow control handling > > > on the > > > transmit side. The IPA driver now uses the modern BQL method to > > > control how much data gets submitted to the hardware at any > > > time. > > > The rmnet driver also uses flow control using the > > > rmnet_map_command() function, that blocks tx on the higher > > > level device when the remote side asks us to. > > > I fear that doing flow control for a single physical device on > > > two > > > separate netdev instances is counterproductive and confuses > > > both sides. > > > > I understand what you're saying here, and instinctively I think > > you're right. > > > > But BQL manages the *local* interface's ability to get rid of > > packets, whereas the QMAP flow control is initiated by the other > > end of the connection (the modem in this case). > > > > With multiplexing, it's possible that one of several logical > > devices on the modem side has exhausted a resource and must > > ask the source of the data on the host side to suspend the > > flow. Meanwhile the other logical devices sharing the physical > > link might be fine, and should not be delayed by the first one. > > > > It is the multiplexing itself that confuses the BQL algorithm. > > The abstraction obscures the *real* rates at which individual > > logical connections are able to transmit data. > > I would assume that the real rate constantly changes, at least > for wireless interfaces that are also shared with other users > on the same network. BQL is meant to deal with that, at least > when using a modern queuing algorithm. > > > Even if the multiple logical interfaces implemented BQL, they > > would not get the feedback they need directly from the IPA > > driver, because transmitting over the physical interface might > > succeed even if the logical interface on the modem side can't > > handle more data. So I think the flow control commands may be > > necessary, given multiplexing. > > Can you describe what kind of multiplexing is actually going on? > I'm still unclear about what we actually use multiple logical > interfaces for here, and how they relate to one another. Each logical interface represents a different "connection" (PDP/EPS context) to the provider network with a distinct IP address and QoS. VLANs may be a suitable analogy but here they are L3+QoS. In realistic example the main interface (say rmnet0) would be used for web browsing and have best-effort QoS. A second interface (say rmnet1) would be used for VOIP and have certain QoS guarantees from both the modem and the network itself. QMAP can also aggregate frames for a given channel (connection/EPS/PDP context/rmnet interface/etc) to better support LTE speeds. Dan > > The rmnet driver could use BQL, and could return NETDEV_TX_BUSY > > for a logical interface when its TX flow has been stopped by a > > QMAP command. That way the feedback for BQL on the logical > > interfaces would be provided in the right place. > > > > I have no good intuition about the interaction between > > two layered BQL managed queues though. > > Returning NETDEV_TX_BUSY is usually a bad idea as that > leads to unnecessary frame drop. > > I do think that using BQL and the QMAP flow command on > the /same/ device would be best, as that throttles the connection > when either of the two algorithms wants us to slow down. > > The question is mainly which of the two devices that should be. > Doing it in the ipa driver is probably easier to implement here, > but ideally I think we'd only have a single queue visible to the > network stack, if we can come up with a way to do that. > > > > - I was a little confused by the location of the rmnet driver in > > > drivers/net/ethernet/... More conventionally, I think as a > > > protocol > > > handler it should go into net/qmap/, with the ipa driver going > > > into drivers/net/qmap/ipa/, similar to what we have fo > > > ethernet, > > > wireless, ppp, appletalk, etc. > > > > > > - The rx_handler uses gro_cells, which as I understand is meant > > > for generic tunnelling setups and takes another loop through > > > NAPI to aggregate data from multiple queues, but in case of > > > IPA's single-queue receive calling gro directly would be > > > simpler > > > and more efficient. > > > > I have been planning to investigate some of the generic GRO > > stuff for IPA but was going to wait on that until the basic > > code was upstream. > > That's ok, that part can easily be changed after the fact, as it > does not impact the user interface or the general design. > > > > From the overall design and the rmnet Kconfig description, it > > > appears as though the intention as that rmnet could be a > > > generic wrapper on top of any device, but from the > > > implementation it seems that IPA is not actually usable that > > > way and would always go through IPA. > > > > As far as I know *nothing* upstream currently uses rmnet; the > > IPA driver will be the first, but as Bjorn said others seem to > > be on the way. I'm not sure what you mean by "IPA is not > > usable that way." Currently the IPA driver assumes a fixed > > configuration, and that configuration assumes the use of QMAP, > > and therefore assumes the rmnet driver is layered above it. > > That doesn't preclude rmnet from using a different back end. > > Yes, that's what I meant above: IPA can only be used through > rmnet (I wrote "through IPA", sorry for the typo), but cannot be > used by itself. > > Arnd
On Mon, 2019-06-03 at 13:04 -0600, Subash Abhinov Kasiviswanathan wrote: > > > I can't (or won't) comment right now on whether IPA needs its own > > > netdev for rmnet to use. The IPA endpoints used for the modem > > > network interfaces are enabled when the netdev is opened and > > > disabled when closed. Outside of that, TX and RX are pretty > > > much immediately passed through to the layer below or above. > > > IPA currently has no other net device operations. > > > > I don't really have issues with the patchset underneath the netdev > > layer. I'm interested in how the various bits present themselves to > > userspace, which is why I am trying to tie this in with Johannes' > > conversation about WWAN devices, netdevs, channels, and how the > > various > > drivers present API for creating data channels that map to > > different > > modem contexts. > > > > So let me rephrase. If the control plane has set up the default > > context > > and sent a QMI Start Network message (or the network attached the > > default one) and the resulting IP details are applied to the IPA > > netdev > > can things just start sending data? Or do we need to create an > > rmnet on > > top to get that working? > > > > Dan > > Hi Dan > > All data from the hardware will have the MAP headers. > We still need to create rmnet devs over the IPA netdev and use it > for > the > data path. > The IPA netdev will pass on the packets which it receives from the > hardware > and queue it to network stack where it will be intercepted by the > rmnet rx handler. Ok, so IPA only needs a netdev so that rmnet has something to send/receive packets to/from? This gets even closer to the discussion in "cellular modem driver APIs - take 2" from last week. Dan
On Tue, Jun 4, 2019 at 5:18 PM Dan Williams <dcbw@redhat.com> wrote: > On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: > > > > Can you describe what kind of multiplexing is actually going on? > > I'm still unclear about what we actually use multiple logical > > interfaces for here, and how they relate to one another. > > Each logical interface represents a different "connection" (PDP/EPS > context) to the provider network with a distinct IP address and QoS. > VLANs may be a suitable analogy but here they are L3+QoS. > > In realistic example the main interface (say rmnet0) would be used for > web browsing and have best-effort QoS. A second interface (say rmnet1) > would be used for VOIP and have certain QoS guarantees from both the > modem and the network itself. > > QMAP can also aggregate frames for a given channel (connection/EPS/PDP > context/rmnet interface/etc) to better support LTE speeds. Thanks, that's a very helpful explanation! Is it correct to say then that the concept of having those separate connections would be required for any proper LTE modem implementation, but the QMAP protocol (and based on that, the rmnet implementation) is Qualcomm specific and shared only among several generations of modems from that one vendor? You mentioned the need to have a common user space interface for configuration, and if the above is true, I agree that we should try to achieve that, either by ensuring rmnet is generic enough to cover other vendors (and non-QMAP clients), or by creating a new user level interface that IPA/rmnet can be adapted to. Arnd
On Tue, 2019-06-04 at 22:04 +0200, Arnd Bergmann wrote: > On Tue, Jun 4, 2019 at 5:18 PM Dan Williams <dcbw@redhat.com> wrote: > > On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: > > > Can you describe what kind of multiplexing is actually going on? > > > I'm still unclear about what we actually use multiple logical > > > interfaces for here, and how they relate to one another. > > > > Each logical interface represents a different "connection" (PDP/EPS > > context) to the provider network with a distinct IP address and > > QoS. > > VLANs may be a suitable analogy but here they are L3+QoS. > > > > In realistic example the main interface (say rmnet0) would be used > > for > > web browsing and have best-effort QoS. A second interface (say > > rmnet1) > > would be used for VOIP and have certain QoS guarantees from both > > the > > modem and the network itself. > > > > QMAP can also aggregate frames for a given channel > > (connection/EPS/PDP > > context/rmnet interface/etc) to better support LTE speeds. > > Thanks, that's a very helpful explanation! > > Is it correct to say then that the concept of having those separate > connections would be required for any proper LTE modem > implementation, > but the QMAP protocol (and based on that, the rmnet implementation) > is Qualcomm specific and shared only among several generations of > modems from that one vendor? Exactly correct. This is what Johannes is discussing in his "cellular modem APIs - take 2" thread about how this should all be organized at the driver level and I think we should figure that out before we commit to IPA-with-a-useless-netdev that requires rmnets to be created on top. That may end up being the solution but let's have that discussion. > > You mentioned the need to have a common user space interface > for configuration, and if the above is true, I agree that we should > try > to achieve that, either by ensuring rmnet is generic enough to > cover other vendors (and non-QMAP clients), or by creating a > new user level interface that IPA/rmnet can be adapted to. I would not suggest making rmnet generic; it's pretty QMAP specific (but QMAP is spoken by many many modems both SoC, USB stick, and PCIe minicard). Instead, I think what Johannes is discussing is a better approach. A kernel WWAN framework with consistent user API that rmnet/IPA/qmi_wwan/MBIM/QMI/serial/Sierra can all implement. That wouldn't affect the core packet processing of IPA/rmnet but instead: 1) when/how an rmnet device actually gets created on top of the IPA (or qmi_wwan) device AND (one of these two) a) whether IPA creates a netdev on probe OR b) whether there is some "WWAN device" kernel object which userspace interacts with create rmnet channels on top of IPA Dan
On 6/4/19 4:29 PM, Dan Williams wrote: > On Tue, 2019-06-04 at 22:04 +0200, Arnd Bergmann wrote: >> On Tue, Jun 4, 2019 at 5:18 PM Dan Williams <dcbw@redhat.com> wrote: >>> On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: >>>> Can you describe what kind of multiplexing is actually going on? >>>> I'm still unclear about what we actually use multiple logical >>>> interfaces for here, and how they relate to one another. >>> >>> Each logical interface represents a different "connection" (PDP/EPS >>> context) to the provider network with a distinct IP address and >>> QoS. >>> VLANs may be a suitable analogy but here they are L3+QoS. >>> >>> In realistic example the main interface (say rmnet0) would be used >>> for >>> web browsing and have best-effort QoS. A second interface (say >>> rmnet1) >>> would be used for VOIP and have certain QoS guarantees from both >>> the >>> modem and the network itself. >>> >>> QMAP can also aggregate frames for a given channel >>> (connection/EPS/PDP >>> context/rmnet interface/etc) to better support LTE speeds. >> >> Thanks, that's a very helpful explanation! >> >> Is it correct to say then that the concept of having those separate >> connections would be required for any proper LTE modem >> implementation, >> but the QMAP protocol (and based on that, the rmnet implementation) >> is Qualcomm specific and shared only among several generations of >> modems from that one vendor? > > Exactly correct. This is what Johannes is discussing in his "cellular > modem APIs - take 2" thread about how this should all be organized at > the driver level and I think we should figure that out before we commit > to IPA-with-a-useless-netdev that requires rmnets to be created on top. > That may end up being the solution but let's have that discussion. I looked at Johannes' message and the follow-on discussion. As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements. HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why. First, the layer of the IPA code that actually interacts with rmnet is very small--less than 3% if you simply do a word count of the source files. Arnd actually suggested eliminating the "ipa_netdev" files and merging their content elsewhere. This suggests two things: - The interface to rmnet is isolated, so the effect of whatever updates are made to support a WWAN (rather than netdev) model will be focused - The vast majority of the driver has nothing to do with that upper layer, and deals almost exclusively with managing the IPA hardware. The idea of a generic framework isn't minor, but it isn't the main focus of the IPA driver either, so I don't think it should hold it up. Second, the IPA code has been out for review recently, and has been the subject of some detailed discussion in the past few weeks. Arnd especially has invested considerable time in review and discussion. Delaying things until after a better generic model is settled on (which I'm guessing might be on the order of months) means the IPA driver will have to be submitted for review again after that, at which point it will no longer be "fresh"; it'll be a bit like starting over. Third, having the code upstream actually means the actual requirements for rmnet-over-IPA are clear and explicit. This might not be a huge deal, but I think it's better to devise a generic WWAN scheme that can refer to actual code than to do so with assumptions about what will work with rmnet (and others). As far as I know, the upstream rmnet has no other upstream back end; IPA will make it "real." I support the idea of developing a generic WWAN framework, and I can assure you I'll be involved enough to perhaps be one of the first to implement a new generic scheme. Optimistically, the IPA code itself hasn't seen much feedback for v2; maybe that means it's in good shape? Anyway, I'd obviously like to get the IPA code accepted sooner rather than later, and I think there are good reasons to do that. -Alex >>> You mentioned the need to have a common user space interface >> for configuration, and if the above is true, I agree that we should >> try >> to achieve that, either by ensuring rmnet is generic enough to >> cover other vendors (and non-QMAP clients), or by creating a >> new user level interface that IPA/rmnet can be adapted to. > > I would not suggest making rmnet generic; it's pretty QMAP specific > (but QMAP is spoken by many many modems both SoC, USB stick, and PCIe > minicard). > > Instead, I think what Johannes is discussing is a better approach. A > kernel WWAN framework with consistent user API that > rmnet/IPA/qmi_wwan/MBIM/QMI/serial/Sierra can all implement. > > That wouldn't affect the core packet processing of IPA/rmnet but > instead: > > 1) when/how an rmnet device actually gets created on top of the IPA (or > qmi_wwan) device > > AND (one of these two) > > a) whether IPA creates a netdev on probe > > OR > > b) whether there is some "WWAN device" kernel object which userspace > interacts with create rmnet channels on top of IPA > > Dan >
On 5/31/19 5:08 PM, Alex Elder wrote: > On 5/31/19 4:12 PM, Arnd Bergmann wrote: >> On Fri, May 31, 2019 at 10:47 PM Alex Elder <elder@linaro.org> wrote: >>> On 5/31/19 2:19 PM, Arnd Bergmann wrote: >>>> On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: >>>>> On 5/31/19 9:58 AM, Dan Williams wrote: >>>>>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: >>>> >>>> Does this mean that IPA can only be used to back rmnet, and rmnet >>>> can only be used on top of IPA, or can or both of them be combined >>>> with another driver to talk to instead? >>> >>> No it does not mean that. >>> >>> As I understand it, one reason for the rmnet layer was to abstract >>> the back end, which would allow using a modem, or using something >>> else (a LAN?), without exposing certain details of the hardware. >>> (Perhaps to support multiplexing, etc. without duplicating that >>> logic in two "back-end" drivers?) >>> >>> To be perfectly honest, at first I thought having IPA use rmnet >>> was a cargo cult thing like Dan suggested, because I didn't see >>> the benefit. I now see why one would use that pass-through layer >>> to handle the QMAP features. >>> >>> But back to your question. The other thing is that I see no >>> reason the IPA couldn't present a "normal" (non QMAP) interface >>> for a modem. It's something I'd really like to be able to do, >>> but I can't do it without having the modem firmware change its >>> configuration for these endpoints. My access to the people who >>> implement the modem firmware has been very limited (something >>> I hope to improve), and unless and until I can get corresponding >>> changes on the modem side to implement connections that don't >>> use QMAP, I can't implement such a thing. >> >> Why would that require firmware changes? What I was thinking >> here is to turn the bits of the rmnet driver that actually do anything >> interesting on the headers into a library module (or a header file >> with inline functions) that can be called directly by the ipa driver, >> keeping the protocol unchanged. OK, I follow up below, but now that I re-read this I think I misunderstood what you were saying. I now think you were just talking about having the QMAP parsing functionality provided by rmnet be put into a library that the IPA driver (and others) could use, rather than having them be two separate but stacked drivers. That seems like a very reasonable idea, and I don't think rmnet does anything particularly special that would preclude it. But: - I have not reviewed the rmnet driver in any detail - It's possible this wouldn't work with other back ends - This would also need to be considered as part of any effort to create a generic WWAN framework (though it wouldn't pose any problem as far as I can tell) > You know, it's possible you're right about not needing > firmware changes. But it has always been my impression > they would be needed. Here's why. Now I'm just following up on the above statement. First, what I was saying initially is that I would like to be able to configure a simple network device that does not implement any of the QMAP (rmnet) protocol. That would allow IPA to be used standalone, without any need for what the rmnet driver provides. This would suffice for the current use, because the driver currently supports *only* a single fixed-configuration data connection to the modem, and has no need for the logical channels (or aggregation) that QMAP provides. I have not done that, because it would require both the AP side and modem side endpoints to change their configuration to *not* use QMAP. I can configure the AP endpoint, but I have no control over how the modem configures its endpoint. Something about your question made me think I might have been misunderstanding that requirement though. I.e., I thought it might be possible for the local (AP side) endpoint to be configured to not use QMAP, regardless of what the modem side does. I didn't inquire, but I reviewed some documents, and I now conclude that my previous understanding was correct. I can't configure an AP endpoint to not use QMAP without also having the modem configure its corresponding endpoint to not use QMAP. The AP and modem need to agree on how their respective "connected" endpoints are configured. -Alex > It looks like this: > > GSI Channel GSI Channel > | | > ---------- v ------- v ------------- > | AP (ep)|=======| IPA |=======|(ep) Modem | > ---------- ------- ------------- > > The AP and Modem each have IPA endpoints (ep), which use GSI channels, > to communicate with the IPA. Each endpoint has configuration options > (such as checksum offload). I *thought* that the configurations of > the two endpoints need to be compatible (e.g., they need to agree on > whether they're aggregating). But with your questioning I now think > you may be right, that only the local endpoint's configuration matters. > > I will inquire further on this. I *know* that the AP and modem > exchange some information about IPA configuration, but looking more > closely that looks like it's all about the configuration of shared > IPA resources, not endpoints. > > That said, the broader design (including the user space code) > surely assumes rmnet, and I don't have any sense of what impact > changing that would make. I am sure that changing it would not > be well received. > > -Alex > >>>> Always passing data from one netdev to another both ways >>>> sounds like it introduces both direct CPU overhead, and >>>> problems with flow control when data gets buffered inbetween. >>> >>> My impression is the rmnet driver is a pretty thin layer, >>> so the CPU overhead is probably not that great (though >>> deaggregating a message is expensive). I agree with you >>> on the flow control. >> >> The CPU overhead I mean is not from executing code in the >> rmnet driver, but from passing packets through the network >> stack between the two drivers, i.e. adding each frame to >> a queue and taking it back out. I'm not sure how this ends >> up working in reality but from a first look it seems like >> we might bounce in an out of the softirq handler inbetween. >> >> Arnd >> >
On 5/30/19 10:53 PM, Alex Elder wrote: > This series presents the driver for the Qualcomm IP Accelerator (IPA). > > This is version 2 of the series. This version has addressed almost > all of the feedback received in the first version: > https://lore.kernel.org/lkml/20190512012508.10608-1-elder@linaro.org/ > More detail is included in the individual patches, but here is a > high-level summary of what's changed since then: > - Two spinlocks have been removed. > - The code for enabling and disabling endpoint interrupts has > been simplified considerably, and the spinlock is no longer > required > - A spinlock used when updating ring buffer pointers is no > longer needed. Integers indexing the ring are used instead > (and they don't even have to be atomic). > - One spinlock remains to protect list updates, but it is always > acquired using spin_lock_bh() (no more irqsave). > - Information about the queueing and completion of messages is now > supplied to the network stack in batches rather than one at a > time. > - I/O completion handling has been simplified, with the IRQ > handler now consisting mainly of disabling the interrupt and > calling napi_schedule(). > - Some comments have been updated and improved througout. > > What follows is the introduction supplied with v1 of the series. Any more feedback? The only comment that I acted on is a trivial suggestion from Dave Miller: change the types for the route_virt and filter_virt fields of the ipa structure from void pointer to u64 pointer. That required no other changes to the code. At this point I plan to post a version 3 of this series in the coming week and it will include just that one change. I might do some comment updates before then as well. But if anyone expects to provide any additional input on the code in the near term, I can delay posting v3 until that has been addressed. If this applies to you, please let me know. (No pressure; things can always wait for v4...) Thanks. -Alex > ----- > > The IPA is a component present in some Qualcomm SoCs that allows > network functions such as aggregation, filtering, routing, and NAT > to be performed without active involvement of the main application > processor (AP). > > Initially, these advanced features are disabled; the IPA driver > simply provides a network interface that makes the modem's LTE > network available to the AP. In addition, only support for the > IPA found in the Qualcomm SDM845 SoC is provided. > > This code is derived from a driver developed internally by Qualcomm. > A version of the original source can be seen here: > https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree > in the "drivers/platform/msm/ipa" directory. Many were involved in > developing this, but the following individuals deserve explicit > acknowledgement for their substantial contributions: > > Abhishek Choubey > Ady Abraham > Chaitanya Pratapa > David Arinzon > Ghanim Fodi > Gidon Studinski > Ravi Gummadidala > Shihuan Liu > Skylar Chang > > A version of this code was posted in November 2018 as an RFC. > https://lore.kernel.org/lkml/20181107003250.5832-1-elder@linaro.org/ > All feedback received was addressed. The code has undergone > considerable further rework since that time, and most of the > "future work" described then has now been completed. > > This code is available in buildable form here, based on kernel > v5.2-rc1: > remote: ssh://git@git.linaro.org/people/alex.elder/linux.git > branch: ipa-v2_kernel-v5.2-rc2 > 75adf2ac1266 arm64: defconfig: enable build of IPA code > > The branch depends on a commit now found in in net-next. It has > been cherry-picked, and (in this branch) has this commit ID: > 13c627b5a078 net: qualcomm: rmnet: Move common struct definitions to include > by > > -Alex > > Alex Elder (17): > bitfield.h: add FIELD_MAX() and field_max() > dt-bindings: soc: qcom: add IPA bindings > soc: qcom: ipa: main code > soc: qcom: ipa: configuration data > soc: qcom: ipa: clocking, interrupts, and memory > soc: qcom: ipa: GSI headers > soc: qcom: ipa: the generic software interface > soc: qcom: ipa: GSI transactions > soc: qcom: ipa: IPA interface to GSI > soc: qcom: ipa: IPA endpoints > soc: qcom: ipa: immediate commands > soc: qcom: ipa: IPA network device and microcontroller > soc: qcom: ipa: AP/modem communications > soc: qcom: ipa: support build of IPA code > MAINTAINERS: add entry for the Qualcomm IPA driver > arm64: dts: sdm845: add IPA information > arm64: defconfig: enable build of IPA code > > .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++ > MAINTAINERS | 6 + > arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 + > arch/arm64/configs/defconfig | 1 + > drivers/net/Kconfig | 2 + > drivers/net/Makefile | 1 + > drivers/net/ipa/Kconfig | 16 + > drivers/net/ipa/Makefile | 7 + > drivers/net/ipa/gsi.c | 1635 +++++++++++++++++ > drivers/net/ipa/gsi.h | 246 +++ > drivers/net/ipa/gsi_private.h | 148 ++ > drivers/net/ipa/gsi_reg.h | 376 ++++ > drivers/net/ipa/gsi_trans.c | 624 +++++++ > drivers/net/ipa/gsi_trans.h | 116 ++ > drivers/net/ipa/ipa.h | 131 ++ > drivers/net/ipa/ipa_clock.c | 297 +++ > drivers/net/ipa/ipa_clock.h | 52 + > drivers/net/ipa/ipa_cmd.c | 377 ++++ > drivers/net/ipa/ipa_cmd.h | 116 ++ > drivers/net/ipa/ipa_data-sdm845.c | 245 +++ > drivers/net/ipa/ipa_data.h | 267 +++ > drivers/net/ipa/ipa_endpoint.c | 1283 +++++++++++++ > drivers/net/ipa/ipa_endpoint.h | 97 + > drivers/net/ipa/ipa_gsi.c | 48 + > drivers/net/ipa/ipa_gsi.h | 49 + > drivers/net/ipa/ipa_interrupt.c | 279 +++ > drivers/net/ipa/ipa_interrupt.h | 53 + > drivers/net/ipa/ipa_main.c | 921 ++++++++++ > drivers/net/ipa/ipa_mem.c | 234 +++ > drivers/net/ipa/ipa_mem.h | 83 + > drivers/net/ipa/ipa_netdev.c | 251 +++ > drivers/net/ipa/ipa_netdev.h | 24 + > drivers/net/ipa/ipa_qmi.c | 402 ++++ > drivers/net/ipa/ipa_qmi.h | 35 + > drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++ > drivers/net/ipa/ipa_qmi_msg.h | 238 +++ > drivers/net/ipa/ipa_reg.h | 279 +++ > drivers/net/ipa/ipa_smp2p.c | 304 +++ > drivers/net/ipa/ipa_smp2p.h | 47 + > drivers/net/ipa/ipa_uc.c | 208 +++ > drivers/net/ipa/ipa_uc.h | 32 + > include/linux/bitfield.h | 14 + > 42 files changed, 10358 insertions(+) > create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml > create mode 100644 drivers/net/ipa/Kconfig > create mode 100644 drivers/net/ipa/Makefile > create mode 100644 drivers/net/ipa/gsi.c > create mode 100644 drivers/net/ipa/gsi.h > create mode 100644 drivers/net/ipa/gsi_private.h > create mode 100644 drivers/net/ipa/gsi_reg.h > create mode 100644 drivers/net/ipa/gsi_trans.c > create mode 100644 drivers/net/ipa/gsi_trans.h > create mode 100644 drivers/net/ipa/ipa.h > create mode 100644 drivers/net/ipa/ipa_clock.c > create mode 100644 drivers/net/ipa/ipa_clock.h > create mode 100644 drivers/net/ipa/ipa_cmd.c > create mode 100644 drivers/net/ipa/ipa_cmd.h > create mode 100644 drivers/net/ipa/ipa_data-sdm845.c > create mode 100644 drivers/net/ipa/ipa_data.h > create mode 100644 drivers/net/ipa/ipa_endpoint.c > create mode 100644 drivers/net/ipa/ipa_endpoint.h > create mode 100644 drivers/net/ipa/ipa_gsi.c > create mode 100644 drivers/net/ipa/ipa_gsi.h > create mode 100644 drivers/net/ipa/ipa_interrupt.c > create mode 100644 drivers/net/ipa/ipa_interrupt.h > create mode 100644 drivers/net/ipa/ipa_main.c > create mode 100644 drivers/net/ipa/ipa_mem.c > create mode 100644 drivers/net/ipa/ipa_mem.h > create mode 100644 drivers/net/ipa/ipa_netdev.c > create mode 100644 drivers/net/ipa/ipa_netdev.h > create mode 100644 drivers/net/ipa/ipa_qmi.c > create mode 100644 drivers/net/ipa/ipa_qmi.h > create mode 100644 drivers/net/ipa/ipa_qmi_msg.c > create mode 100644 drivers/net/ipa/ipa_qmi_msg.h > create mode 100644 drivers/net/ipa/ipa_reg.h > create mode 100644 drivers/net/ipa/ipa_smp2p.c > create mode 100644 drivers/net/ipa/ipa_smp2p.h > create mode 100644 drivers/net/ipa/ipa_uc.c > create mode 100644 drivers/net/ipa/ipa_uc.h >
Hi Alex, all, > > Exactly correct. This is what Johannes is discussing in his "cellular > > modem APIs - take 2" thread about how this should all be organized at > > the driver level and I think we should figure that out before we commit > > to IPA-with-a-useless-netdev that requires rmnets to be created on top. > > That may end up being the solution but let's have that discussion. > > I looked at Johannes' message and the follow-on discussion. Thanks :-) Sorry also, Dan had pointed me to this thread and the discussion, but I was travelling last week and not very reachable. > As I've > made clear before, my work on this has been focused on the IPA transport, > and some of this higher-level LTE architecture is new to me. But it > seems pretty clear that an abstracted WWAN subsystem is a good plan, > because these devices represent a superset of what a "normal" netdev > implements. I'm not sure I'd actually call it a superset. By themselves, these netdevs are actually completely useless to the network stack, AFAICT. Therefore, the overlap with netdevs you can really use with the network stack is pretty small? > HOWEVER I disagree with your suggestion that the IPA code should > not be committed until after that is all sorted out. In part it's > for selfish reasons, but I think there are legitimate reasons to > commit IPA now *knowing* that it will need to be adapted to fit > into the generic model that gets defined and developed. Here > are some reasons why. I can't really argue with those, though I would point out that the converse also holds - if we commit to this now, then we will have to actually keep the API offered by IPA/rmnet today, so we cannot actually remove the netdev again, even if we do migrate it to offer support for a WWAN framework in the future. > Second, the IPA code has been out for review recently, and has been > the subject of some detailed discussion in the past few weeks. Arnd > especially has invested considerable time in review and discussion. > Delaying things until after a better generic model is settled on > (which I'm guessing might be on the order of months) I dunno if it really has to be months. I think we can cobble something together relatively quickly that addresses the needs of IPA more specifically, and then extend later? But OTOH it may make sense to take a more paced approach and think about the details more carefully than we have over in the other thread so far. > Third, having the code upstream actually means the actual requirements > for rmnet-over-IPA are clear and explicit. This might not be a huge > deal, but I think it's better to devise a generic WWAN scheme that > can refer to actual code than to do so with assumptions about what > will work with rmnet (and others). As far as I know, the upstream > rmnet has no other upstream back end; IPA will make it "real." Is that really true? I had previously been told that rmnet actually does have use with a few existing drivers. If true though, then I think this would be the killer argument *in favour* of *not* merging this - because that would mean we *don't* have to actually keep the rmnet API around for all foreseeable future. > I support the idea of developing a generic WWAN framework, and I > can assure you I'll be involved enough to perhaps be one of the > first to implement a new generic scheme. Thanks! johannes
On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg <johannes@sipsolutions.net> wrote: > > As I've made clear before, my work on this has been focused on the IPA transport, > > and some of this higher-level LTE architecture is new to me. But it > > seems pretty clear that an abstracted WWAN subsystem is a good plan, > > because these devices represent a superset of what a "normal" netdev > > implements. > > I'm not sure I'd actually call it a superset. By themselves, these > netdevs are actually completely useless to the network stack, AFAICT. > Therefore, the overlap with netdevs you can really use with the network > stack is pretty small? I think Alex meant the concept of having a type of netdev with a generic user space interface for wwan and similar to a wlan device, as I understood you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/. > > HOWEVER I disagree with your suggestion that the IPA code should > > not be committed until after that is all sorted out. In part it's > > for selfish reasons, but I think there are legitimate reasons to > > commit IPA now *knowing* that it will need to be adapted to fit > > into the generic model that gets defined and developed. Here > > are some reasons why. > > I can't really argue with those, though I would point out that the > converse also holds - if we commit to this now, then we will have to > actually keep the API offered by IPA/rmnet today, so we cannot actually > remove the netdev again, even if we do migrate it to offer support for a > WWAN framework in the future. Right. The interface to support rmnet might be simple enough to keep next to what becomes the generic interface, but it will always continue to be an annoyance. > > Second, the IPA code has been out for review recently, and has been > > the subject of some detailed discussion in the past few weeks. Arnd > > especially has invested considerable time in review and discussion. > > Delaying things until after a better generic model is settled on > > (which I'm guessing might be on the order of months) > > > I dunno if it really has to be months. I think we can cobble something > together relatively quickly that addresses the needs of IPA more > specifically, and then extend later? > > But OTOH it may make sense to take a more paced approach and think > about the details more carefully than we have over in the other thread so far. I would hope that as soon as we can agree on a general approach, it would also be possible to merge a minimal implementation into the kernel along with IPA. Alex already mentioned that IPA in its current state does not actually support more than one data channel, so the necessary setup for it becomes even simpler. At the moment, the rmnet configuration in include/uapi/linux/if_link.h is almost trivial, with the three pieces of information needed being an IFLA_LINK to point to the real device (not needed if there is only one device per channel, instead of two), the IFLA_RMNET_MUX_ID setting the ID of the muxing channel (not needed if there is only one channel ?), a way to specify software bridging between channels (not useful if there is only one channel) and a few flags that I assume must match the remote end: #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) enum { IFLA_RMNET_UNSPEC, IFLA_RMNET_MUX_ID, IFLA_RMNET_FLAGS, __IFLA_RMNET_MAX, }; #define IFLA_RMNET_MAX (__IFLA_RMNET_MAX - 1) struct ifla_rmnet_flags { __u32 flags; __u32 mask; }; > > Third, having the code upstream actually means the actual requirements > > for rmnet-over-IPA are clear and explicit. This might not be a huge > > deal, but I think it's better to devise a generic WWAN scheme that > > can refer to actual code than to do so with assumptions about what > > will work with rmnet (and others). As far as I know, the upstream > > rmnet has no other upstream back end; IPA will make it "real." > > Is that really true? I had previously been told that rmnet actually does > have use with a few existing drivers. > > > If true though, then I think this would be the killer argument *in > favour* of *not* merging this - because that would mean we *don't* have > to actually keep the rmnet API around for all foreseeable future. I would agree with that. From the code I can see no other driver including the rmnet protocol header (see the discussion about moving the header to include/linux in order to merge ipa), and I don't see any other driver referencing ETH_P_MAP either. My understanding is that any driver used by rmnet would require both, but they are all out-of-tree at the moment. Arnd
On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg > <johannes@sipsolutions.net> wrote: > > > > As I've made clear before, my work on this has been focused on > > > the IPA transport, > > > and some of this higher-level LTE architecture is new to me. But > > > it > > > seems pretty clear that an abstracted WWAN subsystem is a good > > > plan, > > > because these devices represent a superset of what a "normal" > > > netdev > > > implements. > > > > I'm not sure I'd actually call it a superset. By themselves, these > > netdevs are actually completely useless to the network stack, > > AFAICT. > > Therefore, the overlap with netdevs you can really use with the > > network > > stack is pretty small? > > I think Alex meant the concept of having a type of netdev with a > generic > user space interface for wwan and similar to a wlan device, as I > understood > you had suggested as well, as opposed to a stacked device as in > rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, > ...)/. > > > > HOWEVER I disagree with your suggestion that the IPA code should > > > not be committed until after that is all sorted out. In part > > > it's > > > for selfish reasons, but I think there are legitimate reasons to > > > commit IPA now *knowing* that it will need to be adapted to fit > > > into the generic model that gets defined and developed. Here > > > are some reasons why. > > > > I can't really argue with those, though I would point out that the > > converse also holds - if we commit to this now, then we will have > > to > > actually keep the API offered by IPA/rmnet today, so we cannot > > actually > > remove the netdev again, even if we do migrate it to offer support > > for a > > WWAN framework in the future. > > Right. The interface to support rmnet might be simple enough to keep > next to what becomes the generic interface, but it will always > continue > to be an annoyance. > > > > Second, the IPA code has been out for review recently, and has > > > been > > > the subject of some detailed discussion in the past few > > > weeks. Arnd > > > especially has invested considerable time in review and > > > discussion. > > > Delaying things until after a better generic model is settled on > > > (which I'm guessing might be on the order of months) > > > > I dunno if it really has to be months. I think we can cobble > > something > > together relatively quickly that addresses the needs of IPA more > > specifically, and then extend later? > > > > But OTOH it may make sense to take a more paced approach and think > > about the details more carefully than we have over in the other > > thread so far. > > I would hope that as soon as we can agree on a general approach, it > would also be possible to merge a minimal implementation into the > kernel > along with IPA. Alex already mentioned that IPA in its current state > does > not actually support more than one data channel, so the necessary > setup for it becomes even simpler. > > At the moment, the rmnet configuration in > include/uapi/linux/if_link.h > is almost trivial, with the three pieces of information needed being > an IFLA_LINK to point to the real device (not needed if there is only > one device per channel, instead of two), the IFLA_RMNET_MUX_ID > setting the ID of the muxing channel (not needed if there is only > one channel ?), a way to specify software bridging between channels > (not useful if there is only one channel) and a few flags that I > assume > must match the remote end: > > #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) > #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) > enum { > IFLA_RMNET_UNSPEC, > IFLA_RMNET_MUX_ID, > IFLA_RMNET_FLAGS, > __IFLA_RMNET_MAX, > }; > #define IFLA_RMNET_MAX (__IFLA_RMNET_MAX - 1) > struct ifla_rmnet_flags { > __u32 flags; > __u32 mask; > }; > > > > Third, having the code upstream actually means the actual > > > requirements > > > for rmnet-over-IPA are clear and explicit. This might not be a > > > huge > > > deal, but I think it's better to devise a generic WWAN scheme > > > that > > > can refer to actual code than to do so with assumptions about > > > what > > > will work with rmnet (and others). As far as I know, the > > > upstream > > > rmnet has no other upstream back end; IPA will make it "real." > > > > Is that really true? I had previously been told that rmnet actually > > does > > have use with a few existing drivers. > > > > > > If true though, then I think this would be the killer argument *in > > favour* of *not* merging this - because that would mean we *don't* > > have > > to actually keep the rmnet API around for all foreseeable future. > > I would agree with that. From the code I can see no other driver > including the rmnet protocol header (see the discussion about moving > the header to include/linux in order to merge ipa), and I don't see > any other driver referencing ETH_P_MAP either. My understanding > is that any driver used by rmnet would require both, but they are > all out-of-tree at the moment. The general plan (and I believe Daniele Palmas was working on it) was to eventually make qmi_wwan use rmnet rather than its internal sysfs- based implementation. qmi_wwan and ipa are at essentially the same level and both could utilize rmnet on top. *That's* what I'd like to see. I don't want to see two different ways to get QMAP packets to modem firmware from two different drivers that really could use the same code. Dan
> The general plan (and I believe Daniele Palmas was working on it) was > to eventually make qmi_wwan use rmnet rather than its internal sysfs- > based implementation. qmi_wwan and ipa are at essentially the same > level and both could utilize rmnet on top. > > *That's* what I'd like to see. I don't want to see two different ways > to get QMAP packets to modem firmware from two different drivers that > really could use the same code. > > Dan qmi_wwan is based on USB and is very different from the IPA interconnect though. AFAIK, they do not have much in common (apart from sending & receiving MAP packets from hardware).
On Tue, 2019-06-11 at 10:52 -0600, Subash Abhinov Kasiviswanathan wrote: > > The general plan (and I believe Daniele Palmas was working on it) > > was > > to eventually make qmi_wwan use rmnet rather than its internal > > sysfs- > > based implementation. qmi_wwan and ipa are at essentially the same > > level and both could utilize rmnet on top. > > > > *That's* what I'd like to see. I don't want to see two different > > ways > > to get QMAP packets to modem firmware from two different drivers > > that > > really could use the same code. > > > > Dan > > qmi_wwan is based on USB and is very different from the IPA > interconnect > though. AFAIK, they do not have much in common (apart from sending & > receiving MAP packets from hardware). That is correct, they are very different drivers but as you state they send and receive MAP packets with the other end via some closer-to- hardware protocol (USB or GSI?) than QMAP. rmnet should handle muxing the QMAP, QoS, and aggregation and pass the resulting packet to the lower layer. That lower layer could be IPA or qmi_wwan, which in turn passes that QMAP packet to USB or GSI or whatever. This is typically how Linux handles clean abstractions between different protocol layers in drivers. Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas for example) where the same firmware interface can be accessed via PCI, SDIO, USB, SPI, etc. The bus-specific code is self-contained and does not creep into the upper more generic parts. Dan
On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: > On Tue, 2019-06-11 at 10:52 -0600, Subash Abhinov Kasiviswanathan wrote: > > rmnet should handle muxing the QMAP, QoS, and aggregation and pass the > resulting packet to the lower layer. That lower layer could be IPA or > qmi_wwan, which in turn passes that QMAP packet to USB or GSI or > whatever. This is typically how Linux handles clean abstractions > between different protocol layers in drivers. > > Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas for > example) where the same firmware interface can be accessed via PCI, > SDIO, USB, SPI, etc. The bus-specific code is self-contained and does > not creep into the upper more generic parts. Yes, I think that is a good model. In case of libertas, we have multiple layers inheritence from the basic device (slightly different in the implementation, but that is how it should be): struct if_cs_card { /* pcmcia specific */ struct lbs_private { /* libertas specific */ struct wireless_dev { /* 802.11 specific */ struct net_device { struct device { ... }; ... }; ... }; ... }; ... }; The outer structure gets allocated when probing the hardware specific driver, and everything below it is implemented as direct function calls into the more generic code, or as function pointers into the more specific code. The current rmnet model is different in that by design the upper layer (rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent in both directions, i.e. ipa has (almost) no knowledge of rmnet, and just has pointers to the other net_device: ipa_device net_device rmnet_port net_device I understand that the rmnet model was intended to provide a cleaner abstraction, but it's not how we normally structure subsystems in Linux, and moving to a model more like how wireless_dev works would improve both readability and performance, as you describe it, it would be more like (ignoring for now the need for multiple connections): ipa_dev rmnet_dev wwan_dev net_device Where each layer is a specialization of the next. Note: this is a common change when moving from proprietary code to upstream code. If a driver module is designed to live out of tree, there is a strong incentive to limit the number of interfaces it uses, but when it gets merged, it becomes much more flexible, as an internal interface between wwan_dev and the hardware driver(s) can be easily changed by modifying all drivers at once. Arnd
On Wed, 2019-06-12 at 10:31 +0200, Arnd Bergmann wrote: > On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: > > On Tue, 2019-06-11 at 10:52 -0600, Subash Abhinov Kasiviswanathan > > wrote: > > > > rmnet should handle muxing the QMAP, QoS, and aggregation and pass > > the > > resulting packet to the lower layer. That lower layer could be IPA > > or > > qmi_wwan, which in turn passes that QMAP packet to USB or GSI or > > whatever. This is typically how Linux handles clean abstractions > > between different protocol layers in drivers. > > > > Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas > > for > > example) where the same firmware interface can be accessed via PCI, > > SDIO, USB, SPI, etc. The bus-specific code is self-contained and > > does > > not creep into the upper more generic parts. > > Yes, I think that is a good model. In case of libertas, we have > multiple > layers inheritence from the basic device (slightly different in the > implementation, > but that is how it should be): To be clear (and I probably wasn't earlier) I wasn't talking as deep about the actual code structures as you are here but this a great discussion. I was trying to make the point that rmnet doesn't need to care about how the QMAP packets get to the device itself; it can be pretty generic so that it can be used by IPA/qmi_wwan/rmnet_smd/etc. Your points below are a great discussion though... > struct if_cs_card { /* pcmcia specific */ > struct lbs_private { /* libertas specific */ > struct wireless_dev { /* 802.11 specific */ > struct net_device { > struct device { > ... > }; > ... > }; > ... > }; > ... > }; > ... > }; > The outer structure gets allocated when probing the hardware specific > driver, and everything below it is implemented as direct function > calls > into the more generic code, or as function pointers into the more > specific > code. > > The current rmnet model is different in that by design the upper > layer > (rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent > in > both directions, i.e. ipa has (almost) no knowledge of rmnet, and > just > has pointers to the other net_device: > > ipa_device > net_device > > rmnet_port > net_device > > I understand that the rmnet model was intended to provide a cleaner > abstraction, but it's not how we normally structure subsystems in > Linux, and moving to a model more like how wireless_dev works > would improve both readability and performance, as you describe > it, it would be more like (ignoring for now the need for multiple > connections): > > ipa_dev > rmnet_dev > wwan_dev > net_device Perhaps I'm assuming too much from this diagram but this shows a 1:1 between wwan_dev and "lower" devices. What Johannes is proposing (IIRC) is something a bit looser where a wwan_dev does not necessarily provide netdev itself, but is instead the central point that various channels (control, data, gps, sim card, etc) register with. That way the wwan_dev can provide an overall view of the WWAN device to userspace, and userspace can talk to the wwan_dev to ask the lower drivers (ipa, rmnet, etc) to create new channels (netdev, tty, otherwise) when the control channel has told the modem firmware to expect one. For example, say you have told the firmware to create a new data channel with ID 5 via QMI (which the kernel is unaware of because it does not process higher-level QMI requests). Perhaps (and this is all just brainstorming) then userspace asks the wwan_dev to create a new data channel with ID 5 and a certain QoS. IPA (or rmnet because that's the data channel provider for IPA) has registered callbacks to the wwan_dev, receives this request, and creates a new rmnet_dev/net_device, and then the wwan_dev passes the ifindex back to userspace so we don't have to play the dance of "match up my request with a random netlink ADD event". The point being that since data channels aren't actually useful until the control channel agrees with the firmware that one should exist, if we have a wwan_dev that represents the entire WWAN device then we don't need the initial-but-useless net_device. Just some thoughts; Johannes can feel free to correct me at any time :) > Where each layer is a specialization of the next. Note: this is a > common change when moving from proprietary code to upstream > code. If a driver module is designed to live out of tree, there > is a strong incentive to limit the number of interfaces it uses, > but when it gets merged, it becomes much more flexible, as > an internal interface between wwan_dev and the hardware driver(s) > can be easily changed by modifying all drivers at once. Yep, I've seen this time and time again. Dan
On Wed, Jun 12, 2019 at 4:28 PM Dan Williams <dcbw@redhat.com> wrote: > On Wed, 2019-06-12 at 10:31 +0200, Arnd Bergmann wrote: > > On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: > I was trying to make the point that rmnet doesn't need to care about > how the QMAP packets get to the device itself; it can be pretty generic > so that it can be used by IPA/qmi_wwan/rmnet_smd/etc. rmnet at the moment is completely generic in that regard already, however it is implemented as a tunnel driver talking to another device rather than an abstraction layer below that driver. > > The current rmnet model is different in that by design the upper > > layer > > (rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent > > in > > both directions, i.e. ipa has (almost) no knowledge of rmnet, and > > just > > has pointers to the other net_device: > > > > ipa_device > > net_device > > > > rmnet_port > > net_device > > > > I understand that the rmnet model was intended to provide a cleaner > > abstraction, but it's not how we normally structure subsystems in > > Linux, and moving to a model more like how wireless_dev works > > would improve both readability and performance, as you describe > > it, it would be more like (ignoring for now the need for multiple > > connections): > > > > ipa_dev > > rmnet_dev > > wwan_dev > > net_device > > Perhaps I'm assuming too much from this diagram but this shows a 1:1 > between wwan_dev and "lower" devices. > > What Johannes is proposing (IIRC) is something a bit looser where a > wwan_dev does not necessarily provide netdev itself, but is instead the > central point that various channels (control, data, gps, sim card, etc) > register with. That way the wwan_dev can provide an overall view of the > WWAN device to userspace, and userspace can talk to the wwan_dev to ask > the lower drivers (ipa, rmnet, etc) to create new channels (netdev, > tty, otherwise) when the control channel has told the modem firmware to > expect one. Right, as I noted above, I simplified it a bit. We probably want to have multiple net_device instances for an ipa_dev, so there has to be a 1:n relationship instead of 1:1 at one of the intermediate levels, but it's not obvious which level that should be. In theory we could even have a single net_device instance correspond to the ipa_dev, but then have multiple IP addresses bound to it, so each IP address corresponds to a channel/queue/napi_struct, but the user visible object remains a single device. I trust that you and Johannes are more qualified than me to make the call on that point. Arnd
On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg > <johannes@sipsolutions.net> wrote: > > > > As I've made clear before, my work on this has been focused on the IPA transport, > > > and some of this higher-level LTE architecture is new to me. But it > > > seems pretty clear that an abstracted WWAN subsystem is a good plan, > > > because these devices represent a superset of what a "normal" netdev > > > implements. > > > > I'm not sure I'd actually call it a superset. By themselves, these > > netdevs are actually completely useless to the network stack, AFAICT. > > Therefore, the overlap with netdevs you can really use with the network > > stack is pretty small? > > I think Alex meant the concept of having a type of netdev with a generic > user space interface for wwan and similar to a wlan device, as I understood > you had suggested as well, as opposed to a stacked device as in > rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/. I guess. It is indeed currently modelled after the stacked devices, but those regular netdevs are inherently useful by themselves, you don't *have* to tunnel or use VLANs after all. With rmnet, the underlying netdev *isn't* useful by itself, because you're always forced to have the stacked rmnet device on top. > > > HOWEVER I disagree with your suggestion that the IPA code should > > > not be committed until after that is all sorted out. In part it's > > > for selfish reasons, but I think there are legitimate reasons to > > > commit IPA now *knowing* that it will need to be adapted to fit > > > into the generic model that gets defined and developed. Here > > > are some reasons why. > > > > I can't really argue with those, though I would point out that the > > converse also holds - if we commit to this now, then we will have to > > actually keep the API offered by IPA/rmnet today, so we cannot actually > > remove the netdev again, even if we do migrate it to offer support for a > > WWAN framework in the future. > > Right. The interface to support rmnet might be simple enough to keep > next to what becomes the generic interface, but it will always continue > to be an annoyance. Not easily, because fundamentally it requires an underlying netdev to have an ifindex, so it wouldn't just be another API to keep around (which I'd classify as an annoyance) but also a whole separate netdev that's exposed by this IPA driver, for basically this purpose only. > > I dunno if it really has to be months. I think we can cobble something > > together relatively quickly that addresses the needs of IPA more > > specifically, and then extend later? > > > > But OTOH it may make sense to take a more paced approach and think > > about the details more carefully than we have over in the other thread so far. > > I would hope that as soon as we can agree on a general approach, it > would also be possible to merge a minimal implementation into the kernel > along with IPA. Alex already mentioned that IPA in its current state does > not actually support more than one data channel, so the necessary > setup for it becomes even simpler. Interesting, I'm not even sure how the driver can stop multiple channels in the rmnet model? > At the moment, the rmnet configuration in include/uapi/linux/if_link.h > is almost trivial, with the three pieces of information needed being > an IFLA_LINK to point to the real device (not needed if there is only > one device per channel, instead of two), the IFLA_RMNET_MUX_ID > setting the ID of the muxing channel (not needed if there is only > one channel ?), a way to specify software bridging between channels > (not useful if there is only one channel) I think the MUX ID is something we *would* want, and we'd probably want a channel type as well, so as to not paint ourselves into a corner where the default ends up being whatever IPA supports right now. The software bridging is very questionable to start with, I'd advocate not supporting that at all but adding tracepoints or similar if needed for debugging instead. > and a few flags that I assume > must match the remote end: > > #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) > #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) I don't really know about these. > > If true though, then I think this would be the killer argument *in > > favour* of *not* merging this - because that would mean we *don't* have > > to actually keep the rmnet API around for all foreseeable future. > > I would agree with that. From the code I can see no other driver > including the rmnet protocol header (see the discussion about moving > the header to include/linux in order to merge ipa), and I don't see > any other driver referencing ETH_P_MAP either. My understanding > is that any driver used by rmnet would require both, but they are > all out-of-tree at the moment. I guess that would mean we have more work to do here, but it also means we don't have to support these interfaces forever. I'm not *entirely* convinced though. rmnet in itself doesn't really seem to require anything from the underlying netdev, so if there's a driver that just blindly passes things through to the hardware expecting the right configuration, we wouldn't really see it this way? OTOH, such a driver would probably blow up completely if somebody tried to use it without rmnet on top, and so it would at least have to check for ETH_P_MAP? johannes
On Wed, 2019-06-12 at 17:06 +0200, Arnd Bergmann wrote: > On Wed, Jun 12, 2019 at 4:28 PM Dan Williams <dcbw@redhat.com> wrote: > > On Wed, 2019-06-12 at 10:31 +0200, Arnd Bergmann wrote: > > > On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: > > > > I was trying to make the point that rmnet doesn't need to care about > > how the QMAP packets get to the device itself; it can be pretty generic > > so that it can be used by IPA/qmi_wwan/rmnet_smd/etc. > > rmnet at the moment is completely generic in that regard already, > however it is implemented as a tunnel driver talking to another > device rather than an abstraction layer below that driver. It doesn't really actually *do* much other than muck with the headers a small amount, but even that isn't really much. You can probably implement that far more efficiently on some devices where you have a semi-decent DMA engine that at least supports S/G. > > > I understand that the rmnet model was intended to provide a cleaner > > > abstraction, but it's not how we normally structure subsystems in > > > Linux, and moving to a model more like how wireless_dev works > > > would improve both readability and performance, as you describe > > > it, it would be more like (ignoring for now the need for multiple > > > connections): > > > > > > ipa_dev > > > rmnet_dev > > > wwan_dev > > > net_device > > > > Perhaps I'm assuming too much from this diagram but this shows a 1:1 > > between wwan_dev and "lower" devices. I guess the fuller picture would be something like ipa_dev rmnet_dev wwan_dev net_device* (i.e. with multiple net_devices) > > What Johannes is proposing (IIRC) is something a bit looser where a > > wwan_dev does not necessarily provide netdev itself, but is instead the > > central point that various channels (control, data, gps, sim card, etc) > > register with. That way the wwan_dev can provide an overall view of the > > WWAN device to userspace, and userspace can talk to the wwan_dev to ask > > the lower drivers (ipa, rmnet, etc) to create new channels (netdev, > > tty, otherwise) when the control channel has told the modem firmware to > > expect one. Yeah, that's more what I had in mind after all our discussions (will continue this below). > Right, as I noted above, I simplified it a bit. We probably want to > have multiple net_device instances for an ipa_dev, so there has > to be a 1:n relationship instead of 1:1 at one of the intermediate > levels, but it's not obvious which level that should be. > > In theory we could even have a single net_device instance correspond > to the ipa_dev, but then have multiple IP addresses bound to it, > so each IP address corresponds to a channel/queue/napi_struct, > but the user visible object remains a single device. I don't think this latter (multiple IP addresses) works well - you want a hardware specific header ("ETH_P_MAP") to carry the channel ID, without looking up the IP address and all that. But anyway, as I alluded to above, I had something like this in mind: driver_dev struct device *dev (USB, PCI, ...) net_device NA net_device NB tty TA ... (I'm cutting out the rmnet layer here for now) while having a separate that just links all the pieces together: wwan_device W ---> dev ---> NA ---> NB ---> TA So the driver is still responsible for creating the netdevs (or can of course delegate that to an "rmnet" library), but then all it also does is register the netdevs with the WWAN core like wwan_add_netdev(dev, NA) and the WWAN core would allocate the wwan_device W for this. That way, the drivers can concentrate on providing all the necessary bits, and - crucially - even *different* drivers can end up linking to the same wwan_device. For example, if you have a modem that has a multi- function USB device, then an ethernet driver might create the netdev and a tty driver might create the control channel, but if they both agree on using the right "struct device" instance, you can still get the correct wwan_device out of it all. And, in fact, some should then be wwan_maybe_add_netdev(dev, N) because the ethernet driver may not know if it attached to a modem or not, but if the control channel also attaches it's a modem for sure, with that ethernet channel attached to it. Additionally, I'm thinking API such as wwan_add(dev, &ops, opsdata) that doesn't automatically attach any channels, but provides "ops" to the core to create appropriate channels. I think this latter would be something for IPA/rmnet to use, perhaps for rmnet to offer the right ops structure. johannes
On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: [...] Looking at the flags again, > #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) This one I'm not sure I understand - seems weird to have such a fundamental thing as a *configuration* on the channel. > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) Similar here? If you have flow control you probably want to use it? > #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) This again looks like a hardware specific feature (ipv4 checksum)? Not sure why this is set by userspace. > #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) This could be set with ethtool instead, I suppose. johannes
On Mon, 2019-06-17 at 13:42 +0200, Johannes Berg wrote: > But anyway, as I alluded to above, I had something like this in mind: I forgot to state this here, but this was *heavily* influenced by discussions with Dan - many thanks to him. > driver_dev > struct device *dev (USB, PCI, ...) > net_device NA > net_device NB > tty TA > ... > > (I'm cutting out the rmnet layer here for now) > > while having a separate that just links all the pieces together: > > wwan_device W > ---> dev > ---> NA > ---> NB > ---> TA > > So the driver is still responsible for creating the netdevs (or can of > course delegate that to an "rmnet" library), but then all it also does > is register the netdevs with the WWAN core like > > wwan_add_netdev(dev, NA) [snip] So to put some more meat to this, here's an API definition for both userspace and internal APIs: diff --git a/include/net/wwan.h b/include/net/wwan.h new file mode 100644 index 000000000000..91413ec01def --- /dev/null +++ b/include/net/wwan.h @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * WWAN stack interfaces + * + * Copyright (C) 2019 Intel Corporation + * + * This defines the interaction of WWAN drivers with the WWAN stack, + * which allows userspace configuration of sessions etc. + */ +#ifndef __NET_WWAN_H +#define __NET_WWAN_H +#include <linux/list.h> +#include <linux/types.h> +#include <linux/device.h> + +struct wwan_device { + u32 id; + void *data; +/* private: */ + struct list_head list; + struct device *dev; + struct wwan_device_ops *ops; +}; + +/** + * struct wwan_netdev_config - WWAN netdevice configuration + * @pdn: PDN identifier this netdev uses + */ +struct wwan_netdev_config { + u32 pdn; +}; + +/** + * wwan_device_ops - WWAN device operations + */ +struct wwan_device_ops { + /** + * @add_netdev: Add a new netdev with the given configuration, must + * return the new netdev pointer but not call wwan_attach_netdev(). + */ + struct net_device *(*add_netdev)(struct wwan_device *wwan, + struct wwan_netdev_config *config); + /** + * @remove_netdev: remove the given netdev + */ + int (*remove_netdev)(struct wwan_device *wwan, struct net_device *dev); + + /* + * More TBD: + * - add/remove serial port channels? + * ideally with some common (library) code to support this + * Or maybe not - serial is pretty limiting. Perhaps instead: + * - support something like AF_WWAN sockets for control data + */ +}; + +/** + * wwan_add - add a WWAN device without preconfigured channels + * @dev: underlying struct device + * @ops: methods to create new channels + * @data: data for the new WWAN device + * + * Returns: a struct wwan_device pointer, or an ERR_PTR(). + */ +struct wwan_device * +wwan_add(struct device *dev, struct wwan_device_ops *ops, void *data); + +/** + * wwan_remove - remove the given WWAN device + * @wwan: WWAN device to remove. + * + * Note that the WWAN device may not be fully removed if it still has + * any channels attached, but nonetheless callers must assume that the + * pointer is no longer valid after calling this function. + */ +void wwan_remove(struct wwan_device *wwan); + +/** + * wwan_attach_netdev - attach a preconfigured netdev to the WWAN device + * @dev: struct device underlying the WWAN device + * @netdev: netdev to attach + * @config: configuration for this netdev + * @tentative: mark the attachment as tentative, don't consider this as + * part of a WWAN device unless other channels are attached as well. + * Set this to %false in drivers that know they're a WWAN device + * and to %true in generic drivers that may or may not be a WWAN + * device and want to wait for other channels. + * + * Returns: a struct wwan_device pointer or ERR_PTR(). Note that a valid + * pointer is returned even for the tentative case. + * + * Note that there's no need to detach again, this happens automatically + * when the netdev is removed. + */ +struct wwan_device * +wwan_attach_netdev(struct device *dev, struct net_device *netdev, + struct wwan_netdev_config *config, bool tentative); + +/* TBD */ +struct wwan_device * +wwan_attach_tty(struct device *dev, struct tty_port *port, bool tentative); +void wwan_detach_tty(struct device *dev, struct tty_port *port); + +#endif /* __NET_WWAN_H */ diff --git a/include/uapi/linux/wwan.h b/include/uapi/linux/wwan.h new file mode 100644 index 000000000000..af327aab881c --- /dev/null +++ b/include/uapi/linux/wwan.h @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note +/* + * WWAN generic netlink interfaces + * + * Copyright (C) 2019 Intel Corporation + * + * This defines the WWAN generic netlink family APIs for userspace + * to control WWAN devices. + */ +#ifndef __UAPI_LINUX_WWAN_H +#define __UAPI_LINUX_WWAN_H +#include <linux/types.h> + +#define WWAN_NAME "wwan" + +enum wwan_commands { + WWAN_CMD_UNSPEC, + + WWAN_CMD_GET_DEVICE, /* get or dump */ + WWAN_CMD_SET_DEVICE, /* set name or similar */ + WWAN_CMD_NEW_DEVICE, /* notification */ + WWAN_CMD_DEL_DEVICE, /* notification */ + + WWAN_CMD_GET_NETDEV, /* probably not needed - show all data in GET_DEVICE dump? */ + WWAN_CMD_SET_NETDEV, /* probably not supported? */ + WWAN_CMD_NEW_NETDEV, + WWAN_CMD_DEL_NETDEV, +}; + +/* TODO DOCS */ +enum wwan_chan_type { + WWAN_CHAN_TYPE_UNDEFINED, + WWAN_CHAN_TYPE_NETDEV, + WWAN_CHAN_TYPE_TTY, + /* ... */ +}; + +enum wwan_chan_attrs { + /** + * @WWAN_CHAN_ATTR_UNSPEC: unused/reserved + */ + WWAN_CHAN_ATTR_UNSPEC, + + /** + * @WWAN_CHAN_ATTR_TYPE: channel type according to &enum wwan_chan_type + */ + WWAN_CHAN_ATTR_TYPE, + + /** + * @WWAN_CHAN_ATTR_IFIDX: interface index (for netdev channels) + */ + WWAN_CHAN_ATTR_IFIDX, + + /* need something for TTY - major/minor number? /dev/ name? */ + + /* ... */ +}; + +enum wwan_attrs { + /** + * @WWAN_ATTR_UNSPEC: unused/reserved + */ + WWAN_ATTR_UNSPEC, + + /** + * @WWAN_ATTR_DEVICE_ID: device ID + */ + WWAN_ATTR_DEVICE_ID, + + /** + * @WWAN_ATTR_DEVICE_NAME: name of the underlying struct device + */ + WWAN_ATTR_DEVICE_NAME, + + /** + * @WWAN_ATTR_IFIDX: interface index for %WWAN_CMD_NEW_NETDEV + */ + WWAN_ATTR_IFIDX, + + /** + * @WWAN_ATTR_PDN: PDN for %WWAN_CMD_NEW_NETDEV + */ + WWAN_ATTR_PDN, + + /** + * @WWAN_ATTR_CHANNELS: nested array of channels in dump, using + * the &enum wwan_chan_attrs. + */ + WWAN_ATTR_CHANNELS, + + /* ... */ +}; + +#endif /* __UAPI_LINUX_WWAN_H */
On 6/17/19 6:28 AM, Johannes Berg wrote: > On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: >> On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg >> <johannes@sipsolutions.net> wrote: >> >>>> As I've made clear before, my work on this has been focused on the IPA transport, >>>> and some of this higher-level LTE architecture is new to me. But it >>>> seems pretty clear that an abstracted WWAN subsystem is a good plan, >>>> because these devices represent a superset of what a "normal" netdev >>>> implements. >>> >>> I'm not sure I'd actually call it a superset. By themselves, these >>> netdevs are actually completely useless to the network stack, AFAICT. >>> Therefore, the overlap with netdevs you can really use with the network >>> stack is pretty small? >> >> I think Alex meant the concept of having a type of netdev with a generic >> user space interface for wwan and similar to a wlan device, as I understood >> you had suggested as well, as opposed to a stacked device as in >> rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/. Yes, that's pretty much what I meant by "superset." We still need netdev functionality (though not between rmnet and ipa). And it sounds like we're talking about a better framework for managing the related WWAN devices that represent logical modem connections. We're discussing more than one spot in the networking stack though, so I can see why "superset" wasn't the right word. > I guess. It is indeed currently modelled after the stacked devices, but > those regular netdevs are inherently useful by themselves, you don't > *have* to tunnel or use VLANs after all. > > With rmnet, the underlying netdev *isn't* useful by itself, because > you're always forced to have the stacked rmnet device on top. Well I had mentioned earlier that I thought IPA could present just a single non-rmnet interface that could be used "directly" (i.e., without rmnet). But that would be a sort of hard-wired thing, and would not be part of the general WWAN framework under discussion. >>>> HOWEVER I disagree with your suggestion that the IPA code should >>>> not be committed until after that is all sorted out. In part it's >>>> for selfish reasons, but I think there are legitimate reasons to >>>> commit IPA now *knowing* that it will need to be adapted to fit >>>> into the generic model that gets defined and developed. Here >>>> are some reasons why. >>> >>> I can't really argue with those, though I would point out that the >>> converse also holds - if we commit to this now, then we will have to >>> actually keep the API offered by IPA/rmnet today, so we cannot actually >>> remove the netdev again, even if we do migrate it to offer support for a >>> WWAN framework in the future. >> >> Right. The interface to support rmnet might be simple enough to keep >> next to what becomes the generic interface, but it will always continue >> to be an annoyance. > > Not easily, because fundamentally it requires an underlying netdev to > have an ifindex, so it wouldn't just be another API to keep around > (which I'd classify as an annoyance) but also a whole separate netdev > that's exposed by this IPA driver, for basically this purpose only. > >>> I dunno if it really has to be months. I think we can cobble something >>> together relatively quickly that addresses the needs of IPA more >>> specifically, and then extend later? >>> >>> But OTOH it may make sense to take a more paced approach and think >>> about the details more carefully than we have over in the other thread so far. >> >> I would hope that as soon as we can agree on a general approach, it >> would also be possible to merge a minimal implementation into the kernel >> along with IPA. Alex already mentioned that IPA in its current state does >> not actually support more than one data channel, so the necessary >> setup for it becomes even simpler. > > Interesting, I'm not even sure how the driver can stop multiple channels > in the rmnet model? Here's a little background. The IPA driver was very large, and in an effort to have an initial driver that was more easily accepted upstream, it was carved down to support a single, very simple use case. It supports only a single channel for carrying network data, and does not expose any of the IPA's other capabilities like filtering and routing (and multiplexing). Originally the IPA code had an IOCTL interface for adding and removing multiplexed channel IDs, but the simplified use case expected only one channel to be used. IOCTLs had to be removed to make the code acceptable for upstream, and again to simplify things, we went with a hard-wired configuration, with a single channel with an assumed set of features in use (TCP offload, basically). Once upstream, we planned to add back features in layers, including adding a netlink interface to control things like managing multiplexed channels. The overall design assumed that the IPA connection between the modem and AP was carrying QMAP protocol though. And the rmnet driver is designed to parse and handle that, so for the design I started with the use of the rmnet driver made sense: it is a shim layer that takes care of rmnet multiplexing and aggregation (and checksum offload). So getting back to your question, the IPA in its current form only has a single "multiplexed" channel carried over the connection between the AP and modem. Previously (and in the future) there was a way to add or remove channels. >> At the moment, the rmnet configuration in include/uapi/linux/if_link.h >> is almost trivial, with the three pieces of information needed being >> an IFLA_LINK to point to the real device (not needed if there is only >> one device per channel, instead of two), the IFLA_RMNET_MUX_ID >> setting the ID of the muxing channel (not needed if there is only >> one channel ?), a way to specify software bridging between channels >> (not useful if there is only one channel) > > I think the MUX ID is something we *would* want, and we'd probably want > a channel type as well, so as to not paint ourselves into a corner where > the default ends up being whatever IPA supports right now. Agreed. > The software bridging is very questionable to start with, I'd advocate > not supporting that at all but adding tracepoints or similar if needed > for debugging instead. To be honest I don't understand the connection between software bridging and debugging, but that's OK. I'm a fan of tracepoints and have always intended to make use of them in the IPA driver. >> and a few flags that I assume >> must match the remote end: >> >> #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) >> #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) >> #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) >> #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) > > I don't really know about these. The hardware can aggregate multiple packets received from the modem into a single buffer, which the rmnet driver is then able to deaggregate. This feature is supposed to help performance but I've always been a little skeptical because it also comes at a cost. This is used as a flag in an rmnet (QMAP) header, which to me seems a little odd. (There should be a distinction between flags needed in a message header and flags that represent properties of a connection or channel.) I believe the only QMAP commands are for doing essentially XON/XOFF flow control on a single channel. In the course of the e-mail discussion in the past few weeks I've come to see why that would be necessary. The checksum offload is done differently, depending on whether it's ingress (download from modem) or egress. For egress, a header is inserted that describes what the hardware should checksum and where it should place the result. For ingress, the hardware appends a trailer that contains information about the computed checksum values. The rmnet driver is currently responsible for inserting the header and parsing the trailer. I'm probably missing something, but I think the checksum offload could be handled by the IPA driver rather than rmnet. It seems to be an add-on that is completely independent of the multiplexing and aggregation capabilities that QMAP provides. >>> If true though, then I think this would be the killer argument *in >>> favour* of *not* merging this - because that would mean we *don't* have >>> to actually keep the rmnet API around for all foreseeable future. This is because it's a user space API? If so I now understand what you mean. As Arnd said (below) this is designed in the way out-of-tree code works and expects. I don't want to advocate for breaking that, but if a general model that supports what's required can be used, I'll adapt the IPA code to suit that. My goal continues to be getting a baseline IPA driver accepted upstream as soon as possible, so I can then start building on that foundation. -Alex >> I would agree with that. From the code I can see no other driver >> including the rmnet protocol header (see the discussion about moving >> the header to include/linux in order to merge ipa), and I don't see >> any other driver referencing ETH_P_MAP either. My understanding >> is that any driver used by rmnet would require both, but they are >> all out-of-tree at the moment. > > I guess that would mean we have more work to do here, but it also means > we don't have to support these interfaces forever. > > I'm not *entirely* convinced though. rmnet in itself doesn't really seem > to require anything from the underlying netdev, so if there's a driver > that just blindly passes things through to the hardware expecting the > right configuration, we wouldn't really see it this way? > > OTOH, such a driver would probably blow up completely if somebody tried > to use it without rmnet on top, and so it would at least have to check > for ETH_P_MAP? > > johannes >
On 6/17/19 6:42 AM, Johannes Berg wrote: > On Wed, 2019-06-12 at 17:06 +0200, Arnd Bergmann wrote: >> On Wed, Jun 12, 2019 at 4:28 PM Dan Williams <dcbw@redhat.com> wrote: >>> On Wed, 2019-06-12 at 10:31 +0200, Arnd Bergmann wrote: >>>> On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: >>> >>> I was trying to make the point that rmnet doesn't need to care about >>> how the QMAP packets get to the device itself; it can be pretty generic >>> so that it can be used by IPA/qmi_wwan/rmnet_smd/etc. >> >> rmnet at the moment is completely generic in that regard already, >> however it is implemented as a tunnel driver talking to another >> device rather than an abstraction layer below that driver. > > It doesn't really actually *do* much other than muck with the headers a > small amount, but even that isn't really much. > > You can probably implement that far more efficiently on some devices > where you have a semi-decent DMA engine that at least supports S/G. If it had a well-defined way of creating new channels to be multiplexed over the connection to the modem, the IPA driver (rather than the rmnet driver) could present network interfaces for each and perform the multiplexing. As I think Arnd suggested, this could at least partially be done with library code (to be shared with other "back-end" interfaces) rather than using a layered driver. This applies to aggregation, channel flow control, and checksum offload as well. But I'm only familiar with IPA; I don't know whether the above statements make any sense for other "back-end" drivers. >>>> I understand that the rmnet model was intended to provide a cleaner >>>> abstraction, but it's not how we normally structure subsystems in >>>> Linux, and moving to a model more like how wireless_dev works >>>> would improve both readability and performance, as you describe >>>> it, it would be more like (ignoring for now the need for multiple >>>> connections): >>>> >>>> ipa_dev >>>> rmnet_dev >>>> wwan_dev >>>> net_device >>> >>> Perhaps I'm assuming too much from this diagram but this shows a 1:1 >>> between wwan_dev and "lower" devices. > > I guess the fuller picture would be something like > > ipa_dev > rmnet_dev > wwan_dev > net_device* > > (i.e. with multiple net_devices) > >>> What Johannes is proposing (IIRC) is something a bit looser where a >>> wwan_dev does not necessarily provide netdev itself, but is instead the >>> central point that various channels (control, data, gps, sim card, etc) >>> register with. That way the wwan_dev can provide an overall view of the >>> WWAN device to userspace, and userspace can talk to the wwan_dev to ask >>> the lower drivers (ipa, rmnet, etc) to create new channels (netdev, >>> tty, otherwise) when the control channel has told the modem firmware to >>> expect one. > > Yeah, that's more what I had in mind after all our discussions (will > continue this below). This is great. The start of a more concrete discussion of the pieces that are missing... >> Right, as I noted above, I simplified it a bit. We probably want to >> have multiple net_device instances for an ipa_dev, so there has >> to be a 1:n relationship instead of 1:1 at one of the intermediate >> levels, but it's not obvious which level that should be. >> >> In theory we could even have a single net_device instance correspond >> to the ipa_dev, but then have multiple IP addresses bound to it, >> so each IP address corresponds to a channel/queue/napi_struct, >> but the user visible object remains a single device. > > I don't think this latter (multiple IP addresses) works well - you want > a hardware specific header ("ETH_P_MAP") to carry the channel ID, > without looking up the IP address and all that. I agree with this. It's not just multiple IP addresses for an interface, it really is multiplexed--with channel ids. It's another addressing parameter orthogonal to the IP space. > But anyway, as I alluded to above, I had something like this in mind: > > driver_dev > struct device *dev (USB, PCI, ...) > net_device NA > net_device NB > tty TA > ... > > (I'm cutting out the rmnet layer here for now) > > while having a separate that just links all the pieces together: > > wwan_device W > ---> dev > ---> NA > ---> NB > ---> TA > > So the driver is still responsible for creating the netdevs (or can of > course delegate that to an "rmnet" library), but then all it also does > is register the netdevs with the WWAN core like > > wwan_add_netdev(dev, NA) > > and the WWAN core would allocate the wwan_device W for this. That would be nice. I believe you're saying that (in my case) the IPA driver creates and owns the netdevices. But I think the IPA driver would register with the WWAN core as a "provider," and then the WWAN core would subsequently request that it instantiate netdevices to represent channels on demand (rather than registering them). > That way, the drivers can concentrate on providing all the necessary > bits, and - crucially - even *different* drivers can end up linking to > the same wwan_device. For example, if you have a modem that has a multi- > function USB device, then an ethernet driver might create the netdev and > a tty driver might create the control channel, but if they both agree on > using the right "struct device" instance, you can still get the correct > wwan_device out of it all. > > And, in fact, some should then be > > wwan_maybe_add_netdev(dev, N) > > because the ethernet driver may not know if it attached to a modem or > not, but if the control channel also attaches it's a modem for sure, > with that ethernet channel attached to it. > > Additionally, I'm thinking API such as > > wwan_add(dev, &ops, opsdata) > > that doesn't automatically attach any channels, but provides "ops" to > the core to create appropriate channels. I think this latter would be > something for IPA/rmnet to use, perhaps for rmnet to offer the right ops > structure. Yes, that's more like what I meant above. I see you're thinking as you write... -Alex > > johannes >
On Tue, Jun 18, 2019 at 3:16 PM Alex Elder <elder@linaro.org> wrote: > On 6/17/19 6:28 AM, Johannes Berg wrote: > > On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > > I'm probably missing something, but I think the checksum > offload could be handled by the IPA driver rather than > rmnet. It seems to be an add-on that is completely > independent of the multiplexing and aggregation capabilities > that QMAP provides. My best guess is that it is part of rmnet simply because this can be done in a generic way for any qmap based back-end, and rmnet was intended as the abstraction for qmap. A better implementation of the checksumming might be to split it out into a library that is in turn used by qmap drivers. Since this should be transparent to the user interface, it can be moved there later. > >>> If true though, then I think this would be the killer argument *in > >>> favour* of *not* merging this - because that would mean we *don't* have > >>> to actually keep the rmnet API around for all foreseeable future. > > This is because it's a user space API? If so I now understand > what you mean. Yes, I think agreeing on the general user interface is (as usual) the one thing that has to be done as a prerequisite. I had originally hoped that by removing the ioctl interface portion of the driver, this could be avoided, but that was before I had any idea on the upper layers. Arnd
On 6/17/19 7:14 AM, Johannes Berg wrote: > On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > > [...] > > Looking at the flags again, I sort of talked about this in my message a little earlier, but I now see I was partially mistaken. I thought these flags were used in messages but they're real device ("port") configuration flags. >> #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) > > This one I'm not sure I understand - seems weird to have such a > fundamental thing as a *configuration* on the channel. Let me use the term "connection" to refer to the single pathway that carries data between the AP and modem. And "channel" to refer to one of several multiplexed data streams carried over that connection. (If there's better terminology, please say so; I just want to be clear in what I'm talking about.) Deaggregation is a connection property, not a channel property. And it looks like that's exactly how it's used in the rmnet driver. The hardware is capable of aggregating QMAP packets arriving on a connection into a single buffer, so this provides a way of requesting it do that. >> #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > > Similar here? If you have flow control you probably want to use it? I agree with that, though perhaps there are cases where it is pointless, or can't be supported, so one might want to simply *not* implement/advertise the feature. I don't know. >> #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) > > This again looks like a hardware specific feature (ipv4 checksum)? Not > sure why this is set by userspace. > >> #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) > > This could be set with ethtool instead, I suppose. As I said in my earlier message, I think I concur about this. I think the IPA driver could easily hide the checksum offload capability, and if it can truly be controlled as needed using existing methods there's no need to encumber the WWAN framework with it. -Alex > johannes >
On 6/17/19 7:25 AM, Johannes Berg wrote: > On Mon, 2019-06-17 at 13:42 +0200, Johannes Berg wrote: > >> But anyway, as I alluded to above, I had something like this in mind: > > I forgot to state this here, but this was *heavily* influenced by > discussions with Dan - many thanks to him. Thanks for getting even more concrete with this. Code is the most concise way of describing things, once the general ideas seem to be coming together. I'm not going to comment on the specific code bits, but I have some more general questions and comments on the design. Some of these are simply due to my lack of knowledge of how WWAN/modem interactions normally work. First, a few terms (correct or improve as you like): - WWAN device is a hardware device (like IPA) that presents a connection between AP and modem, and presents an interface that allows the use of that connection to be managed. - WWAN netdevice represents a Linux network interface, with its operations and queues, etc., but implements a standardized set of WWAN-specific operations. It represents a logical ' channel whose data is multiplexed over the WWAN device. - WWAN channel is a user space abstraction that corresponds with a WWAN netdevice (but I'm not clear on all the ways they differ or interact). - The WWAN core is kernel code that presents abstractions for WWAN devices and netdevices, so they can be managed in a generic way. It is for configuration and communication and is not at all involved in the data path. You're saying that the WWAN driver space calls wwan_add() to register itself as a new WWAN device. You're also saying that a WWAN device "attaches" a WWAN netdevice, which is basically notifying the WWAN core that the new netdev/channel is available for use. - I trust that a "tentative" attachement is necessary. But I'm not sure what makes it transition into becoming a "real" one, or how that event gets communicated. Some questions: - What causes a new channel to be created? Is it initiated by the WWAN device driver? Does the modem request that it get created? User space? Both? - What causes a created channel to be removed? - You distinguish between attaching a netdevice and (what I'll call) activating it. What causes activation? - How are the attributes of a WWAN device or channel set, or communicated? - Are there any attributes that are only optionally supported, and if so, how are the supported ones communicated? - Which WWAN channel attributes must be set *before* the channel is activated, and can't be changed? Are there any that can be changed dynamically? And while the whole point of this is to make things generic, it might be nice to have a way to implement a new feature before it can be "standardized". Thanks. -Alex PS I don't want to exclude anybody but we could probably start a different mail chain on this topic... >> driver_dev >> struct device *dev (USB, PCI, ...) >> net_device NA >> net_device NB >> tty TA >> ... >> . . .
On Tue, 2019-06-18 at 10:20 -0500, Alex Elder wrote: > On 6/17/19 7:25 AM, Johannes Berg wrote: > > On Mon, 2019-06-17 at 13:42 +0200, Johannes Berg wrote: > > > > > But anyway, as I alluded to above, I had something like this in > > > mind: > > > > I forgot to state this here, but this was *heavily* influenced by > > discussions with Dan - many thanks to him. > > Thanks for getting even more concrete with this. Code is the > most concise way of describing things, once the general ideas > seem to be coming together. > > I'm not going to comment on the specific code bits, but I have > some more general questions and comments on the design. Some > of these are simply due to my lack of knowledge of how WWAN/modem > interactions normally work. > > First, a few terms (correct or improve as you like): > - WWAN device is a hardware device (like IPA) that presents a > connection between AP and modem, and presents an interface > that allows the use of that connection to be managed. > - WWAN netdevice represents a Linux network interface, with its > operations and queues, etc., but implements a standardized > set of WWAN-specific operations. It represents a logical > ' channel whose data is multiplexed over the WWAN device. > - WWAN channel is a user space abstraction that corresponds > with a WWAN netdevice (but I'm not clear on all the ways > they differ or interact). When Johannes and I have talked about "WWAN channel" we mean a control or data or other channel. That could be QMI, AT, MBIM control, GPS, PCSC, QMAP, MBIM data, PPP TTY, DM/DIAG, CDC-ETHER, CDC-NCM, Sierra HIP, etc. Or even voice-call audio :) A netdev is a Linux abstraction of a WWAN *data* channel, be that QMI or CDC-ETHER or whatever. > - The WWAN core is kernel code that presents abstractions > for WWAN devices and netdevices, so they can be managed > in a generic way. It is for configuration and communication > and is not at all involved in the data path. > > You're saying that the WWAN driver space calls wwan_add() > to register itself as a new WWAN device. > > You're also saying that a WWAN device "attaches" a WWAN > netdevice, which is basically notifying the WWAN core > that the new netdev/channel is available for use. > - I trust that a "tentative" attachement is necessary. But > I'm not sure what makes it transition into becoming a > "real" one, or how that event gets communicated. Linux usually tries to keep drivers generic and focused; each driver is written for a specific function. For example, a USB device usually provides multiple USB interfaces which will be bound to different Linux drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc. These drivers are often generic and we may not have enough information in one driver to know that the parent of this interface is a WWAN device. But another driver might. Since probing is asynchronous we may have cdc-acm bind to a device and provide a TTY before cdc-ether (which does know it's a WWAN) binds and provides the netdevice. > Some questions: > - What causes a new channel to be created? Is it initiated > by the WWAN device driver? Does the modem request that > it get created? User space? Both? Either created at driver bind time in the kernel (usually control channels) or initiated by userspace when the WWAN management process has coordinated with the firmware for another channel. Honestly userspace should probably always create the netdevices (since they are always useless until userspace coordinates with the firmware about them) but that's not how things are yet. [ A concrete example... Assume a QMI device has an existing packet data connection which is abstracted by a netdevice on the Linux side. Now the WWAN management daemon wants to create a second packet data connection with a different APN (maybe an MMS connection, maybe a VOIP one, maybe an IPv6). It sends a WDS Start Network request to the modem firmware and receives a new QMI Packet Data Handle. The management daemon must somehow get a netdevice associated with this new Packet Data Handle. It would ask the WWAN kernel device to create a new data channel with the PDH, and would get back the ifindex of that netdevice which it would configure with the IP that it gets from the firmware via the WDS Get Current Settings QMI request. The WWAN device would forward the request down to IPA (or rmnet) which would then create the netdevice using the PDH as the QMAP MUX ID for that netdevice's traffic.] > - What causes a created channel to be removed? Driver removal, userspace WWAN daemon terminating the packet data connection which the channel represents, the modem terminating the packet data connection (eg network initiated disconnect), etc. > - You distinguish between attaching a netdevice and (what > I'll call) activating it. What causes activation? Can you describe what you mean by "activating"? Do you mean successfully TX/RX packets via the netdev and the outside world? I read "attach" here as simply associating an existing netdev with the "parent" WWAN device. A purely Linux operation that is only book- keeping and may not have any interaction with the modem. > - How are the attributes of a WWAN device or channel set, > or communicated? Via netlink attributes when userspace asks the WWAN device to create a new channel. In the control methods I've seen, only userspace really knows the channel identifier that it and the modem have agreed on (eg what the MUX ID in the QMAP header would be, or the MBIM Session ID). > - Are there any attributes that are only optionally supported, > and if so, how are the supported ones communicated? Yeah, capabilities would be important here and I don't think Johannes accounted for that yet. > - Which WWAN channel attributes must be set *before* the > channel is activated, and can't be changed? Are there any > that can be changed dynamically? I would assume userspace must pass the agreed identifier (QMUX ID, MBIM session ID, etc) when creating the channel and that wouldn't change. I think a world where you can dynamically change the MUX ID/SessionID/etc is a more complicated one. Things like QoS could change but I don't recall if modems allow that; eg does a +CGEQOS (or equivalent QMI WDS LTE QoS Parameters request) take effect while the bearer is active, or is it only respected on bearer creation? > And while the whole point of this is to make things generic, > it might be nice to have a way to implement a new feature > before it can be "standardized". That would be nice, but I'd rather have the conversation about if/how to standardize things before they make it into the kernel and have their API set in stone... which is how we ended up with 5 ways of doing the same thing already. Dan > Thanks. > > -Alex > > PS I don't want to exclude anybody but we could probably start > a different mail chain on this topic... > > > > driver_dev > > > struct device *dev (USB, PCI, ...) > > > net_device NA > > > net_device NB > > > tty TA > > > ... > > > > > . . .
Just to add to Dan's response, I think he's captured our discussions and thoughts well. > First, a few terms (correct or improve as you like): Thanks for defining, we don't do that nearly often enough. > - WWAN device is a hardware device (like IPA) that presents a > connection between AP and modem, and presents an interface > that allows the use of that connection to be managed. Yes. But I was actually thinking of a "wwan_dev" to be a separate structure, not *directly* owned by a single driver and used to represent the hardware like a (hypothetical) "struct ipa_dev". > - WWAN netdevice represents a Linux network interface, with its > operations and queues, etc., but implements a standardized > set of WWAN-specific operations. It represents a logical > ' channel whose data is multiplexed over the WWAN device. I'm not sure I'd asy it has much WWAN-specific operations? But yeah, I guess it might. > - WWAN channel is a user space abstraction that corresponds > with a WWAN netdevice (but I'm not clear on all the ways > they differ or interact). As Dan said, this could be a different abstraction than a netdevice, like a TTY, etc. > - The WWAN core is kernel code that presents abstractions > for WWAN devices and netdevices, so they can be managed > in a generic way. It is for configuration and communication > and is not at all involved in the data path. > > You're saying that the WWAN driver space calls wwan_add() > to register itself as a new WWAN device. Assuming it knows that it is in fact a WWAN device, like IPA. > You're also saying that a WWAN device "attaches" a WWAN > netdevice, which is basically notifying the WWAN core > that the new netdev/channel is available for use. > - I trust that a "tentative" attachement is necessary. But > I'm not sure what makes it transition into becoming a > "real" one, or how that event gets communicated. I think Dan explained this one well. This wasn't actually on my radar until he pointed it out. Really this only exists with USB devices that appear as multiple functions (ethernet, tty, ...) but still represent a single WWAN device, with each function not necessarily being aware of that since it's just a function driver. Hopefully at least one of the function drivers will be able to figure it out, and then we can combine all of the functions into the WWAN device abstraction. [snip - Dan's explanations are great] Dan also said: > > I read "attach" here as simply associating an existing netdev with the > > "parent" WWAN device. A purely Linux operation that is only book- > > keeping and may not have any interaction with the modem. Now I'm replying out of thread, but yes, that's what I had in mind. What I meant by attaching (in this case) is just that you actually mark that it is (or might be, if tentatively attached) part of a WWAN device. > - Are there any attributes that are only optionally supported, > and if so, how are the supported ones communicated? As Dan said, good point. I hadn't really considered that for now. I sort of know that we need it, but for the sake of simplicity decided to elide it for now. I'm just not sure what really are needed, and netlink attributes make adding them (and discovering the valid ones) pretty easy in the future, when a need arises. > - Which WWAN channel attributes must be set *before* the > channel is activated, and can't be changed? Are there any > that can be changed dynamically? It's a good question. I threw a "u32 pdn" in there, but I'm not actually sure that's what you *really* need? Maybe the modem and userspace just agree on some arbitrary "session identifier"? Dan mentions "MUX ID" or "MBIM Session ID", maybe there really is no good general term for this and we should just call it a "session identifier" and agree that it depends on the control protocol (MBIM vs. QMI vs. ...)? > And while the whole point of this is to make things generic, > it might be nice to have a way to implement a new feature > before it can be "standardized". Not sure I understand this? FWIW, I actually came to this because we want to upstream a driver for an Intel modem, but ... can't really make up our mind on whether or not to use VLAN tags, something like rmnet (but we obviously cannot use rmnet, so that'd be another vendor specific interface like rmnet), or sysfs, or any of the other methods we have today ... :-) johannes
On Tue, 2019-06-18 at 08:45 -0500, Alex Elder wrote: > If it had a well-defined way of creating new channels to be > multiplexed over the connection to the modem, the IPA driver > (rather than the rmnet driver) could present network interfaces > for each and perform the multiplexing. Right. That's what I was thinking of. I actually expect this to fare much better going forward with 5G around the corner, since you'll want to eventually take advantage of multi- queue TX or RSS for RX, queue size control and what not, as speeds increase. Because of these things I think the whole "layered netdev" approach is actually *wrong* rather than just inconvenient. In particular, in the Intel driver, you're going to have multiple hardware queues, one for each ongoing session. This means that multiplexing it over a layered netdev like rmnet or something like VLAN (which some driver does too) actually prevents us from doing this properly - it means we need to implement ndo_select_queue() and multiple queues on the underlying netdev etc., and then we no longer have the ability to use actual multi-queue. It becomes messy very very quickly. > As I think Arnd > suggested, this could at least partially be done with library > code (to be shared with other "back-end" interfaces) rather > than using a layered driver. This applies to aggregation, > channel flow control, and checksum offload as well. Right. > But I'm only familiar with IPA; I don't know whether the above > statements make any sense for other "back-end" drivers. I think they do, in different ways. Intel probably wouldn't have a library - there isn't actually much of a MUX header because there are different hardware queues for the different sessions. > This is great. The start of a more concrete discussion of the > pieces that are missing... :-) I think I said before - it should be pretty easy to mold some code around the API I proposed there and have something reasonably functional soon. > That would be nice. I believe you're saying that (in my case) > the IPA driver creates and owns the netdevices. Yes. > But I think the IPA driver would register with the WWAN core as > a "provider," and then the WWAN core would subsequently request > that it instantiate netdevices to represent channels on demand > (rather than registering them). Yeah, I guess you could call it that way. Really there are two possible ways (and they intersect to some extent). One is the whole multi-function device, where a single WWAN device is composed of channels offered by actually different drivers, e.g. for a typical USB device you might have something like cdc_ether and the usb_wwan TTY driver. In this way, we need to "compose" the WWAN device similarly, e.g. by using the underlying USB device "struct device" pointer to tie it together. The other is something like IPA or the Intel modem driver, where the device is actually a single (e.g. PCIe) device and just has a single driver, but that single driver offers different channels. Now, it's not clear to me where IPA actually falls, because so far we've been talking about the IPA driver only as providing *netdevs*, not any control channels, so I'm not actually sure where the control channel is. For the Intel device, however, the control channel is definitely provided by exactly the same driver as the data channels (netdevs). "provider" is a good word, and in fact the Intel driver would also be a provider for a GNSS channel (TBD how to represent, a tty?), one or multiple debug/tracing channels, data channels (netdevs), AT command channels (mbim, ...?) (again tbd how to represent, ttys?), etc. What I showed in the header files I posted so far was the provider only having "data channel" ops (create/remove a netdev) but for each channel type we either want a new method there, or we just change the method to be something like int (*create_channel)(..., enum wwan_chan_type chan_type, ...); and simply require that the channel is attached to the wwan device with the representation-specific call (wwan_attach_netdev, wwan_attach_tty, ...). This is a bit less comfortable because then it's difficult to know what was actually created upon the request, so it's probably better to have different methods for the different types of representations (like I had - add_netdev, add_tty, ...). Note also that I said "representation-specific", while passing a "channel type", so for this we'd actually need a convention on what channel type has what kind of representation, which again gets awkward. Better to make it explicit. (And even then, we might be able to let userspace have some control, e.g. the driver might be able to create a debug channel as both a TTY or something else) johannes
On Tue, 2019-06-18 at 08:16 -0500, Alex Elder wrote: > On 6/17/19 6:28 AM, Johannes Berg wrote: > > On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > > > On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg > > > <johannes@sipsolutions.net> wrote: > > > > > > > > As I've made clear before, my work on this has been focused on the IPA transport, > > > > > and some of this higher-level LTE architecture is new to me. But it > > > > > seems pretty clear that an abstracted WWAN subsystem is a good plan, > > > > > because these devices represent a superset of what a "normal" netdev > > > > > implements. > > > > > > > > I'm not sure I'd actually call it a superset. By themselves, these > > > > netdevs are actually completely useless to the network stack, AFAICT. > > > > Therefore, the overlap with netdevs you can really use with the network > > > > stack is pretty small? > > > > > > I think Alex meant the concept of having a type of netdev with a generic > > > user space interface for wwan and similar to a wlan device, as I understood > > > you had suggested as well, as opposed to a stacked device as in > > > rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/. > > Yes, that's pretty much what I meant by "superset." We still need > netdev functionality (though not between rmnet and ipa). And it sounds > like we're talking about a better framework for managing the related > WWAN devices that represent logical modem connections. We're discussing > more than one spot in the networking stack though, so I can see why > "superset" wasn't the right word. Right, ok, gotcha. I was focused (here at least) much more on the netdevs, rather than the whole "superset" of functionality :-) > > I guess. It is indeed currently modelled after the stacked devices, but > > those regular netdevs are inherently useful by themselves, you don't > > *have* to tunnel or use VLANs after all. > > > > With rmnet, the underlying netdev *isn't* useful by itself, because > > you're always forced to have the stacked rmnet device on top. > > Well I had mentioned earlier that I thought IPA could present just > a single non-rmnet interface that could be used "directly" (i.e., > without rmnet). But that would be a sort of hard-wired thing, and > would not be part of the general WWAN framework under discussion. Oh, I guess I didn't see that (got to the thread late), but is that actually useful? It doesn't seem very useful to me since you can't actually do anything there. > Here's a little background. That's great to have :-) Let me read and comment/ask questions. > The IPA driver was very large, and in an effort to have an initial driver > that was more easily accepted upstream, it was carved down to support > a single, very simple use case. It supports only a single channel for > carrying network data, and does not expose any of the IPA's other > capabilities like filtering and routing (and multiplexing). Ok. But it *does* use (or even require using) rmnet, so it has multiple channels in a sense, no? > Originally the IPA code had an IOCTL interface for adding and removing > multiplexed channel IDs, but the simplified use case expected only one > channel to be used. What did those channels do? Create different netdevs? Something else? > IOCTLs had to be removed to make the code acceptable > for upstream, and again to simplify things, we went with a hard-wired > configuration, with a single channel with an assumed set of features > in use (TCP offload, basically). Once upstream, we planned to add back > features in layers, including adding a netlink interface to control > things like managing multiplexed channels. Right, ok. > The overall design assumed that the IPA connection between the modem > and AP was carrying QMAP protocol though. And the rmnet driver is > designed to parse and handle that, so for the design I started with > the use of the rmnet driver made sense: it is a shim layer that takes > care of rmnet multiplexing and aggregation (and checksum offload). Sure, I can't really disagree. It's just that we have an ongoing discussion separately about whether or not rmnet really makes sense itself, mostly starting from our interest in supporting the Intel modem, and realizing that we have like 5 or 6 different driver-specific interfaces of doing the same thing. > So getting back to your question, the IPA in its current form only > has a single "multiplexed" channel carried over the connection > between the AP and modem. Previously (and in the future) there > was a way to add or remove channels. What would those channels do? I've not really been very clear with the differentiation between a channel and what's multiplexed inside of the channel. Using the terminology you defined in your other mail, are you saying that IPA (originally) allowed multiple *connections* to the device, or is there basically just one connection, with multiple (QMAP-muxed) *channels* on top of it? If the latter, why did IPA need ioctls, rather than rmnet? > > The software bridging is very questionable to start with, I'd advocate > > not supporting that at all but adding tracepoints or similar if needed > > for debugging instead. > > To be honest I don't understand the connection between software > bridging and debugging, but that's OK. It's a mess. Basically, AFAICT, the only use for the rmnet bridging is in fact debugging. What it does, again AFAICT, is mirror out all the rmnet packets to the bridge if you attach it to a bridge, so that then you can attach another netdev to the bridge and forward all the rmnet packets to another system for debugging. It's a very weird way of doing this, IMHO. > I'm a fan of tracepoints > and have always intended to make use of them in the IPA driver. :-) > The hardware can aggregate multiple packets received from the > modem into a single buffer, which the rmnet driver is then able > to deaggregate. Right, I gathered that much, but I'm not really sure I see why userspace would even be allowed to control this? Either the device is doing it or not, but the driver is going to have to cope either way? > This feature is supposed to help performance > but I've always been a little skeptical because it also comes > at a cost. This is used as a flag in an rmnet (QMAP) header, > which to me seems a little odd. (There should be a distinction > between flags needed in a message header and flags that represent > properties of a connection or channel.) I'm not going to comment on the QMAP protocol, I know nothing about it :-) > I believe the only QMAP commands are for doing essentially > XON/XOFF flow control on a single channel. In the course of > the e-mail discussion in the past few weeks I've come to see > why that would be necessary. It does make sense, because you only have a single hardware (DMA) channel in these cases, so you implement flow control in software on top. (As I said before, the Intel modem uses different hardware channels for different sessions, so doesn't need something like this - the hardware ring just fills up and there's your flow control) > The checksum offload is done differently, depending on whether > it's ingress (download from modem) or egress. For egress, > a header is inserted that describes what the hardware should > checksum and where it should place the result. For ingress, > the hardware appends a trailer that contains information > about the computed checksum values. The rmnet driver is > currently responsible for inserting the header and parsing > the trailer. Sure, makes sense, but again - if you can negotiate with the modem on whether it's going to do RX CSUM offload, you can tell it with ethtool (assuming you do have a netdev per channel, I guess), and for TX CSUM you can just have the header or not depending on whether you want it and the modem is capable of it. > I'm probably missing something, but I think the checksum > offload could be handled by the IPA driver rather than > rmnet. It seems to be an add-on that is completely > independent of the multiplexing and aggregation capabilities > that QMAP provides. Agree. > > > > If true though, then I think this would be the killer argument *in > > > > favour* of *not* merging this - because that would mean we *don't* have > > > > to actually keep the rmnet API around for all foreseeable future. > > This is because it's a user space API? If so I now understand > what you mean. Yes. > As Arnd said (below) this is designed in the way out-of-tree code > works and expects. I don't want to advocate for breaking that, > but if a general model that supports what's required can be used, > I'll adapt the IPA code to suit that. > > My goal continues to be getting a baseline IPA driver accepted > upstream as soon as possible, so I can then start building on > that foundation. Yeah. My goal is actually the same, but for the Intel driver, but I don't have much code yet (it's being cleaned up now) :-) johannes
On Tue, 2019-06-18 at 09:00 -0500, Alex Elder wrote: > Deaggregation is a connection property, not a channel property. That'd make sense, yes. > And it looks like that's exactly how it's used in the rmnet > driver. Yeah, I think you're right. I got confused by the whole use of "port" there, but it seems like "port" actually refers to the underlying netdev. Which is really strange too, btw, because you configure the "port" to agg/non-agg when you add a new channel to it ... So it seems like it's part of the channel configuration, when it's not! Anyway, I think for now we could probably live with not having this configurable for the IPA driver, and if it *does* need to be configurable, it seems like it should be a driver configuration, not a channel configuration - so something like a debugfs hook if you really just need to play with it for performance testing, or a module parameter, or something else? Or even, in the WWAN framework, a knob that we provide there for the WWAN device, rather than for the (newly created) channel. > The hardware is capable of aggregating QMAP packets > arriving on a connection into a single buffer, so this provides > a way of requesting it do that. > > > > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > > > > Similar here? If you have flow control you probably want to use it? > > I agree with that, though perhaps there are cases where it > is pointless, or can't be supported, so one might want to > simply *not* implement/advertise the feature. I don't know. Sure, but then that's likely something the driver would need to know, not necessarily userspace? johannes
On Tue, Jun 18, 2019 at 9:14 PM Johannes Berg <johannes@sipsolutions.net> wrote: > On Tue, 2019-06-18 at 08:16 -0500, Alex Elder wrote: > > On 6/17/19 6:28 AM, Johannes Berg wrote: > > So getting back to your question, the IPA in its current form only > > has a single "multiplexed" channel carried over the connection > > between the AP and modem. Previously (and in the future) there > > was a way to add or remove channels. > > What would those channels do? > > I've not really been very clear with the differentiation between a > channel and what's multiplexed inside of the channel. > > Using the terminology you defined in your other mail, are you saying > that IPA (originally) allowed multiple *connections* to the device, or > is there basically just one connection, with multiple (QMAP-muxed) > *channels* on top of it? > > If the latter, why did IPA need ioctls, rather than rmnet? From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code. > > > The software bridging is very questionable to start with, I'd advocate > > > not supporting that at all but adding tracepoints or similar if needed > > > for debugging instead. > > > > To be honest I don't understand the connection between software > > bridging and debugging, but that's OK. > > It's a mess. Basically, AFAICT, the only use for the rmnet bridging is > in fact debugging. What it does, again AFAICT, is mirror out all the > rmnet packets to the bridge if you attach it to a bridge, so that then > you can attach another netdev to the bridge and forward all the rmnet > packets to another system for debugging. > > It's a very weird way of doing this, IMHO. My understanding for this was that the idea is to use it for connecting bridging between distinct hardware devices behind ipa: if IPA drives both a USB-ether gadget and the 5G modem, you can use to talk to Linux running rmnet, but you can also use rmnet to provide fast usb tethering to 5g and bypass the rest of the network stack. That again may have been a wrong guess on my part. > > I believe the only QMAP commands are for doing essentially > > XON/XOFF flow control on a single channel. In the course of > > the e-mail discussion in the past few weeks I've come to see > > why that would be necessary. > > It does make sense, because you only have a single hardware (DMA) > channel in these cases, so you implement flow control in software on > top. > > (As I said before, the Intel modem uses different hardware channels for > different sessions, so doesn't need something like this - the hardware > ring just fills up and there's your flow control) ipa definitely has multiple hardware queues, and the Alex' driver does implement the data path on those, just not the configuration to enable them. Guessing once more, I suspect the the XON/XOFF flow control was a workaround for the fact that rmnet and ipa have separate queues. The hardware channel on IPA may fill up, but user space talks to rmnet and still add more frames to it because it doesn't know IPA is busy. Another possible explanation would be that this is actually forwarding state from the base station to tell the driver to stop sending data over the air. Arnd
On Tue, Jun 18, 2019 at 9:03 PM Johannes Berg <johannes@sipsolutions.net> wrote: > On Tue, 2019-06-18 at 08:45 -0500, Alex Elder wrote: > Really there are two possible ways (and they intersect to some extent). > > One is the whole multi-function device, where a single WWAN device is > composed of channels offered by actually different drivers, e.g. for a > typical USB device you might have something like cdc_ether and the > usb_wwan TTY driver. In this way, we need to "compose" the WWAN device > similarly, e.g. by using the underlying USB device "struct device" > pointer to tie it together. > > The other is something like IPA or the Intel modem driver, where the > device is actually a single (e.g. PCIe) device and just has a single > driver, but that single driver offers different channels. I would hope we can simplify this to expect only the second model, where you have a 'struct device' corresponding to hardware and the driver for it creates one wwan_device that user space talks to. Clearly the multi-function device hardware has to be handled somehow, but it would seem much cleaner in the long run to do that using a special workaround rather than putting this into the core interface. E.g. have a driver that lets you create a wwan_device by passing netdev and a tty chardev into a configuration interface, and from that point on use the generic wwan abstraction. > Now, it's not clear to me where IPA actually falls, because so far we've > been talking about the IPA driver only as providing *netdevs*, not any > control channels, so I'm not actually sure where the control channel is. The IPA driver today only handles the data path, because Alex removed the control channel. IPA is the driver that needs to talk to the hardware, both for data and control when finished. rmnet is a pure software construct that also contains both a data and control side and is designed to be independent of the lower hardware. Arnd
On Tue, 2019-06-18 at 22:09 +0200, Arnd Bergmann wrote: > > > One is the whole multi-function device, where a single WWAN device is > > composed of channels offered by actually different drivers, e.g. for a > > typical USB device you might have something like cdc_ether and the > > usb_wwan TTY driver. In this way, we need to "compose" the WWAN device > > similarly, e.g. by using the underlying USB device "struct device" > > pointer to tie it together. > > > > The other is something like IPA or the Intel modem driver, where the > > device is actually a single (e.g. PCIe) device and just has a single > > driver, but that single driver offers different channels. > > I would hope we can simplify this to expect only the second model, > where you have a 'struct device' corresponding to hardware and the > driver for it creates one wwan_device that user space talks to. I'm not sure. Fundamentally, we have drivers in Linux for the ethernet part, for the TTY part, and for whatever other part might be in a given USB multi- function device. > Clearly the multi-function device hardware has to be handled somehow, > but it would seem much cleaner in the long run to do that using > a special workaround rather than putting this into the core interface. I don't think it really makes the core interface much more complex or difficult though, and it feels easier than writing a completely different USB driver yet again for all these devices? As far as I understand from Dan, sometimes they really are no different from a generic USB TTY and a generic USB ethernet, except you know that if those show up together it's a modem. > E.g. have a driver that lets you create a wwan_device by passing > netdev and a tty chardev into a configuration interface, and from that > point on use the generic wwan abstraction. Yeah, but where do you hang that driver? Maybe the TTY function is actually a WWAN specific USB driver, but the ethernet is something generic that can also work with pure ethernet USB devices, and it's difficult to figure out how to tie those together. The modules could load in completely different order, or even the ethernet module could load but the TTY one doesn't because it's not configured, or vice versa. > > Now, it's not clear to me where IPA actually falls, because so far we've > > been talking about the IPA driver only as providing *netdevs*, not any > > control channels, so I'm not actually sure where the control channel is. > > The IPA driver today only handles the data path, because Alex removed > the control channel. IPA is the driver that needs to talk to the hardware, > both for data and control when finished. rmnet is a pure software construct > that also contains both a data and control side and is designed to be > independent of the lower hardware. I'd actually be interested in what the control path should be like. Is it also muxed on QMAP in the same way? johannes
On Tue, Jun 18, 2019 at 10:15 PM Johannes Berg <johannes@sipsolutions.net> wrote: > On Tue, 2019-06-18 at 22:09 +0200, Arnd Bergmann wrote: > > > One is the whole multi-function device, where a single WWAN device is > > > composed of channels offered by actually different drivers, e.g. for a > > > typical USB device you might have something like cdc_ether and the > > > usb_wwan TTY driver. In this way, we need to "compose" the WWAN device > > > similarly, e.g. by using the underlying USB device "struct device" > > > pointer to tie it together. > > > > > > The other is something like IPA or the Intel modem driver, where the > > > device is actually a single (e.g. PCIe) device and just has a single > > > driver, but that single driver offers different channels. > > > > I would hope we can simplify this to expect only the second model, > > where you have a 'struct device' corresponding to hardware and the > > driver for it creates one wwan_device that user space talks to. > > I'm not sure. > > Fundamentally, we have drivers in Linux for the ethernet part, for the > TTY part, and for whatever other part might be in a given USB multi- > function device. > > > Clearly the multi-function device hardware has to be handled somehow, > > but it would seem much cleaner in the long run to do that using > > a special workaround rather than putting this into the core interface. > > I don't think it really makes the core interface much more complex or > difficult though, and it feels easier than writing a completely > different USB driver yet again for all these devices? > > As far as I understand from Dan, sometimes they really are no different > from a generic USB TTY and a generic USB ethernet, except you know that > if those show up together it's a modem. > > > E.g. have a driver that lets you create a wwan_device by passing > > netdev and a tty chardev into a configuration interface, and from that > > point on use the generic wwan abstraction. > > Yeah, but where do you hang that driver? Maybe the TTY function is > actually a WWAN specific USB driver, but the ethernet is something > generic that can also work with pure ethernet USB devices, and it's > difficult to figure out how to tie those together. The modules could > load in completely different order, or even the ethernet module could > load but the TTY one doesn't because it's not configured, or vice versa. That was more or less my point: The current drivers exist, but don't lean themselves to fitting into a new framework, so maybe the best answer is not to try fitting them. To clarify: I'm not suggesting to write new USB drivers for these at all, but instead keep three parts that are completely unaware of each other a) a regular netdevice driver b) a regular tty driver c) the new wwan subsystem that expects a device to be created from a hardware driver but knows nothing of a) and b) To connect these together, we need one glue driver that implements the wwan_device and talks to a) and b) as the hardware. There are many ways to do that. One way would be to add a tty ldisc driver. A small user space helper opens the chardev, sets the ldisc and then uses an ldisc specific ioctl command to create a wwan device by passing an identifier of the netdevice and then exits. From that point on, you have a wwan device like any other. Arnd
On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: > > From my understanding, the ioctl interface would create the lower > netdev after talking to the firmware, and then user space would use > the rmnet interface to create a matching upper-level device for that. > This is an artifact of the strong separation of ipa and rmnet in the > code. Huh. But if rmnet has muxing, and IPA supports that, why would you ever need multiple lower netdevs? > > > > The software bridging [...] > > My understanding for this was that the idea is to use it for > connecting bridging between distinct hardware devices behind > ipa: if IPA drives both a USB-ether gadget and the 5G modem, > you can use to talk to Linux running rmnet, but you can also > use rmnet to provide fast usb tethering to 5g and bypass the > rest of the network stack. That again may have been a wrong > guess on my part. Hmm. Interesting. It didn't really look to me like that, but I'm really getting lost in the code. Anyway, it seems weird, because then you'd just bridge the upper netdev with the other ethernet and don't need special logic? And I don't see how the ethernet headers would work with this now. > ipa definitely has multiple hardware queues, and the Alex' > driver does implement the data path on those, just not the > configuration to enable them. OK, but perhaps you don't actually have enough to use one for each session? > Guessing once more, I suspect the the XON/XOFF flow control > was a workaround for the fact that rmnet and ipa have separate > queues. The hardware channel on IPA may fill up, but user space > talks to rmnet and still add more frames to it because it doesn't > know IPA is busy. > > Another possible explanation would be that this is actually > forwarding state from the base station to tell the driver to > stop sending data over the air. Yeah, but if you actually have a hardware queue per upper netdev then you don't really need this - you just stop the netdev queue when the hardware queue is full, and you have flow control automatically. So I really don't see any reason to have these messages going back and forth unless you plan to have multiple sessions muxed on a single hardware queue. And really, if you don't mux multiple sessions onto a single hardware queue, you don't need a mux header either, so it all adds up :-) johannes
On Tue, 2019-06-18 at 22:33 +0200, Arnd Bergmann wrote: > > Yeah, but where do you hang that driver? Maybe the TTY function is > > actually a WWAN specific USB driver, but the ethernet is something > > generic that can also work with pure ethernet USB devices, and it's > > difficult to figure out how to tie those together. The modules could > > load in completely different order, or even the ethernet module could > > load but the TTY one doesn't because it's not configured, or vice versa. > > That was more or less my point: The current drivers exist, but don't > lean themselves to fitting into a new framework, so maybe the best > answer is not to try fitting them. > > To clarify: I'm not suggesting to write new USB drivers for these at all, > but instead keep three parts that are completely unaware of each other > a) a regular netdevice driver > b) a regular tty driver > c) the new wwan subsystem that expects a device to be created > from a hardware driver but knows nothing of a) and b) > > To connect these together, we need one glue driver that implements > the wwan_device and talks to a) and b) as the hardware. There are > many ways to do that. One way would be to add a tty ldisc driver. > A small user space helper opens the chardev, sets the ldisc > and then uses an ldisc specific ioctl command to create a wwan > device by passing an identifier of the netdevice and then exits. > From that point on, you have a wwan device like any other. Well, ok. I don't think it'd really work that way ("passing an identifier of the netdevice") because you could have multiple netdevices, but I see what you mean in principle. It seems to me though that this is far more complex than what I'm proposing? What I'm proposing there doesn't even need any userspace involvement, as long as all the pieces are in the different sub-drivers, they'd fall out automatically. And realistically, the wwan_device falls out anyway at some point, the only question is if we really make one specific driver be the "owner" of it. I'm suggesting that we don't, and just make its lifetime depend on the links to parts it has (unless something like IPA actually wants to be an owner). johannes
On Tue, Jun 18, 2019 at 10:36 PM Johannes Berg <johannes@sipsolutions.net> wrote: > > On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: > > > > From my understanding, the ioctl interface would create the lower > > netdev after talking to the firmware, and then user space would use > > the rmnet interface to create a matching upper-level device for that. > > This is an artifact of the strong separation of ipa and rmnet in the > > code. > > Huh. But if rmnet has muxing, and IPA supports that, why would you ever > need multiple lower netdevs? From my reading of the code, there is always exactly a 1:1 relationship between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ decapsulation of the qmap data and forwards it to the ipa netdev, which then just passes data through between a hardware queue and its netdevice. [side note: on top of that, rmnet also does "aggregation", which may be a confusing term that only means transferring multiple frames at once] > > ipa definitely has multiple hardware queues, and the Alex' > > driver does implement the data path on those, just not the > > configuration to enable them. > > OK, but perhaps you don't actually have enough to use one for each > session? I'm lacking the terminology here, but what I understood was that the netdev and queue again map to a session. > > Guessing once more, I suspect the the XON/XOFF flow control > > was a workaround for the fact that rmnet and ipa have separate > > queues. The hardware channel on IPA may fill up, but user space > > talks to rmnet and still add more frames to it because it doesn't > > know IPA is busy. > > > > Another possible explanation would be that this is actually > > forwarding state from the base station to tell the driver to > > stop sending data over the air. > > Yeah, but if you actually have a hardware queue per upper netdev then > you don't really need this - you just stop the netdev queue when the > hardware queue is full, and you have flow control automatically. > > So I really don't see any reason to have these messages going back and > forth unless you plan to have multiple sessions muxed on a single > hardware queue. Sure, I definitely understand what you mean, and I agree that would be the right way to do it. All I said is that this is not how it was done in rmnet (this was again my main concern about the rmnet design after I learned it was required for ipa) ;-) Arnd
On Tue, 2019-06-18 at 22:55 +0200, Arnd Bergmann wrote: > On Tue, Jun 18, 2019 at 10:36 PM Johannes Berg > <johannes@sipsolutions.net> wrote: > > > > On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: > > > > > > From my understanding, the ioctl interface would create the lower > > > netdev after talking to the firmware, and then user space would use > > > the rmnet interface to create a matching upper-level device for that. > > > This is an artifact of the strong separation of ipa and rmnet in the > > > code. > > > > Huh. But if rmnet has muxing, and IPA supports that, why would you ever > > need multiple lower netdevs? > > From my reading of the code, there is always exactly a 1:1 relationship > between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ > decapsulation of the qmap data and forwards it to the ipa netdev, > which then just passes data through between a hardware queue and > its netdevice. I'll take your word for it. Seems very odd, given that the whole point of the QMAP header seems to be ... muxing? > [side note: on top of that, rmnet also does "aggregation", which may > be a confusing term that only means transferring multiple frames > at once] Right, but it's not all that much interesting in the context of this discussion. > Sure, I definitely understand what you mean, and I agree that would > be the right way to do it. All I said is that this is not how it was done > in rmnet (this was again my main concern about the rmnet design > after I learned it was required for ipa) ;-) :-) Well, I guess though if the firmware wants us to listen to those on/off messages we'll have to do that one way or the other. Oh. Maybe it's just *because* rmnet is layered on top, and thus you fundamentally cannot do flow control the way I described - not because you have multiple session on the same hardware ring, but because you abstracted the hardware ring away too much ... johannes
On Tue, Jun 18, 2019 at 10:39 PM Johannes Berg <johannes@sipsolutions.net> wrote: > On Tue, 2019-06-18 at 22:33 +0200, Arnd Bergmann wrote: > It seems to me though that this is far more complex than what I'm > proposing? What I'm proposing there doesn't even need any userspace > involvement, as long as all the pieces are in the different sub-drivers, > they'd fall out automatically. > > And realistically, the wwan_device falls out anyway at some point, the > only question is if we really make one specific driver be the "owner" of > it. I'm suggesting that we don't, and just make its lifetime depend on > the links to parts it has (unless something like IPA actually wants to > be an owner). My feeling so far is that having the wwan_device be owned by a device gives a nicer abstraction model that is also simpler for the common case. A device driver like ipa would end up with a probe() function that does does wwan_device_alloc/wwan_device_register, corresponding to alloc_etherdev/register_netdev, and then communicates through callbacks. I agree the compound device case would get more complex by shoehorning it into this model, but that can be a valid tradeoff if it's the exceptional case rather than the common one. Arnd
On 2019-06-18 14:55, Arnd Bergmann wrote: > On Tue, Jun 18, 2019 at 10:36 PM Johannes Berg > <johannes@sipsolutions.net> wrote: >> >> On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: >> > >> > From my understanding, the ioctl interface would create the lower >> > netdev after talking to the firmware, and then user space would use >> > the rmnet interface to create a matching upper-level device for that. >> > This is an artifact of the strong separation of ipa and rmnet in the >> > code. >> >> Huh. But if rmnet has muxing, and IPA supports that, why would you >> ever >> need multiple lower netdevs? > > From my reading of the code, there is always exactly a 1:1 relationship > between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ > decapsulation of the qmap data and forwards it to the ipa netdev, > which then just passes data through between a hardware queue and > its netdevice. > There is a n:1 relationship between rmnet and IPA. rmnet does the de-muxing to multiple netdevs based on the mux id in the MAP header for RX packets and vice versa. > [side note: on top of that, rmnet also does "aggregation", which may > be a confusing term that only means transferring multiple frames > at once] > >> > ipa definitely has multiple hardware queues, and the Alex' >> > driver does implement the data path on those, just not the >> > configuration to enable them. >> >> OK, but perhaps you don't actually have enough to use one for each >> session? > > I'm lacking the terminology here, but what I understood was that > the netdev and queue again map to a session. > >> > Guessing once more, I suspect the the XON/XOFF flow control >> > was a workaround for the fact that rmnet and ipa have separate >> > queues. The hardware channel on IPA may fill up, but user space >> > talks to rmnet and still add more frames to it because it doesn't >> > know IPA is busy. >> > >> > Another possible explanation would be that this is actually >> > forwarding state from the base station to tell the driver to >> > stop sending data over the air. >> >> Yeah, but if you actually have a hardware queue per upper netdev then >> you don't really need this - you just stop the netdev queue when the >> hardware queue is full, and you have flow control automatically. >> >> So I really don't see any reason to have these messages going back and >> forth unless you plan to have multiple sessions muxed on a single >> hardware queue. > Hardware may flow control specific PDNs (rmnet interfaces) based on QoS - not necessarily only in case of hardware queue full. > Sure, I definitely understand what you mean, and I agree that would > be the right way to do it. All I said is that this is not how it was > done > in rmnet (this was again my main concern about the rmnet design > after I learned it was required for ipa) ;-) > > Arnd
On Tue, Jun 18, 2019 at 11:15 PM Subash Abhinov Kasiviswanathan <subashab@codeaurora.org> wrote: > > On 2019-06-18 14:55, Arnd Bergmann wrote: > > On Tue, Jun 18, 2019 at 10:36 PM Johannes Berg > > <johannes@sipsolutions.net> wrote: > >> > >> On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: > >> > > >> > From my understanding, the ioctl interface would create the lower > >> > netdev after talking to the firmware, and then user space would use > >> > the rmnet interface to create a matching upper-level device for that. > >> > This is an artifact of the strong separation of ipa and rmnet in the > >> > code. > >> > >> Huh. But if rmnet has muxing, and IPA supports that, why would you > >> ever > >> need multiple lower netdevs? > > > > From my reading of the code, there is always exactly a 1:1 relationship > > between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ > > decapsulation of the qmap data and forwards it to the ipa netdev, > > which then just passes data through between a hardware queue and > > its netdevice. > > > > There is a n:1 relationship between rmnet and IPA. > rmnet does the de-muxing to multiple netdevs based on the mux id > in the MAP header for RX packets and vice versa. Oh, so you mean that even though IPA supports multiple channels and multiple netdev instances for a physical device, all the rmnet devices end up being thrown into a single channel in IPA? What are the other channels for in IPA? I understand that there is one channel for commands that is separate, while the others are for network devices, but that seems to make no sense if we only use a single channel for rmnet data. > >> Yeah, but if you actually have a hardware queue per upper netdev then > >> you don't really need this - you just stop the netdev queue when the > >> hardware queue is full, and you have flow control automatically. > >> > >> So I really don't see any reason to have these messages going back and > >> forth unless you plan to have multiple sessions muxed on a single > >> hardware queue. > > > > Hardware may flow control specific PDNs (rmnet interfaces) based on QoS > - > not necessarily only in case of hardware queue full. Right, I guess that makes sense if everything ends up in a single queue in IPA. Arnd
>> There is a n:1 relationship between rmnet and IPA. >> rmnet does the de-muxing to multiple netdevs based on the mux id >> in the MAP header for RX packets and vice versa. > > Oh, so you mean that even though IPA supports multiple channels > and multiple netdev instances for a physical device, all the > rmnet devices end up being thrown into a single channel in IPA? > > What are the other channels for in IPA? I understand that there > is one channel for commands that is separate, while the others > are for network devices, but that seems to make no sense if > we only use a single channel for rmnet data. > AFAIK, the other channels are for use cases like tethering. There is only a single channel which is used for RX data which is then de-muxed using rmnet.
On Tue, 2019-06-18 at 23:06 +0200, Arnd Bergmann wrote: > On Tue, Jun 18, 2019 at 10:39 PM Johannes Berg > <johannes@sipsolutions.net> wrote: > > On Tue, 2019-06-18 at 22:33 +0200, Arnd Bergmann wrote: > > It seems to me though that this is far more complex than what I'm > > proposing? What I'm proposing there doesn't even need any userspace > > involvement, as long as all the pieces are in the different sub- > > drivers, > > they'd fall out automatically. > > > > And realistically, the wwan_device falls out anyway at some point, > > the > > only question is if we really make one specific driver be the > > "owner" of > > it. I'm suggesting that we don't, and just make its lifetime depend > > on > > the links to parts it has (unless something like IPA actually wants > > to > > be an owner). > > My feeling so far is that having the wwan_device be owned by a device > gives a nicer abstraction model that is also simpler for the common > case. A device driver like ipa would end up with a probe() function > that does does wwan_device_alloc/wwan_device_register, corresponding > to alloc_etherdev/register_netdev, and then communicates through > callbacks. > > I agree the compound device case would get more complex by > shoehorning it into this model, but that can be a valid tradeoff > if it's the exceptional case rather than the common one. In my experience, the compound device model is by far the most prevalent for regular Linux distros or anything *not* running on an SoC with an integrated modem. But it's also quite common for Android, no? drivers/net/ethernet/msm/ has rmnet and IPA ethernet drivers while arch/arm/mach-msm/ has various SMD-related control channel drivers like smd_tty.c and smd_qmi.c and smd_nmea.c. At least that's how I remember older SMD-based devices being in the 8xxx and 9xxx time. Ideally those setups can benefit from this framework as well, without having to write entirely new composite drivers for those devices. Dan
On Wed, 2019-06-19 at 12:47 -0600, Subash Abhinov Kasiviswanathan wrote: > > > There is a n:1 relationship between rmnet and IPA. > > > rmnet does the de-muxing to multiple netdevs based on the mux id > > > in the MAP header for RX packets and vice versa. > > > > Oh, so you mean that even though IPA supports multiple channels > > and multiple netdev instances for a physical device, all the > > rmnet devices end up being thrown into a single channel in IPA? > > > > What are the other channels for in IPA? I understand that there > > is one channel for commands that is separate, while the others > > are for network devices, but that seems to make no sense if > > we only use a single channel for rmnet data. > > > > AFAIK, the other channels are for use cases like tethering. > There is only a single channel which is used for RX > data which is then de-muxed using rmnet. That seems odd, since tethering is no different than any other data channel in QMI, just that it may have a different APN and QoS guarantees. Dan
On 6/18/19 1:06 PM, Dan Williams wrote: > On Tue, 2019-06-18 at 10:20 -0500, Alex Elder wrote: >> On 6/17/19 7:25 AM, Johannes Berg wrote: >>> On Mon, 2019-06-17 at 13:42 +0200, Johannes Berg wrote: >>> >>>> But anyway, as I alluded to above, I had something like this in >>>> mind: Sorry for the delay. There's a lot here to go through, and with each message the picture is (slowly) getting a bit clearer for me. Still, there are some broad tradeoffs to consider and I think we need to get a little more specific again. I'm going to start a new thread (or rather re-subject a response to the very first one) that tries to do a fresh start that takes into account the discussion so far. I will also be talking with some people inside Qualcomm (including Subash) soon to make sure we don't miss any requirements or insights they know of that I don't realize are important. But before I send anything new I'm going to respond to a few things. >>> I forgot to state this here, but this was *heavily* influenced by >>> discussions with Dan - many thanks to him. >> >> Thanks for getting even more concrete with this. Code is the >> most concise way of describing things, once the general ideas >> seem to be coming together. >> >> I'm not going to comment on the specific code bits, but I have >> some more general questions and comments on the design. Some >> of these are simply due to my lack of knowledge of how WWAN/modem >> interactions normally work. >> >> First, a few terms (correct or improve as you like): >> - WWAN device is a hardware device (like IPA) that presents a >> connection between AP and modem, and presents an interface >> that allows the use of that connection to be managed. >> - WWAN netdevice represents a Linux network interface, with its >> operations and queues, etc., but implements a standardized >> set of WWAN-specific operations. It represents a logical >> ' channel whose data is multiplexed over the WWAN device. >> - WWAN channel is a user space abstraction that corresponds >> with a WWAN netdevice (but I'm not clear on all the ways >> they differ or interact). > > When Johannes and I have talked about "WWAN channel" we mean a control > or data or other channel. That could be QMI, AT, MBIM control, GPS, > PCSC, QMAP, MBIM data, PPP TTY, DM/DIAG, CDC-ETHER, CDC-NCM, Sierra > HIP, etc. Or even voice-call audio :) > > A netdev is a Linux abstraction of a WWAN *data* channel, be that QMI > or CDC-ETHER or whatever. I think I now understand this. My only focus with the IPA driver has been the network data driver. I'll go into more detail later but I now see that there are other entities on a WWAN device that do not require a netdev. >> - The WWAN core is kernel code that presents abstractions >> for WWAN devices and netdevices, so they can be managed >> in a generic way. It is for configuration and communication >> and is not at all involved in the data path. >> >> You're saying that the WWAN driver space calls wwan_add() >> to register itself as a new WWAN device. >> >> You're also saying that a WWAN device "attaches" a WWAN >> netdevice, which is basically notifying the WWAN core >> that the new netdev/channel is available for use. >> - I trust that a "tentative" attachement is necessary. But >> I'm not sure what makes it transition into becoming a >> "real" one, or how that event gets communicated. > > Linux usually tries to keep drivers generic and focused; each driver is > written for a specific function. For example, a USB device usually > provides multiple USB interfaces which will be bound to different Linux > drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc. So USB has some attributes similar to what we're talking about here. But if I'm not mistaken we want some sort of an overall management scheme as well. > These drivers are often generic and we may not have enough information > in one driver to know that the parent of this interface is a WWAN > device. But another driver might. Since probing is asynchronous we may > have cdc-acm bind to a device and provide a TTY before cdc-ether (which > does know it's a WWAN) binds and provides the netdevice. Is this why Johannes wanted to have a "maybe attach" method? I don't like the "maybe" API unless there's no other way to do it. Instead I think it would be better for the probing driver to register with a whatever the WWAN core is, and then have the WWAN core be responsible for pulling things all together when it receives a request to do so. I.e., something in user space should request that a registered data interface be brought up, and at that time everything "knows" it's implemented as part of a WWAN device. >> Some questions: >> - What causes a new channel to be created? Is it initiated >> by the WWAN device driver? Does the modem request that >> it get created? User space? Both? > > Either created at driver bind time in the kernel (usually control > channels) or initiated by userspace when the WWAN management process > has coordinated with the firmware for another channel. Honestly So maybe: - Hardware probe detects a WWAN device - The drivers that detect the WWAN device register it with the WWAN core code. - A control channel is instantiated at/before the time the WWAN device is registered - Something in user space should manage the bring-up of any other things on the WWAN device thereafter > userspace should probably always create the netdevices (since they are > always useless until userspace coordinates with the firmware about > them) but that's not how things are yet. That's too bad. How hard would that be to change? > [ A concrete example... > > Assume a QMI device has an existing packet data connection which is > abstracted by a netdevice on the Linux side. Now the WWAN management > daemon wants to create a second packet data connection with a different > APN (maybe an MMS connection, maybe a VOIP one, maybe an IPv6). It > sends a WDS Start Network request to the modem firmware and receives a > new QMI Packet Data Handle. > > The management daemon must somehow get a netdevice associated with this > new Packet Data Handle. It would ask the WWAN kernel device to create a > new data channel with the PDH, and would get back the ifindex of that > netdevice which it would configure with the IP that it gets from the > firmware via the WDS Get Current Settings QMI request. > > The WWAN device would forward the request down to IPA (or rmnet) which > would then create the netdevice using the PDH as the QMAP MUX ID for > that netdevice's traffic.] OK yes I'm following this now. I appreciate the example. >> - What causes a created channel to be removed? > > Driver removal, userspace WWAN daemon terminating the packet data > connection which the channel represents, the modem terminating the > packet data connection (eg network initiated disconnect), etc. OK this is as I expected. Driver (or device) removal is somewhat obvious, but you're confirming user space might request it as well. >> - You distinguish between attaching a netdevice and (what >> I'll call) activating it. What causes activation? > > Can you describe what you mean by "activating"? Do you mean > successfully TX/RX packets via the netdev and the outside world? Johannes mentioned an API to "maybe attach" a device. That begs the question of what happens if this request does *not* attach. Does the attach request have to be made again, or is it done automatically with a notification, or something else? So by "activation" I was trying to refer to the notion of this subsequent successful attach. > I read "attach" here as simply associating an existing netdev with the > "parent" WWAN device. A purely Linux operation that is only book- > keeping and may not have any interaction with the modem. If that's the case I would want the "activation" to be a separate step. The attach would do the bookkeeping, and generally shouldn't fail. An attached interface would be brought up ("activated") separately and might fail if things aren't quite ready yet. >> - How are the attributes of a WWAN device or channel set, >> or communicated? > > Via netlink attributes when userspace asks the WWAN device to create a > new channel. In the control methods I've seen, only userspace really > knows the channel identifier that it and the modem have agreed on (eg > what the MUX ID in the QMAP header would be, or the MBIM Session ID). Yes, that's the way it's worked for rmnet and IPA. Previously it was IOCTL requests but it's currently hard-wired. >> - Are there any attributes that are only optionally supported, >> and if so, how are the supported ones communicated? > > Yeah, capabilities would be important here and I don't think Johannes > accounted for that yet. > >> - Which WWAN channel attributes must be set *before* the >> channel is activated, and can't be changed? Are there any >> that can be changed dynamically? > > I would assume userspace must pass the agreed identifier (QMUX ID, MBIM > session ID, etc) when creating the channel and that wouldn't change. I > think a world where you can dynamically change the MUX ID/SessionID/etc > is a more complicated one. > > Things like QoS could change but I don't recall if modems allow that; > eg does a +CGEQOS (or equivalent QMI WDS LTE QoS Parameters request) > take effect while the bearer is active, or is it only respected on > bearer creation? You are speaking in a language I'm only now coming to understand. I think the point of my question is clear though--I think both static and dynamic attributes need to be taken into account. >> And while the whole point of this is to make things generic, >> it might be nice to have a way to implement a new feature >> before it can be "standardized". > > That would be nice, but I'd rather have the conversation about if/how > to standardize things before they make it into the kernel and have > their API set in stone... which is how we ended up with 5 ways of doing > the same thing already. Agreed. -Alex > Dan > >> Thanks. >> >> -Alex >> >> PS I don't want to exclude anybody but we could probably start >> a different mail chain on this topic... >> >>>> driver_dev >>>> struct device *dev (USB, PCI, ...) >>>> net_device NA >>>> net_device NB >>>> tty TA >>>> ... >>>> >> >> . . . >
On 6/18/19 1:48 PM, Johannes Berg wrote: > Just to add to Dan's response, I think he's captured our discussions and > thoughts well. > >> First, a few terms (correct or improve as you like): > > Thanks for defining, we don't do that nearly often enough. > >> - WWAN device is a hardware device (like IPA) that presents a >> connection between AP and modem, and presents an interface >> that allows the use of that connection to be managed. > > Yes. But I was actually thinking of a "wwan_dev" to be a separate > structure, not *directly* owned by a single driver and used to represent > the hardware like a (hypothetical) "struct ipa_dev". I think you're talking about creating a coordination interface that allows multiple drivers to interact with a WWAN device, which might implement several independent features. >> - WWAN netdevice represents a Linux network interface, with its >> operations and queues, etc., but implements a standardized >> set of WWAN-specific operations. It represents a logical >> ' channel whose data is multiplexed over the WWAN device. > > I'm not sure I'd asy it has much WWAN-specific operations? But yeah, I > guess it might. I want to withdraw this notion of a "WWAN netdevice"... >> - WWAN channel is a user space abstraction that corresponds >> with a WWAN netdevice (but I'm not clear on all the ways >> they differ or interact). > > As Dan said, this could be a different abstraction than a netdevice, > like a TTY, etc. Right, I get that now. . . . >> - Which WWAN channel attributes must be set *before* the >> channel is activated, and can't be changed? Are there any >> that can be changed dynamically? > > It's a good question. I threw a "u32 pdn" in there, but I'm not actually > sure that's what you *really* need? > > Maybe the modem and userspace just agree on some arbitrary "session > identifier"? Dan mentions "MUX ID" or "MBIM Session ID", maybe there > really is no good general term for this and we should just call it a > "session identifier" and agree that it depends on the control protocol > (MBIM vs. QMI vs. ...)? > >> And while the whole point of this is to make things generic, >> it might be nice to have a way to implement a new feature >> before it can be "standardized". > > Not sure I understand this? I'm talking about a way to experiment with new functionality in a way that's explicitly not part of the interface. But doing that isn't necessary and it's probably not a good idea anyway. > FWIW, I actually came to this because we want to upstream a driver for > an Intel modem, but ... can't really make up our mind on whether or not > to use VLAN tags, something like rmnet (but we obviously cannot use > rmnet, so that'd be another vendor specific interface like rmnet), or > sysfs, or any of the other methods we have today ... :-) OK cool then we have some common needs. Let's get this defined so we can use it for both! -Alex > > johannes >
On 6/18/19 2:03 PM, Johannes Berg wrote: > On Tue, 2019-06-18 at 08:45 -0500, Alex Elder wrote: > >> If it had a well-defined way of creating new channels to be >> multiplexed over the connection to the modem, the IPA driver >> (rather than the rmnet driver) could present network interfaces >> for each and perform the multiplexing. > > Right. That's what I was thinking of. . . . >> But I think the IPA driver would register with the WWAN core as >> a "provider," and then the WWAN core would subsequently request >> that it instantiate netdevices to represent channels on demand >> (rather than registering them). > > Yeah, I guess you could call it that way. > > Really there are two possible ways (and they intersect to some extent). > > One is the whole multi-function device, where a single WWAN device is > composed of channels offered by actually different drivers, e.g. for a > typical USB device you might have something like cdc_ether and the > usb_wwan TTY driver. In this way, we need to "compose" the WWAN device > similarly, e.g. by using the underlying USB device "struct device" > pointer to tie it together. I *think* this model makes the most sense. But at this point it would take very little to convince me otherwise... (And then I saw Arnd's message advocating the other one, unfortunately...) > The other is something like IPA or the Intel modem driver, where the > device is actually a single (e.g. PCIe) device and just has a single > driver, but that single driver offers different channels. What I don't like about this is that it's more monolithic. It seems better to have the low-level IPA or Intel modem driver (or any other driver that can support communication between the AP and WWAN device) present communication paths that other function- specific drivers can attach to and use. > Now, it's not clear to me where IPA actually falls, because so far we've > been talking about the IPA driver only as providing *netdevs*, not any > control channels, so I'm not actually sure where the control channel is. There is user space code that handles all of this, and as far as I can tell, parts of it will always remain proprietary. > For the Intel device, however, the control channel is definitely > provided by exactly the same driver as the data channels (netdevs). I do see the need for a control interface. But I suspect it would *overlap* with what you describe and might need to be more general and/or extensible. Are there control channels specific to use for a modem--like a "modem control interface" or something? Is there something broader, like "this WWAN device supports functions A, and B with protocols X, Y; please open a connection to A with protocol X." Do both exist? I'm just trying to contain whatever a "control channel" really represents, and what it would be associated with. > "provider" is a good word, and in fact the Intel driver would also be a > provider for a GNSS channel (TBD how to represent, a tty?), one or > multiple debug/tracing channels, data channels (netdevs), AT command > channels (mbim, ...?) (again tbd how to represent, ttys?), etc. Yes, this is much clearer to me now. > What I showed in the header files I posted so far was the provider only > having "data channel" ops (create/remove a netdev) but for each channel > type we either want a new method there, or we just change the method to > be something like > > int (*create_channel)(..., enum wwan_chan_type chan_type, ...); > > and simply require that the channel is attached to the wwan device with > the representation-specific call (wwan_attach_netdev, wwan_attach_tty, > ...). Or maybe have the WWAN device present interfaces with attributes, and have drivers that are appropriate for each interface attach to only the ones they recognize they support. -Alex > This is a bit less comfortable because then it's difficult to know what > was actually created upon the request, so it's probably better to have > different methods for the different types of representations (like I had > - add_netdev, add_tty, ...). > > Note also that I said "representation-specific", while passing a > "channel type", so for this we'd actually need a convention on what > channel type has what kind of representation, which again gets awkward. > Better to make it explicit. > > (And even then, we might be able to let userspace have some control, > e.g. the driver might be able to create a debug channel as both a TTY or > something else) > > johannes >
On 6/18/19 2:14 PM, Johannes Berg wrote: > On Tue, 2019-06-18 at 08:16 -0500, Alex Elder wrote: >> On 6/17/19 6:28 AM, Johannes Berg wrote: >>> On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: >>>> On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg >>>> <johannes@sipsolutions.net> wrote: >>>> >>>>>> As I've made clear before, my work on this has been focused on the IPA transport, . . . >> The IPA driver was very large, and in an effort to have an initial driver >> that was more easily accepted upstream, it was carved down to support >> a single, very simple use case. It supports only a single channel for >> carrying network data, and does not expose any of the IPA's other >> capabilities like filtering and routing (and multiplexing). > > Ok. But it *does* use (or even require using) rmnet, so it has multiple > channels in a sense, no? Yes. It's a multiplexing protocol, supporting one *or more* channels. >> Originally the IPA code had an IOCTL interface for adding and removing >> multiplexed channel IDs, but the simplified use case expected only one >> channel to be used. > > What did those channels do? Create different netdevs? Something else I don't know. The code I started with only supported the use of one channel, but assumed the use of rmnet anyway (for aggregation and checksum offload most likely). . . . >> So getting back to your question, the IPA in its current form only >> has a single "multiplexed" channel carried over the connection >> between the AP and modem. Previously (and in the future) there >> was a way to add or remove channels. > > What would those channels do? > > I've not really been very clear with the differentiation between a > channel and what's multiplexed inside of the channel. > > Using the terminology you defined in your other mail, are you saying > that IPA (originally) allowed multiple *connections* to the device, or > is there basically just one connection, with multiple (QMAP-muxed) > *channels* on top of it? One connection, with the ability to have multiple multiplexed channels. > If the latter, why did IPA need ioctls, rather than rmnet? The "full" IPA driver supported a lot more than what is being proposed right now. The strategy for getting support upstream was to drastically reduce the size of the driver by focusing on a single use case: providing a netdev data interface served by the modem. But even to support that, the IPA driver needed to allow user space to identify certain resources that need to be used for implementing that netdev, or configuring whether certain features (e.g. download checksum) were to be used. . . . >> The hardware can aggregate multiple packets received from the >> modem into a single buffer, which the rmnet driver is then able >> to deaggregate. > > Right, I gathered that much, but I'm not really sure I see why userspace > would even be allowed to control this? Either the device is doing it or > not, but the driver is going to have to cope either way? Maybe because doing aggregation or not is a policy decision? And/or a tunable parameter? There might be a more appropriate way to do this. . . . >> My goal continues to be getting a baseline IPA driver accepted >> upstream as soon as possible, so I can then start building on >> that foundation. > > Yeah. My goal is actually the same, but for the Intel driver, but I > don't have much code yet (it's being cleaned up now) :-) Well then I guess I'll beat you to it (or I *hope* I do)... -Alex > > johannes >
On 6/18/19 2:22 PM, Johannes Berg wrote: > On Tue, 2019-06-18 at 09:00 -0500, Alex Elder wrote: . . . > Anyway, I think for now we could probably live with not having this > configurable for the IPA driver, and if it *does* need to be > configurable, it seems like it should be a driver configuration, not a > channel configuration - so something like a debugfs hook if you really > just need to play with it for performance testing, or a module > parameter, or something else? > > Or even, in the WWAN framework, a knob that we provide there for the > WWAN device, rather than for the (newly created) channel. Agreed. I think a knob is appropriate, it's just a question of how that control exposed. Same answer to your question below. -Alex >> The hardware is capable of aggregating QMAP packets >> arriving on a connection into a single buffer, so this provides >> a way of requesting it do that. >> >>>> #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) >>> >>> Similar here? If you have flow control you probably want to use it? >> >> I agree with that, though perhaps there are cases where it >> is pointless, or can't be supported, so one might want to >> simply *not* implement/advertise the feature. I don't know. > > Sure, but then that's likely something the driver would need to know, > not necessarily userspace? > > johannes >
OK I want to try to organize a little more concisely some of the
discussion on this, because there is a very large amount of volume
to date and I think we need to try to narrow the focus back down
again.
I'm going to use a few terms here. Some of these I really don't
like, but I want to be unambiguous *and* (at least for now) I want
to avoid the very overloaded term "device".
I have lots more to say, but let's start with a top-level picture,
to make sure we're all on the same page.
WWAN Communication
Channel (Physical)
| ------------------------
------------ v | :+ Control | \
| |-----------| :+ Data | |
| AP | | WWAN unit :+ Voice | > Functions
| |===========| :+ GPS | |
------------ ^ | :+ ... | /
| -------------------------
Multiplexed WWAN
Communication
Channel (Physical)
- The *AP* is the main CPU complex that's running Linux on one or
more CPU cores.
- A *WWAN unit* is an entity that shares one or more physical
*WWAN communication channels* with the AP.
- A *WWAN communication channel* is a bidirectional means of
carrying data between the AP and WWAN unit.
- A WWAN communication channel carries data using a *WWAN protocol*.
- A WWAN unit implements one or more *WWAN functions*, such as
5G data, LTE voice, GPS, and so on.
- A WWAN unit shall implement a *WWAN control function*, used to
manage the use of other WWAN functions, as well as the WWAN unit
itself.
- The AP communicates with a WWAN function using a WWAN protocol.
- A WWAN physical channel can be *multiplexed*, in which case it
carries the data for one or more *WWAN logical channels*.
- A multiplexed WWAN communication channel uses a *WWAN wultiplexing
protocol*, which is used to separate independent data streams
carrying other WWAN protocols.
- A WWAN logical channel carries a bidirectional stream of WWAN
protocol data between an entity on the AP and a WWAN function.
Does that adequately represent a very high-level picture of what
we're trying to manage?
And if I understand it right, the purpose of the generic framework
being discussed is to define a common mechanism for managing (i.e.,
discovering, creating, destroying, querying, configuring, enabling,
disabling, etc.) WWAN units and the functions they implement, along
with the communication and logical channels used to communicate with
them.
Comments?
-Alex
On Mon, Jun 24, 2019 at 6:21 PM Alex Elder <elder@linaro.org> wrote: > On 6/18/19 2:03 PM, Johannes Berg wrote: > > > Really there are two possible ways (and they intersect to some extent). > > > > One is the whole multi-function device, where a single WWAN device is > > composed of channels offered by actually different drivers, e.g. for a > > typical USB device you might have something like cdc_ether and the > > usb_wwan TTY driver. In this way, we need to "compose" the WWAN device > > similarly, e.g. by using the underlying USB device "struct device" > > pointer to tie it together. > > I *think* this model makes the most sense. But at this point > it would take very little to convince me otherwise... (And then > I saw Arnd's message advocating the other one, unfortunately...) > > > The other is something like IPA or the Intel modem driver, where the > > device is actually a single (e.g. PCIe) device and just has a single > > driver, but that single driver offers different channels. > > What I don't like about this is that it's more monolithic. It > seems better to have the low-level IPA or Intel modem driver (or > any other driver that can support communication between the AP > and WWAN device) present communication paths that other function- > specific drivers can attach to and use. I did not understand Johannes description as two competing models for the same code, but rather two kinds of existing hardware that a new driver system would have to deal with. I was trying to simplify it to just having the second model, by adding a hack to support the first, but my view was rather unpopular so far, so if everyone agrees on one way to do it, don't worry about me ;-) > > Now, it's not clear to me where IPA actually falls, because so far we've > > been talking about the IPA driver only as providing *netdevs*, not any > > control channels, so I'm not actually sure where the control channel is. > > There is user space code that handles all of this, and as far as I > can tell, parts of it will always remain proprietary. Two replies on this: - to answer Johannes question, my understanding is that the interface between kernel and firmware/hardware for IPA has a single 'struct device' that is used for both the data and the control channels, rather than having a data channel and an independent control device, so this falls into the same category as the Intel one (please correct me on that) - The user space being proprietary is exactly what we need to avoid with the wwan subsystem. We need to be able to use the same method for setting up Intel, Qualcomm, Samsung, Unisoc or Hisilicon modems or anything else that hooks into the subsystem, and support that in network manager as well as the Android equivalent. If Qualcomm wants to provide their own proprietary user space solution, we can't stop them, but then that should also work on all the others unless they intentionally break it. ;-) > > and simply require that the channel is attached to the wwan device with > > the representation-specific call (wwan_attach_netdev, wwan_attach_tty, > > ...). > > Or maybe have the WWAN device present interfaces with attributes, > and have drivers that are appropriate for each interface attach > to only the ones they recognize they support. I think you both mean the same thing here, a structure with callback pointers that may or may not be filled by the driver depending on its capabilities. What we should try to avoid though is a way to add driver private interfaces that risk having multiple drivers create similar functionality in incompatible ways. Arnd
Sorry, I neglected to add Dan and Johannes--who have been primary contributors in this discussion--to this. Adding now. -Alex On 6/24/19 11:30 AM, Alex Elder wrote: > OK I want to try to organize a little more concisely some of the > discussion on this, because there is a very large amount of volume > to date and I think we need to try to narrow the focus back down > again. > > I'm going to use a few terms here. Some of these I really don't > like, but I want to be unambiguous *and* (at least for now) I want > to avoid the very overloaded term "device". > > I have lots more to say, but let's start with a top-level picture, > to make sure we're all on the same page. > > WWAN Communication > Channel (Physical) > | ------------------------ > ------------ v | :+ Control | \ > | |-----------| :+ Data | | > | AP | | WWAN unit :+ Voice | > Functions > | |===========| :+ GPS | | > ------------ ^ | :+ ... | / > | ------------------------- > Multiplexed WWAN > Communication > Channel (Physical) > > - The *AP* is the main CPU complex that's running Linux on one or > more CPU cores. > - A *WWAN unit* is an entity that shares one or more physical > *WWAN communication channels* with the AP. > - A *WWAN communication channel* is a bidirectional means of > carrying data between the AP and WWAN unit. > - A WWAN communication channel carries data using a *WWAN protocol*. > - A WWAN unit implements one or more *WWAN functions*, such as > 5G data, LTE voice, GPS, and so on. > - A WWAN unit shall implement a *WWAN control function*, used to > manage the use of other WWAN functions, as well as the WWAN unit > itself. > - The AP communicates with a WWAN function using a WWAN protocol. > - A WWAN physical channel can be *multiplexed*, in which case it > carries the data for one or more *WWAN logical channels*. > - A multiplexed WWAN communication channel uses a *WWAN wultiplexing > protocol*, which is used to separate independent data streams > carrying other WWAN protocols. > - A WWAN logical channel carries a bidirectional stream of WWAN > protocol data between an entity on the AP and a WWAN function. > > Does that adequately represent a very high-level picture of what > we're trying to manage? > > And if I understand it right, the purpose of the generic framework > being discussed is to define a common mechanism for managing (i.e., > discovering, creating, destroying, querying, configuring, enabling, > disabling, etc.) WWAN units and the functions they implement, along > with the communication and logical channels used to communicate with > them. > > Comments? > > -Alex >
On Mon, 2019-06-24 at 11:30 -0500, Alex Elder wrote: > OK I want to try to organize a little more concisely some of the > discussion on this, because there is a very large amount of volume > to date and I think we need to try to narrow the focus back down > again. > > I'm going to use a few terms here. Some of these I really don't > like, but I want to be unambiguous *and* (at least for now) I want > to avoid the very overloaded term "device". > > I have lots more to say, but let's start with a top-level picture, > to make sure we're all on the same page. > > WWAN Communication > Channel (Physical) > | ------------------------ > ------------ v | :+ Control | \ > > |-----------| :+ Data | | > > AP | | WWAN unit :+ Voice | > Functions > > |===========| :+ GPS | | > ------------ ^ | :+ ... | / > | ------------------------- > Multiplexed WWAN > Communication > Channel (Physical) > > - The *AP* is the main CPU complex that's running Linux on one or > more CPU cores. > - A *WWAN unit* is an entity that shares one or more physical > *WWAN communication channels* with the AP. You could just say "WWAN modem" here. > - A *WWAN communication channel* is a bidirectional means of > carrying data between the AP and WWAN unit. > - A WWAN communication channel carries data using a *WWAN protocol*. > - A WWAN unit implements one or more *WWAN functions*, such as > 5G data, LTE voice, GPS, and so on. Go more generic here. Not just 5G data but any WWAN IP-based data (GPRS, EDGE, CDMA, UMTS, EVDO, LTE, 5G, etc). And not just LTE voice but any voice data; plenty of devices don't support LTE but still have "WWAN logical communication channels" > - A WWAN unit shall implement a *WWAN control function*, used to > manage the use of other WWAN functions, as well as the WWAN unit > itself. > - The AP communicates with a WWAN function using a WWAN protocol. > - A WWAN physical channel can be *multiplexed*, in which case it > carries the data for one or more *WWAN logical channels*. It's unclear to me what "physical" means here. USB Interface or Endpoint or PCI Function or SMD channel? Or kernel TTY device? For example on Qualcomm-based USB dongles a given USB Interface's Endpoint represents a QMAP "IP data" channel which itself could be multiplexed into separate "IP data" channels. Or that USB Endpoint(s) could be exposed as a TTY which itself can be MUX-ed dynamically using GSM 07.10. To me "physical" usually means the bus type (PCI, USB, SMD, whatever). A Linux hardware driver (IPA, qmi_wwan, option, sierra, etc) binds to that physical entity using hardware IDs (USB or PCI VID/PID, devicetree properties) and exposes some "WWAN logical communication channels". Those logical channels might be multiplexed and another driver (rmnet) could handle exposing the de-muxed logical channels that the muxed logical channel carries. > - A multiplexed WWAN communication channel uses a *WWAN wultiplexing > protocol*, which is used to separate independent data streams > carrying other WWAN protocols. > - A WWAN logical channel carries a bidirectional stream of WWAN > protocol data between an entity on the AP and a WWAN function. It *usually* is bidirectional. For example some GPS logical communication channels just start spitting out NMEA when you give the control function a command. The NMEA ports themselves don't accept any input. > Does that adequately represent a very high-level picture of what > we're trying to manage? Yes, pretty well. Thanks for trying to specify it all. > And if I understand it right, the purpose of the generic framework > being discussed is to define a common mechanism for managing (i.e., > discovering, creating, destroying, querying, configuring, enabling, > disabling, etc.) WWAN units and the functions they implement, along > with the communication and logical channels used to communicate with > them. Yes. Dan > Comments? > > -Alex
On 6/24/19 2:54 PM, Dan Williams wrote: > On Mon, 2019-06-24 at 11:30 -0500, Alex Elder wrote: >> OK I want to try to organize a little more concisely some of the >> discussion on this, because there is a very large amount of volume >> to date and I think we need to try to narrow the focus back down >> again. >> >> I'm going to use a few terms here. Some of these I really don't >> like, but I want to be unambiguous *and* (at least for now) I want >> to avoid the very overloaded term "device". >> >> I have lots more to say, but let's start with a top-level picture, >> to make sure we're all on the same page. >> >> WWAN Communication >> Channel (Physical) >> | ------------------------ >> ------------ v | :+ Control | \ >>> |-----------| :+ Data | | >>> AP | | WWAN unit :+ Voice | > Functions >>> |===========| :+ GPS | | >> ------------ ^ | :+ ... | / >> | ------------------------- >> Multiplexed WWAN >> Communication >> Channel (Physical) >> >> - The *AP* is the main CPU complex that's running Linux on one or >> more CPU cores. >> - A *WWAN unit* is an entity that shares one or more physical >> *WWAN communication channels* with the AP. > > You could just say "WWAN modem" here. That sounds great to me. >> - A *WWAN communication channel* is a bidirectional means of >> carrying data between the AP and WWAN unit. >> - A WWAN communication channel carries data using a *WWAN protocol*. >> - A WWAN unit implements one or more *WWAN functions*, such as >> 5G data, LTE voice, GPS, and so on. > > Go more generic here. Not just 5G data but any WWAN IP-based data > (GPRS, EDGE, CDMA, UMTS, EVDO, LTE, 5G, etc). And not just LTE voice > but any voice data; plenty of devices don't support LTE but still have > "WWAN logical communication channels" I really meant *any* sort of function, and was only trying to give a few examples. So yes, my meaning was completely generic, as you suggest. >> - A WWAN unit shall implement a *WWAN control function*, used to >> manage the use of other WWAN functions, as well as the WWAN unit >> itself. >> - The AP communicates with a WWAN function using a WWAN protocol. >> - A WWAN physical channel can be *multiplexed*, in which case it >> carries the data for one or more *WWAN logical channels*. > > It's unclear to me what "physical" means here. USB Interface or > Endpoint or PCI Function or SMD channel? Or kernel TTY device? I'm trying to distinguish between (let's say) "hardware" communication channels (such as what IPA basically provides) and logical ones, whose data is multiplexed over a "hardware"/"physical" channel. Maybe "link" would be a better term for what I referred to as a physical channel, and then have "channels" be multiplexed over a link. If you have a good suggestion, please offer it. But I think yes, USB interface, TTY device, etc. are what I mean. I wanted to capture the fact that there might be more than one of these (for example a user space QMI link for control, and IPA link for data?), and that any one of these might also be multiplexed. Multiplexing is a pretty basic capability of a network link, and I think the only reason I called it out separately is to be explicit that it is needs to be supported. > For example on Qualcomm-based USB dongles a given USB Interface's > Endpoint represents a QMAP "IP data" channel which itself could be > multiplexed into separate "IP data" channels. Or that USB Endpoint(s) > could be exposed as a TTY which itself can be MUX-ed dynamically using > GSM 07.10. Yeah. In this case the hardware connection between the AP and the USB dongle would provide a WWAN link (which I think corresponds to the QMAP "IP data" channel). And if you wanted to multiplex that you would use a multiplexing protocol (like QMAP). And that protocol would carry one or more logical channels, each using its own WWAN protocol. It sounds like GSM 07.10 would be another WWAN multiplexing protocol. > To me "physical" usually means the bus type (PCI, USB, SMD, whatever). > A Linux hardware driver (IPA, qmi_wwan, option, sierra, etc) binds to > that physical entity using hardware IDs (USB or PCI VID/PID, devicetree > properties) and exposes some "WWAN logical communication channels". (Or perhaps exposes the ability to create "WWAN logical channels.") "Physical" is probably not a good term. And perhaps focusing on the transport isn't right--maybe the focus should mainly be on the WWAN modem entity. But one of the things we're trying to address here is that there might be several distinct "physical" paths through which the AP and modem can communicate, so a driver's ability to provide such a path should be a sort of first class abstraction. I'm really trying to get this discussion to converge a little, to have a few anchor points to build on. I hope we're getting there. > Those logical channels might be multiplexed and another driver (rmnet) > could handle exposing the de-muxed logical channels that the muxed > logical channel carries. > >> - A multiplexed WWAN communication channel uses a *WWAN wultiplexing >> protocol*, which is used to separate independent data streams >> carrying other WWAN protocols. >> - A WWAN logical channel carries a bidirectional stream of WWAN >> protocol data between an entity on the AP and a WWAN function. > > It *usually* is bidirectional. For example some GPS logical > communication channels just start spitting out NMEA when you give the > control function a command. The NMEA ports themselves don't accept any > input. That's fine... I don't think there's anything wrong with a particular application not using both directions. >> Does that adequately represent a very high-level picture of what >> we're trying to manage? > > Yes, pretty well. Thanks for trying to specify it all. I think we're making progress. I have some more thoughts but I think I'll wait until tomorrow to share them. Thanks a lot Dan. At some point it might be better to have a conference call to make better progress, but I'm not suggesting that yet. -Alex >> And if I understand it right, the purpose of the generic framework >> being discussed is to define a common mechanism for managing (i.e., >> discovering, creating, destroying, querying, configuring, enabling, >> disabling, etc.) WWAN units and the functions they implement, along >> with the communication and logical channels used to communicate with >> them. > > Yes. > > Dan > >> Comments? >> >> -Alex >
Hi Alex, I'll just pick a few or your messages and reply there - some other subthreads seem to have pretty much completed. > Sorry for the delay. There's a lot here to go through, and with > each message the picture is (slowly) getting a bit clearer for me. > Still, there are some broad tradeoffs to consider and I think we > need to get a little more specific again. I'm going to start a > new thread (or rather re-subject a response to the very first one) > that tries to do a fresh start that takes into account the > discussion so far. > > I will also be talking with some people inside Qualcomm (including > Subash) soon to make sure we don't miss any requirements or insights > they know of that I don't realize are important. That's much appreciated. > > Linux usually tries to keep drivers generic and focused; each driver is > > written for a specific function. For example, a USB device usually > > provides multiple USB interfaces which will be bound to different Linux > > drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc. > > So USB has some attributes similar to what we're talking about > here. But if I'm not mistaken we want some sort of an overall > management scheme as well. Yes. For the record, I think the part about "keep drivers generic and focused" really only works for USB devices that expose different pieces that look like any other network device or a TTY device on the USB level, just the combination of these things (and knowing about that) really makes them a modem. For things like IPA or the (hypothetical) Intel driver we're talking about, it's still all managed by a single (PCIe) driver. For the Intel device in particular, even all the control channels are over exactly the same transport mechanism as the data channels. > > These drivers are often generic and we may not have enough information > > in one driver to know that the parent of this interface is a WWAN > > device. But another driver might. Since probing is asynchronous we may > > have cdc-acm bind to a device and provide a TTY before cdc-ether (which > > does know it's a WWAN) binds and provides the netdevice. > > Is this why Johannes wanted to have a "maybe attach" method? Yes. > I don't like the "maybe" API unless there's no other way to do it. > > Instead I think it would be better for the probing driver to register > with a whatever the WWAN core is, and then have the WWAN core be > responsible for pulling things all together when it receives a > request to do so. I.e., something in user space should request > that a registered data interface be brought up, and at that > time everything "knows" it's implemented as part of a WWAN > device. Right, but then we just punt to userspace. Mostly we *do* (eventually!) know that it's a WWAN device, just not every component can detect it. Some components typically can. So for example, you might have a USB multi-function device with a network function (looks just like ethernet pretty much) but another TTY control channel that actually has some specific WWAN IDs, so that part can know it's a WWAN. Here, the ethernet function would need "maybe" attach, and the control channel would "definitively" attach, pulling it together as a WWAN device without requiring any more action or information. > So maybe: > - Hardware probe detects a WWAN device > - The drivers that detect the WWAN device register it with the > WWAN core code. > - A control channel is instantiated at/before the time the WWAN > device is registered > - Something in user space should manage the bring-up of any > other things on the WWAN device thereafter But those things need to actually get connected first :-) In IPA/Intel case this is easy since it's a single driver. But if there's multi-function device with ethernet being a completely separate driver, the control channel cannot even reach that to tell it to create a new data channel. > > userspace should probably always create the netdevices (since they are > > always useless until userspace coordinates with the firmware about > > them) but that's not how things are yet. > > That's too bad. How hard would that be to change? Depends, but as I said above it's probably orthogonal to the question. The data channel driver would still need to attach to the WWAN device somehow so it becomes reachable by the control plane (note this isn't the same as "control channel" since the latter talks to the modem, the control plane talks to the kernel drivers). > > > - What causes a created channel to be removed? > > > > Driver removal, userspace WWAN daemon terminating the packet data > > connection which the channel represents, the modem terminating the > > packet data connection (eg network initiated disconnect), etc. > > OK this is as I expected. Driver (or device) removal is somewhat > obvious, but you're confirming user space might request it as well. If userspace actually had the ability to create (data) channels, then it would have the ability to also remove them. Right now, this may or may not be supported by the drivers that act together to form the interfaces to a WWAN device. > > > - You distinguish between attaching a netdevice and (what > > > I'll call) activating it. What causes activation? > > > > Can you describe what you mean by "activating"? Do you mean > > successfully TX/RX packets via the netdev and the outside world? > > Johannes mentioned an API to "maybe attach" a device. That begs > the question of what happens if this request does *not* attach. > Does the attach request have to be made again, or is it done > automatically with a notification, or something else? > > So by "activation" I was trying to refer to the notion of this > subsequent successful attach. Oh. Well, what I was thinking that "maybe attach" would just be a sort of "in-limbo" WWAN device that doesn't get visible to userspace or the control plane until something did a "definitively attach" to it so it was known to be a WWAN device. The case of "maybe attach but never get to definitive attach" would be the case where the USB driver bound a real ethernet device, for example, not something that looks like an ethernet device but really is part of a modem. OTOH, "activating" a data channel is also needed somehow through the control channel by talking to the modem, i.e. making a connection. In the ideal case we'd not even *have* a netdev until it makes sense to create a data channel, but in reality a lot of devices have one around all the time (or even only support one), AFAICT. > > I read "attach" here as simply associating an existing netdev with the > > "parent" WWAN device. A purely Linux operation that is only book- > > keeping and may not have any interaction with the modem. > > If that's the case I would want the "activation" to be a separate > step. The attach would do the bookkeeping, and generally shouldn't > fail. An attached interface would be brought up ("activated") > separately and might fail if things aren't quite ready yet. Right, but netdevs need to be brought up anyway, and that can fail? > > > - How are the attributes of a WWAN device or channel set, > > > or communicated? > > > > Via netlink attributes when userspace asks the WWAN device to create a > > new channel. In the control methods I've seen, only userspace really > > knows the channel identifier that it and the modem have agreed on (eg > > what the MUX ID in the QMAP header would be, or the MBIM Session ID). > > Yes, that's the way it's worked for rmnet and IPA. Previously it > was IOCTL requests but it's currently hard-wired. Right. We're just trying to lift it out of the Qualcomm sphere into something more generically useful. johannes
On Mon, 2019-06-24 at 18:40 +0200, Arnd Bergmann wrote: > On Mon, Jun 24, 2019 at 6:21 PM Alex Elder <elder@linaro.org> wrote: > > On 6/18/19 2:03 PM, Johannes Berg wrote: > > > > > Really there are two possible ways (and they intersect to some extent). > > > > > > One is the whole multi-function device, where a single WWAN device is > > > composed of channels offered by actually different drivers, e.g. for a > > > typical USB device you might have something like cdc_ether and the > > > usb_wwan TTY driver. In this way, we need to "compose" the WWAN device > > > similarly, e.g. by using the underlying USB device "struct device" > > > pointer to tie it together. > > > > I *think* this model makes the most sense. But at this point > > it would take very little to convince me otherwise... (And then > > I saw Arnd's message advocating the other one, unfortunately...) > > > > > The other is something like IPA or the Intel modem driver, where the > > > device is actually a single (e.g. PCIe) device and just has a single > > > driver, but that single driver offers different channels. > > > > What I don't like about this is that it's more monolithic. It > > seems better to have the low-level IPA or Intel modem driver (or > > any other driver that can support communication between the AP > > and WWAN device) present communication paths that other function- > > specific drivers can attach to and use. > > I did not understand Johannes description as two competing models > for the same code, but rather two kinds of existing hardware that > a new driver system would have to deal with. Right. > I was trying to simplify it to just having the second model, by adding > a hack to support the first, but my view was rather unpopular so > far, so if everyone agrees on one way to do it, don't worry about me ;-) :-) However, to also reply to Alex: I don't know exactly how IPA works, but for the Intel modem at least you can't fundamentally have two drivers for different parts of the functionality, since it's just a single piece of hardware and you need to allocate hardware resources from a common pool etc. So you cannot split the driver into "Intel modem control channel driver" and "Intel modem data channel driver". In fact, it's just a single "struct device" on the PCIe bus that you can bind to, and only one driver can bind at a time. So, IOW, I'm not sure I see how you'd split that up. I guess you could if you actually do something like the "rmnet" model, and I suppose you're free to do that for IPA if you like, but I tend to think that's actually a burden, not a win since you just get more complex code that needs to interact with more pieces. A single driver for a single hardware that knows about the few types of channels seems simpler to me. > - to answer Johannes question, my understanding is that the interface > between kernel and firmware/hardware for IPA has a single 'struct > device' that is used for both the data and the control channels, > rather than having a data channel and an independent control device, > so this falls into the same category as the Intel one (please correct > me on that) That sounds about the same then, right. Are the control channels to IPA are actually also tunnelled over the rmnet protocol? And even if they are, perhaps they have a different hardware queue or so? That'd be the case for Intel - different hardware queue, same (or at least similar) protocol spoken for the DMA hardware itself, but different contents of the messages obviously. > - The user space being proprietary is exactly what we need to avoid > with the wwan subsystem. We need to be able to use the same > method for setting up Intel, Qualcomm, Samsung, Unisoc or > Hisilicon modems or anything else that hooks into the subsystem, > and support that in network manager as well as the Android > equivalent. > If Qualcomm wants to provide their own proprietary user space > solution, we can't stop them, but then that should also work on > all the others unless they intentionally break it. ;-) :-) I tend to think there's always going to be some level of specific handling here, because e.g. the Intel modem can expose MBIM or AT command control channels, but not much else (that'd be useful for us anyway, since we don't know how to speak debug protocol etc.). Other modems will expose *only* AT commands, or *only* MBIM, and yet others may also offer QMI and then that might be preferable. > > > and simply require that the channel is attached to the wwan device with > > > the representation-specific call (wwan_attach_netdev, wwan_attach_tty, > > > ...). > > > > Or maybe have the WWAN device present interfaces with attributes, > > and have drivers that are appropriate for each interface attach > > to only the ones they recognize they support. > > I think you both mean the same thing here, a structure with callback > pointers that may or may not be filled by the driver depending on its > capabilities. :-) > What we should try to avoid though is a way to add driver private > interfaces that risk having multiple drivers create similar functionality > in incompatible ways. Right. johannes
On Mon, 2019-06-24 at 12:06 -0500, Alex Elder wrote: > > OK I want to try to organize a little more concisely some of the > > discussion on this, because there is a very large amount of volume > > to date and I think we need to try to narrow the focus back down > > again. Sounds good to me! > > I'm going to use a few terms here. Some of these I really don't > > like, but I want to be unambiguous *and* (at least for now) I want > > to avoid the very overloaded term "device". > > > > I have lots more to say, but let's start with a top-level picture, > > to make sure we're all on the same page. > > > > WWAN Communication > > Channel (Physical) > > | ------------------------ > > ------------ v | :+ Control | \ > > > |-----------| :+ Data | | > > > AP | | WWAN unit :+ Voice | > Functions > > > |===========| :+ GPS | | > > > > ------------ ^ | :+ ... | / > > | ------------------------- > > Multiplexed WWAN > > Communication > > Channel (Physical) Sounds right to me. I'm not sure if you're distinguishing here between the "Data function" and multiple data channels to the data function, but at this point I guess it doesn't matter. > > - The *AP* is the main CPU complex that's running Linux on one or > > more CPU cores. > > - A *WWAN unit* is an entity that shares one or more physical > > *WWAN communication channels* with the AP. > > - A *WWAN communication channel* is a bidirectional means of > > carrying data between the AP and WWAN unit. > > - A WWAN communication channel carries data using a *WWAN protocol*. > > - A WWAN unit implements one or more *WWAN functions*, such as > > 5G data, LTE voice, GPS, and so on. > > - A WWAN unit shall implement a *WWAN control function*, used to > > manage the use of other WWAN functions, as well as the WWAN unit > > itself. I think here we need to be more careful. I don't know how you want to call it, but we actually have multiple levels of control here. You have * hardware control, to control how you actually use the (multiple or not) physical communication channel(s) to the WWAN unit * this is partially exposed to userspace via the WWAN netlink family or something like that, so userspace can create new netdevs to tx/rx with the "data function" and to the network; note that it could be one or multiple * WWAN control, which is typically userspace communicating with the WWAN control function in the WWAN unit, but this can take different forms (as I mentioned earlier, e.g. AT commands, MBIM, QMI) > > - The AP communicates with a WWAN function using a WWAN protocol. Right, that's just device specific (IPA vs. Intel vs. ...) > > - A WWAN physical channel can be *multiplexed*, in which case it > > carries the data for one or more *WWAN logical channels*. This ... depends a bit on how you exactly define a physical channel here. Is that, to you, the PCIe/USB link? In that case, yes, obviously you have only one physical channel for each WWAN unit. However, I'd probably see this slightly differently, because e.g. the Intel modem has multiple DMA engines, and so you actually have multiple DMA rings to talk to the WWAN unit, and I'd have called each DMA ring a physical channel. And then, you just have a 1:1 from physical to logical channel since it doesn't actually carry a multiplexing protocol. > > - A multiplexed WWAN communication channel uses a *WWAN wultiplexing > > protocol*, which is used to separate independent data streams > > carrying other WWAN protocols. Like just described, this isn't really needed and is a device-specific property. > > - A WWAN logical channel carries a bidirectional stream of WWAN > > protocol data between an entity on the AP and a WWAN function. > > > > Does that adequately represent a very high-level picture of what > > we're trying to manage? Pretty much. I only disagree slightly on the control planes (there are multiple, and multiple options for the "Control function" one), and on the whole notion of physical link/logical link/multiplexing which is device specific. > > And if I understand it right, the purpose of the generic framework > > being discussed is to define a common mechanism for managing (i.e., > > discovering, creating, destroying, querying, configuring, enabling, > > disabling, etc.) WWAN units and the functions they implement, along > > with the communication and logical channels used to communicate with > > them. Well, some subset of that matrix, the framework won't actually destroy WWAN units I hope ;-) But yes. I'd probably captured it in layers, and say that we have a WWAN framework layer - discover, query, configure WWAN units - enable, disable channels to the functions inside the WWAN units WWAN device driver - implement (partial) API offered by WWAN framework layer to allow these things (sometimes may not allow creating more control or data channels for example, and fixed function channels are precreated, but then can still discover data about the device and configure the channels - implement the device-specific protocols etc. necessary to achieve this Userspace - uses control function channel (e.g. TTY) to talk directly to the WWAN unit's control function - uses WWAN framework APIs to create/configure/... (other) function channels (it may be necessary to create a control channel even, before being able to use it, since different options (AT/MBIM/QMI) may be there - configures netdevs (data function channels) after their creation johannes
On 6/25/19 9:14 AM, Johannes Berg wrote: > Hi Alex, > > I'll just pick a few or your messages and reply there - some other > subthreads seem to have pretty much completed. > . . . >>> Linux usually tries to keep drivers generic and focused; each driver is >>> written for a specific function. For example, a USB device usually >>> provides multiple USB interfaces which will be bound to different Linux >>> drivers like a TTY, cdc-ether, QMI (via qmi_wwan), cdc-acm, etc. >> >> So USB has some attributes similar to what we're talking about >> here. But if I'm not mistaken we want some sort of an overall >> management scheme as well. > > Yes. For the record, I think the part about "keep drivers generic and > focused" really only works for USB devices that expose different pieces > that look like any other network device or a TTY device on the USB > level, just the combination of these things (and knowing about that) > really makes them a modem. > > For things like IPA or the (hypothetical) Intel driver we're talking > about, it's still all managed by a single (PCIe) driver. For the Intel > device in particular, even all the control channels are over exactly the > same transport mechanism as the data channels. Actually the setup for IPA requires certain things to be done via QMI by something outside the IPA driver, and it uses a separate communication path. But I understand what you're saying. . . . >> I don't like the "maybe" API unless there's no other way to do it. >> >> Instead I think it would be better for the probing driver to register >> with a whatever the WWAN core is, and then have the WWAN core be >> responsible for pulling things all together when it receives a >> request to do so. I.e., something in user space should request >> that a registered data interface be brought up, and at that >> time everything "knows" it's implemented as part of a WWAN >> device. > > Right, but then we just punt to userspace. Mostly we *do* (eventually!) > know that it's a WWAN device, just not every component can detect it. > Some components typically can. We need to identify the existence of a WWAN device (which is I guess--typically? always?--a modem). Perhaps that can be discovered in some cases but I think it means a node described by Device Tree. > So for example, you might have a USB multi-function device with a > network function (looks just like ethernet pretty much) but another TTY > control channel that actually has some specific WWAN IDs, so that part > can know it's a WWAN. > > Here, the ethernet function would need "maybe" attach, and the control > channel would "definitively" attach, pulling it together as a WWAN > device without requiring any more action or information. So you're saying you have a single Ethernet driver, and it can drive an Ethernet device connected to a WWAN, or not connected to a WWAN, without any changes. The only distinction is that if the device is part of a WWAN it needs to register with the WWAN framework. Is that right? >> So maybe: >> - Hardware probe detects a WWAN device >> - The drivers that detect the WWAN device register it with the >> WWAN core code. >> - A control channel is instantiated at/before the time the WWAN >> device is registered >> - Something in user space should manage the bring-up of any >> other things on the WWAN device thereafter > > But those things need to actually get connected first :-) What I meant is that the registering with the "WWAN core code" is what does that "connecting." The WWAN code has the information about what got registered. But as I said above, this WWAN device needs to be identified, and I think (at least for IPA) that will require something in Device Tree. That will "connect" them. Or I might be misunderstanding your point. > In IPA/Intel case this is easy since it's a single driver. But if > there's multi-function device with ethernet being a completely separate > driver, the control channel cannot even reach that to tell it to create > a new data channel. There's a lot more to talk about with control. I think you discuss this in another message, and I'll get to that shortly. But I think understand your point, and again I think it comes back to having an abstraction that represents the modem, distinct from (but "connected" to) the functions it implements/includes. >>> userspace should probably always create the netdevices (since they are >>> always useless until userspace coordinates with the firmware about >>> them) but that's not how things are yet. >> >> That's too bad. How hard would that be to change? > > Depends, but as I said above it's probably orthogonal to the question. > The data channel driver would still need to attach to the WWAN device > somehow so it becomes reachable by the control plane (note this isn't > the same as "control channel" since the latter talks to the modem, the > control plane talks to the kernel drivers). > >>>> - What causes a created channel to be removed? >>> >>> Driver removal, userspace WWAN daemon terminating the packet data >>> connection which the channel represents, the modem terminating the >>> packet data connection (eg network initiated disconnect), etc. >> >> OK this is as I expected. Driver (or device) removal is somewhat >> obvious, but you're confirming user space might request it as well. > > If userspace actually had the ability to create (data) channels, then it > would have the ability to also remove them. Right now, this may or may > not be supported by the drivers that act together to form the interfaces > to a WWAN device. I think this (user space control) needs to be an option, but it doesn't have to be the only way. . . . You made some other good clarifications in this message but I'm going to try to capture them elsewhere rather than respond here. -Alex
On 6/25/19 9:19 AM, Johannes Berg wrote: > On Mon, 2019-06-24 at 18:40 +0200, Arnd Bergmann wrote: >> On Mon, Jun 24, 2019 at 6:21 PM Alex Elder <elder@linaro.org> wrote: >>> On 6/18/19 2:03 PM, Johannes Berg wrote: >>> >>>> Really there are two possible ways (and they intersect to some extent). >>>> >>>> One is the whole multi-function device, where a single WWAN device is >>>> composed of channels offered by actually different drivers, e.g. for a >>>> typical USB device you might have something like cdc_ether and the >>>> usb_wwan TTY driver. In this way, we need to "compose" the WWAN device >>>> similarly, e.g. by using the underlying USB device "struct device" >>>> pointer to tie it together. >>> >>> I *think* this model makes the most sense. But at this point >>> it would take very little to convince me otherwise... (And then >>> I saw Arnd's message advocating the other one, unfortunately...) >>> >>>> The other is something like IPA or the Intel modem driver, where the >>>> device is actually a single (e.g. PCIe) device and just has a single >>>> driver, but that single driver offers different channels. >>> >>> What I don't like about this is that it's more monolithic. It >>> seems better to have the low-level IPA or Intel modem driver (or >>> any other driver that can support communication between the AP >>> and WWAN device) present communication paths that other function- >>> specific drivers can attach to and use. >> >> I did not understand Johannes description as two competing models >> for the same code, but rather two kinds of existing hardware that >> a new driver system would have to deal with. > > Right. > >> I was trying to simplify it to just having the second model, by adding >> a hack to support the first, but my view was rather unpopular so >> far, so if everyone agrees on one way to do it, don't worry about me ;-) > > :-) > > However, to also reply to Alex: I don't know exactly how IPA works, but > for the Intel modem at least you can't fundamentally have two drivers > for different parts of the functionality, since it's just a single piece > of hardware and you need to allocate hardware resources from a common > pool etc. So you cannot split the driver into "Intel modem control > channel driver" and "Intel modem data channel driver". In fact, it's > just a single "struct device" on the PCIe bus that you can bind to, and > only one driver can bind at a time. Interesting. So a single modem driver needs to implement *all* of the features/functions? Like GPS or data log or whatever, all needs to share the same struct device? Or does what you're describing apply to a subset of the modem's functionality? Or something else? > So, IOW, I'm not sure I see how you'd split that up. I guess you could > if you actually do something like the "rmnet" model, and I suppose > you're free to do that for IPA if you like, but I tend to think that's > actually a burden, not a win since you just get more complex code that > needs to interact with more pieces. A single driver for a single > hardware that knows about the few types of channels seems simpler to me. > >> - to answer Johannes question, my understanding is that the interface >> between kernel and firmware/hardware for IPA has a single 'struct >> device' that is used for both the data and the control channels, >> rather than having a data channel and an independent control device, >> so this falls into the same category as the Intel one (please correct >> me on that) I don't think that's quite right, but it might be partially right. There is a single device representing IPA, but the picture is a little more complicated. The IPA hardware is actually something that sits *between* the AP and the modem. It implements one form of communication pathway (IP data), but there are others (including QMI, which presents a network-like interface but it's actually implemented via clever use of shared memory and interrupts). What we're talking about here is WWAN/modem management more generally though. It *sounds* like the Intel modem is more like a single device, which requires a single driver, that seems to implement a bunch of distinct functions. On this I'm not very knowledgeable but for Qualcomm there is user space code that is in charge of overall management of the modem. It implements what I think you're calling control functions, negotiating with the modem to allow new data channels to be created. Normally the IPA driver would provide information to user space about available resources, but would only make a communication pathway available when requested. I'm going to leave it at that for now. > That sounds about the same then, right. > > Are the control channels to IPA are actually also tunnelled over the > rmnet protocol? And even if they are, perhaps they have a different > hardware queue or so? That'd be the case for Intel - different hardware > queue, same (or at least similar) protocol spoken for the DMA hardware > itself, but different contents of the messages obviously. I want to be careful talking about "control" but for IPA it comes from user space. For the purpose of getting initial code upstream, all of that control functionality (which was IOCTL based) has been removed, and a fixed configuration is assumed. >> - The user space being proprietary is exactly what we need to avoid >> with the wwan subsystem. We need to be able to use the same >> method for setting up Intel, Qualcomm, Samsung, Unisoc or >> Hisilicon modems or anything else that hooks into the subsystem, >> and support that in network manager as well as the Android >> equivalent. >> If Qualcomm wants to provide their own proprietary user space >> solution, we can't stop them, but then that should also work on >> all the others unless they intentionally break it. ;-) I won't comment on this, in part because I really don't know right now what is proprietary or why. I think that having user space (proprietary or not) be able to provide management capability is a good thing. If a unified kernel interface provides a common/generic way to manage the modem, I don't know why Qualcomm wouldn't adapt their code to use it. But I can't really speak for Qualcomm. . . . -Alex
On 6/25/19 9:34 AM, Johannes Berg wrote: > On Mon, 2019-06-24 at 12:06 -0500, Alex Elder wrote: > >>> OK I want to try to organize a little more concisely some of the >>> discussion on this, because there is a very large amount of volume >>> to date and I think we need to try to narrow the focus back down >>> again. > > Sounds good to me! . . . >>> - A WWAN unit shall implement a *WWAN control function*, used to >>> manage the use of other WWAN functions, as well as the WWAN unit >>> itself. > > I think here we need to be more careful. I don't know how you want to > call it, but we actually have multiple levels of control here. I completely agree with you. From what I understand there exists a control channel (or even more than one?) that serves a very specific purpose in modem management. The main reason I mention the WWAN control function is that someone (maybe you) indicated that a control channel automatically gets created. But I agree, we need to be careful to avoid confusion here. > You have > * hardware control, to control how you actually use the (multiple or > not) physical communication channel(s) to the WWAN unit > * this is partially exposed to userspace via the WWAN netlink family or > something like that, so userspace can create new netdevs to tx/rx > with the "data function" and to the network; note that it could be > one or multiple > * WWAN control, which is typically userspace communicating with the > WWAN control function in the WWAN unit, but this can take different > forms (as I mentioned earlier, e.g. AT commands, MBIM, QMI) > >>> - The AP communicates with a WWAN function using a WWAN protocol. > > Right, that's just device specific (IPA vs. Intel vs. ...) > >>> - A WWAN physical channel can be *multiplexed*, in which case it >>> carries the data for one or more *WWAN logical channels*. > > This ... depends a bit on how you exactly define a physical channel > here. Is that, to you, the PCIe/USB link? In that case, yes, obviously > you have only one physical channel for each WWAN unit. I think that was what I was trying to capture. There exists one or more "physical" communication paths between the AP and WWAN unit/modem. And while one path *could* carry just one type of traffic, it could also carry multiple logical channels of traffic by multiplexing. > However, I'd probably see this slightly differently, because e.g. the > Intel modem has multiple DMA engines, and so you actually have multiple > DMA rings to talk to the WWAN unit, and I'd have called each DMA ring a > physical channel. And then, you just have a 1:1 from physical to logical > channel since it doesn't actually carry a multiplexing protocol. Understood. . . . > I only disagree slightly on the control planes (there are multiple, and > multiple options for the "Control function" one), and on the whole > notion of physical link/logical link/multiplexing which is device > specific. > >>> And if I understand it right, the purpose of the generic framework >>> being discussed is to define a common mechanism for managing (i.e., >>> discovering, creating, destroying, querying, configuring, enabling, >>> disabling, etc.) WWAN units and the functions they implement, along >>> with the communication and logical channels used to communicate with >>> them. > > Well, some subset of that matrix, the framework won't actually destroy > WWAN units I hope ;-) Hardware self-destruct would be an optional behavior. > But yes. I'd probably captured it in layers, and say that we have a > > WWAN framework layer > - discover, query, configure WWAN units > - enable, disable channels to the functions inside the WWAN units > > WWAN device driver > - implement (partial) API offered by WWAN framework layer to allow > these things > (sometimes may not allow creating more control or data channels for > example, and fixed function channels are precreated, but then can > still discover data about the device and configure the channels > - implement the device-specific protocols etc. necessary to achieve > this > > Userspace > - uses control function channel (e.g. TTY) to talk directly to the WWAN > unit's control function > - uses WWAN framework APIs to create/configure/... (other) function > channels > (it may be necessary to create a control channel even, before being > able to use it, since different options (AT/MBIM/QMI) may be there > - configures netdevs (data function channels) after their creation I don't think I have any argument with this. I'm going to try to put together something that goes beyond what I wrote in this message, to try to capture what I think we agree on in a sort of loose design document. Thanks Johannes. -Alex
On 6/24/19 11:40 AM, Arnd Bergmann wrote: > On Mon, Jun 24, 2019 at 6:21 PM Alex Elder <elder@linaro.org> wrote: >> On 6/18/19 2:03 PM, Johannes Berg wrote: >> >>> Really there are two possible ways (and they intersect to some extent). . . . >>> The other is something like IPA or the Intel modem driver, where the >>> device is actually a single (e.g. PCIe) device and just has a single >>> driver, but that single driver offers different channels. >> >> What I don't like about this is that it's more monolithic. It >> seems better to have the low-level IPA or Intel modem driver (or >> any other driver that can support communication between the AP >> and WWAN device) present communication paths that other function- >> specific drivers can attach to and use. > > I did not understand Johannes description as two competing models > for the same code, but rather two kinds of existing hardware that > a new driver system would have to deal with. Based on my understanding of what he said in a message I just responded to, I think you are exactly right. . . . > What we should try to avoid though is a way to add driver private > interfaces that risk having multiple drivers create similar functionality > in incompatible ways. Agreed. -Alex
On Wed, Jun 26, 2019 at 3:39 PM Alex Elder <elder@linaro.org> wrote: > On 6/25/19 9:19 AM, Johannes Berg wrote: > > On Mon, 2019-06-24 at 18:40 +0200, Arnd Bergmann wrote: > > > So, IOW, I'm not sure I see how you'd split that up. I guess you could > > if you actually do something like the "rmnet" model, and I suppose > > you're free to do that for IPA if you like, but I tend to think that's > > actually a burden, not a win since you just get more complex code that > > needs to interact with more pieces. A single driver for a single > > hardware that knows about the few types of channels seems simpler to me. > > > >> - to answer Johannes question, my understanding is that the interface > >> between kernel and firmware/hardware for IPA has a single 'struct > >> device' that is used for both the data and the control channels, > >> rather than having a data channel and an independent control device, > >> so this falls into the same category as the Intel one (please correct > >> me on that) > > I don't think that's quite right, but it might be partially > right. There is a single device representing IPA, but the > picture is a little more complicated. > > The IPA hardware is actually something that sits *between* the > AP and the modem. It implements one form of communication > pathway (IP data), but there are others (including QMI, which > presents a network-like interface but it's actually implemented > via clever use of shared memory and interrupts). Can you clarify how QMI fits in here? Do you mean one has to talk to both IPA and QMI to use the modem, or are these two alternative implementations for the same basic purpose? > > That sounds about the same then, right. > > > > Are the control channels to IPA are actually also tunnelled over the > > rmnet protocol? And even if they are, perhaps they have a different > > hardware queue or so? That'd be the case for Intel - different hardware > > queue, same (or at least similar) protocol spoken for the DMA hardware > > itself, but different contents of the messages obviously. > > I want to be careful talking about "control" but for IPA it comes > from user space. For the purpose of getting initial code upstream, > all of that control functionality (which was IOCTL based) has been > removed, and a fixed configuration is assumed. My previous understanding was that from the hardware perspective there is only one control interface, which is for IPA. Part of this is abstracted to user space with ioctl commands to the IPA driver, and then one must set up rmnet to match these by configuring an rmnet device over netlink messages from user space, but rmnet does not have a control protocol with the hardware. The exception here is the flow control, which is handled using in-band XON/OFF messages from the modem to rmnet (and corresponding Acks the other way) that IPA just passes through. If we also need to talk to QMI, that would be something completely different though. Arnd
On Wed, 2019-06-26 at 08:39 -0500, Alex Elder wrote: > > However, to also reply to Alex: I don't know exactly how IPA works, but > > for the Intel modem at least you can't fundamentally have two drivers > > for different parts of the functionality, since it's just a single piece > > of hardware and you need to allocate hardware resources from a common > > pool etc. So you cannot split the driver into "Intel modem control > > channel driver" and "Intel modem data channel driver". In fact, it's > > just a single "struct device" on the PCIe bus that you can bind to, and > > only one driver can bind at a time. > > Interesting. So a single modem driver needs to implement > *all* of the features/functions? Like GPS or data log or > whatever, all needs to share the same struct device? > Or does what you're describing apply to a subset of the > modem's functionality? Or something else? Well, what is even the "implement the functions"? I mean, as kernel drivers we're really just in the business of providing communication channels to those functions. E.g. if you have a GNSS/GPS device, you might just have another TTY channel with NMEA data coming out, or something like that, right? But from a kernel POV yes, I don't see how you could create multiple function drivers for something behind the same PCIe device (unless it actually appeared as multiple virtual functions or such, like the bigger ethernet NICs, but it doesn't). But this points out to me that I was actually not quite accurate when I spoke about struct device before in the USB context with function drivers, but I have to do some research before I can correct myself correctly. > > So, IOW, I'm not sure I see how you'd split that up. I guess you could > > if you actually do something like the "rmnet" model, and I suppose > > you're free to do that for IPA if you like, but I tend to think that's > > actually a burden, not a win since you just get more complex code that > > needs to interact with more pieces. A single driver for a single > > hardware that knows about the few types of channels seems simpler to me. > > > > > - to answer Johannes question, my understanding is that the interface > > > between kernel and firmware/hardware for IPA has a single 'struct > > > device' that is used for both the data and the control channels, > > > rather than having a data channel and an independent control device, > > > so this falls into the same category as the Intel one (please correct > > > me on that) > > I don't think that's quite right, but it might be partially > right. There is a single device representing IPA, but the > picture is a little more complicated. > > The IPA hardware is actually something that sits *between* the > AP and the modem. It implements one form of communication > pathway (IP data), but there are others (including QMI, which > presents a network-like interface but it's actually implemented > via clever use of shared memory and interrupts). OK. Well, I guess this then might eventually become a bit of a hybrid - you eventually want one WWAN device to represent it all to userspace, but might actually have multiple devices with different drivers (from the kernel POV)? But this is more like all the USB devices work. I just have to figure out how to correctly tie them together - "struct device" may or may not be right? I need to check how this functions. I guess for something where you have DT (like you allude to elsewhere) you could just capture all this in DT by having some phandle link or something? > What we're talking about here is WWAN/modem management more > generally though. It *sounds* like the Intel modem is > more like a single device, which requires a single driver, > that seems to implement a bunch of distinct functions. Yes. > On this I'm not very knowledgeable but for Qualcomm there is > user space code that is in charge of overall management of > the modem. It implements what I think you're calling control > functions, negotiating with the modem to allow new data channels > to be created. Normally the IPA driver would provide information > to user space about available resources, but would only make a > communication pathway available when requested. Right. > > Are the control channels to IPA are actually also tunnelled over the > > rmnet protocol? And even if they are, perhaps they have a different > > hardware queue or so? That'd be the case for Intel - different hardware > > queue, same (or at least similar) protocol spoken for the DMA hardware > > itself, but different contents of the messages obviously. > > I want to be careful talking about "control" but for IPA it comes > from user space. For the purpose of getting initial code upstream, > all of that control functionality (which was IOCTL based) has been > removed, and a fixed configuration is assumed. But something that's ioctl based is just one form of "control" pathway. I was thinking of the AT or MBIM commands "control" channel. And then ioctls are likely something that terminates in the *driver*, right? I mean, the driver wouldn't get an ioctl and actually talk AT commands to the device ... But yes, the various control planes are confusing, we need to disentangle that. I tried over in the other email by layering where the control terminates. johannes
On Wed, 2019-06-26 at 15:58 +0200, Arnd Bergmann wrote: > > > The IPA hardware is actually something that sits *between* the > > AP and the modem. It implements one form of communication > > pathway (IP data), but there are others (including QMI, which > > presents a network-like interface but it's actually implemented > > via clever use of shared memory and interrupts). > > Can you clarify how QMI fits in here? Do you mean one has to > talk to both IPA and QMI to use the modem, or are these two > alternative implementations for the same basic purpose? I'm not going to comment on QMI specifically, because my understanding might well be wrong, and any response to your question will likely correct my understanding :-) (Thus, you should probably also ignore everything I ever said about QMI) > My previous understanding was that from the hardware perspective > there is only one control interface, which is for IPA. Part of this > is abstracted to user space with ioctl commands to the IPA driver, > and then one must set up rmnet to match these by configuring > an rmnet device over netlink messages from user space, but > rmnet does not have a control protocol with the hardware. Right so this is why I say it's confusing when we just talk about "control interface" or "path". I see multiple layers of control * hardware control, which you mention here. This might be things like "enable/disable aggregation on an rmnet channel" etc. I guess this type of thing would have been implemented with ioctls? Not the aggregation specifically, but things that affect how you set up the hardware. * modem control, which we conflate, but can be like AT commands or MBIM. From the kernel driver POV, this is actually just another channel it provides for userspace to talk to the modem. johannes
On Wed, 2019-06-26 at 08:36 -0500, Alex Elder wrote: > > We need to identify the existence of a WWAN device (which is I > guess--typically? always?--a modem). Perhaps that can be > discovered in some cases but I think it means a node described > by Device Tree. Yeah, perhaps that's something you could do. I'm not sure though. For one, for USB devices, obviously it isn't :-) And even for IPA you might want to support existing DTs I guess. > So you're saying you have a single Ethernet driver, and it can > drive an Ethernet device connected to a WWAN, or not connected > to a WWAN, without any changes. The only distinction is that > if the device is part of a WWAN it needs to register with the > WWAN framework. Is that right? That's what I'm thinking, and I believe (mostly from discussions with Dan) that this actually exists. > > > So maybe: > > > - Hardware probe detects a WWAN device > > > - The drivers that detect the WWAN device register it with the > > > WWAN core code. > > > - A control channel is instantiated at/before the time the WWAN > > > device is registered > > > - Something in user space should manage the bring-up of any > > > other things on the WWAN device thereafter > > > > But those things need to actually get connected first :-) > > What I meant is that the registering with the "WWAN core code" > is what does that "connecting." The WWAN code has the information > about what got registered. But as I said above, this WWAN device > needs to be identified, and I think (at least for IPA) that will > require something in Device Tree. That will "connect" them. > > Or I might be misunderstanding your point. No, I think we're mostly agreeing, just thinking about different scenarios. I think for IPA you don't really *need* anything in the DT though - as soon as the IPA driver is loaded you know for sure you actually have a modem there, and the IPA driver presumably loads based on some existing probing (didn't look at it now). Now, I don't know how the QMI channel to the modem is set up, so of course you'd want a way of identifying that the two channels (IPA and QMI) go to the same device and link them together in the WWAN framework. > > If userspace actually had the ability to create (data) channels, then it > > would have the ability to also remove them. Right now, this may or may > > not be supported by the drivers that act together to form the interfaces > > to a WWAN device. > > I think this (user space control) needs to be an option, but > it doesn't have to be the only way. Agree. johannes
On Wed, 2019-06-26 at 08:40 -0500, Alex Elder wrote: > > I think here we need to be more careful. I don't know how you want to > > call it, but we actually have multiple levels of control here. > > I completely agree with you. From what I understand there exists > a control channel (or even more than one?) that serves a very > specific purpose in modem management. The main reason I mention > the WWAN control function is that someone (maybe you) indicated > that a control channel automatically gets created. It may or may not, right. I just bought a cheap used USB modem, and it just comes with two USB TTY channels, so I guess for data it does PPP on top of that. But those channels are created automatically once you connect the device to the system. OTOH, for something like the Intel modem, we might well decide not to create *any* channels on driver load, since you have the option of using AT commands or MBIM (but I believe both at the same time wouldn't really make sense, if even allowed). > > This ... depends a bit on how you exactly define a physical channel > > here. Is that, to you, the PCIe/USB link? In that case, yes, obviously > > you have only one physical channel for each WWAN unit. > > I think that was what I was trying to capture. There exists > one or more "physical" communication paths between the AP > and WWAN unit/modem. And while one path *could* carry just > one type of traffic, it could also carry multiple logical > channels of traffic by multiplexing. Right. (What I wasn't aware is that QMI is actually a different physical path. I thought it was just a protocol multiplexed on top of the same IPA physical path). > I don't think I have any argument with this. I'm going to try to > put together something that goes beyond what I wrote in this message, > to try to capture what I think we agree on in a sort of loose design > document. Awesome, thanks a lot! johannes
This series presents the driver for the Qualcomm IP Accelerator (IPA). This is version 2 of the series. This version has addressed almost all of the feedback received in the first version: https://lore.kernel.org/lkml/20190512012508.10608-1-elder@linaro.org/ More detail is included in the individual patches, but here is a high-level summary of what's changed since then: - Two spinlocks have been removed. - The code for enabling and disabling endpoint interrupts has been simplified considerably, and the spinlock is no longer required - A spinlock used when updating ring buffer pointers is no longer needed. Integers indexing the ring are used instead (and they don't even have to be atomic). - One spinlock remains to protect list updates, but it is always acquired using spin_lock_bh() (no more irqsave). - Information about the queueing and completion of messages is now supplied to the network stack in batches rather than one at a time. - I/O completion handling has been simplified, with the IRQ handler now consisting mainly of disabling the interrupt and calling napi_schedule(). - Some comments have been updated and improved througout. What follows is the introduction supplied with v1 of the series. ----- The IPA is a component present in some Qualcomm SoCs that allows network functions such as aggregation, filtering, routing, and NAT to be performed without active involvement of the main application processor (AP). Initially, these advanced features are disabled; the IPA driver simply provides a network interface that makes the modem's LTE network available to the AP. In addition, only support for the IPA found in the Qualcomm SDM845 SoC is provided. This code is derived from a driver developed internally by Qualcomm. A version of the original source can be seen here: https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree in the "drivers/platform/msm/ipa" directory. Many were involved in developing this, but the following individuals deserve explicit acknowledgement for their substantial contributions: Abhishek Choubey Ady Abraham Chaitanya Pratapa David Arinzon Ghanim Fodi Gidon Studinski Ravi Gummadidala Shihuan Liu Skylar Chang A version of this code was posted in November 2018 as an RFC. https://lore.kernel.org/lkml/20181107003250.5832-1-elder@linaro.org/ All feedback received was addressed. The code has undergone considerable further rework since that time, and most of the "future work" described then has now been completed. This code is available in buildable form here, based on kernel v5.2-rc1: remote: ssh://git@git.linaro.org/people/alex.elder/linux.git branch: ipa-v2_kernel-v5.2-rc2 75adf2ac1266 arm64: defconfig: enable build of IPA code The branch depends on a commit now found in in net-next. It has been cherry-picked, and (in this branch) has this commit ID: 13c627b5a078 net: qualcomm: rmnet: Move common struct definitions to include by -Alex Alex Elder (17): bitfield.h: add FIELD_MAX() and field_max() dt-bindings: soc: qcom: add IPA bindings soc: qcom: ipa: main code soc: qcom: ipa: configuration data soc: qcom: ipa: clocking, interrupts, and memory soc: qcom: ipa: GSI headers soc: qcom: ipa: the generic software interface soc: qcom: ipa: GSI transactions soc: qcom: ipa: IPA interface to GSI soc: qcom: ipa: IPA endpoints soc: qcom: ipa: immediate commands soc: qcom: ipa: IPA network device and microcontroller soc: qcom: ipa: AP/modem communications soc: qcom: ipa: support build of IPA code MAINTAINERS: add entry for the Qualcomm IPA driver arm64: dts: sdm845: add IPA information arm64: defconfig: enable build of IPA code .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++ MAINTAINERS | 6 + arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 + arch/arm64/configs/defconfig | 1 + drivers/net/Kconfig | 2 + drivers/net/Makefile | 1 + drivers/net/ipa/Kconfig | 16 + drivers/net/ipa/Makefile | 7 + drivers/net/ipa/gsi.c | 1635 +++++++++++++++++ drivers/net/ipa/gsi.h | 246 +++ drivers/net/ipa/gsi_private.h | 148 ++ drivers/net/ipa/gsi_reg.h | 376 ++++ drivers/net/ipa/gsi_trans.c | 624 +++++++ drivers/net/ipa/gsi_trans.h | 116 ++ drivers/net/ipa/ipa.h | 131 ++ drivers/net/ipa/ipa_clock.c | 297 +++ drivers/net/ipa/ipa_clock.h | 52 + drivers/net/ipa/ipa_cmd.c | 377 ++++ drivers/net/ipa/ipa_cmd.h | 116 ++ drivers/net/ipa/ipa_data-sdm845.c | 245 +++ drivers/net/ipa/ipa_data.h | 267 +++ drivers/net/ipa/ipa_endpoint.c | 1283 +++++++++++++ drivers/net/ipa/ipa_endpoint.h | 97 + drivers/net/ipa/ipa_gsi.c | 48 + drivers/net/ipa/ipa_gsi.h | 49 + drivers/net/ipa/ipa_interrupt.c | 279 +++ drivers/net/ipa/ipa_interrupt.h | 53 + drivers/net/ipa/ipa_main.c | 921 ++++++++++ drivers/net/ipa/ipa_mem.c | 234 +++ drivers/net/ipa/ipa_mem.h | 83 + drivers/net/ipa/ipa_netdev.c | 251 +++ drivers/net/ipa/ipa_netdev.h | 24 + drivers/net/ipa/ipa_qmi.c | 402 ++++ drivers/net/ipa/ipa_qmi.h | 35 + drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++ drivers/net/ipa/ipa_qmi_msg.h | 238 +++ drivers/net/ipa/ipa_reg.h | 279 +++ drivers/net/ipa/ipa_smp2p.c | 304 +++ drivers/net/ipa/ipa_smp2p.h | 47 + drivers/net/ipa/ipa_uc.c | 208 +++ drivers/net/ipa/ipa_uc.h | 32 + include/linux/bitfield.h | 14 + 42 files changed, 10358 insertions(+) create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml create mode 100644 drivers/net/ipa/Kconfig create mode 100644 drivers/net/ipa/Makefile create mode 100644 drivers/net/ipa/gsi.c create mode 100644 drivers/net/ipa/gsi.h create mode 100644 drivers/net/ipa/gsi_private.h create mode 100644 drivers/net/ipa/gsi_reg.h create mode 100644 drivers/net/ipa/gsi_trans.c create mode 100644 drivers/net/ipa/gsi_trans.h create mode 100644 drivers/net/ipa/ipa.h create mode 100644 drivers/net/ipa/ipa_clock.c create mode 100644 drivers/net/ipa/ipa_clock.h create mode 100644 drivers/net/ipa/ipa_cmd.c create mode 100644 drivers/net/ipa/ipa_cmd.h create mode 100644 drivers/net/ipa/ipa_data-sdm845.c create mode 100644 drivers/net/ipa/ipa_data.h create mode 100644 drivers/net/ipa/ipa_endpoint.c create mode 100644 drivers/net/ipa/ipa_endpoint.h create mode 100644 drivers/net/ipa/ipa_gsi.c create mode 100644 drivers/net/ipa/ipa_gsi.h create mode 100644 drivers/net/ipa/ipa_interrupt.c create mode 100644 drivers/net/ipa/ipa_interrupt.h create mode 100644 drivers/net/ipa/ipa_main.c create mode 100644 drivers/net/ipa/ipa_mem.c create mode 100644 drivers/net/ipa/ipa_mem.h create mode 100644 drivers/net/ipa/ipa_netdev.c create mode 100644 drivers/net/ipa/ipa_netdev.h create mode 100644 drivers/net/ipa/ipa_qmi.c create mode 100644 drivers/net/ipa/ipa_qmi.h create mode 100644 drivers/net/ipa/ipa_qmi_msg.c create mode 100644 drivers/net/ipa/ipa_qmi_msg.h create mode 100644 drivers/net/ipa/ipa_reg.h create mode 100644 drivers/net/ipa/ipa_smp2p.c create mode 100644 drivers/net/ipa/ipa_smp2p.h create mode 100644 drivers/net/ipa/ipa_uc.c create mode 100644 drivers/net/ipa/ipa_uc.h