[v7,3/6] dt-bindings: mailbox: imx-mu: add generic MU channel support

Commit Message

Oleksij Rempel July 26, 2018, 6:53 a.m. UTC

Each MU has four pairs of rx/tx data register with four rx/tx interrupts
which can also be used as a separate channel.

Reviewed-by: Rob Herring <[e-mail]>
Reviewed-by: Dong Aisheng <[e-mail]>
Signed-off-by: Oleksij Rempel <[e-mail]>
---
 Documentation/devicetree/bindings/mailbox/fsl,mu.txt | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

Comments

Jassi Brar July 26, 2018, 9:49 a.m. UTC | #1

On Thu, Jul 26, 2018 at 12:23 PM, Oleksij Rempel
<[e-mail]> wrote:

> Each MU has four pairs of rx/tx data register with four rx/tx interrupts
> which can also be used as a separate channel.
>
So the hardware actually supports 4 channels.

> -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> +- #mbox-cells:  Must be:
> +               0 - for single channel mode. i.MX8* SCU protocol specific.
> +               1 - for multichannel (generic) mode.
> +
No, please.
DT bindings should reflect the real hardware, and not the software
mode we want the driver to work in.
Please define mbox-cells=1  and have the i.MX8* platform always ask
for channel-0.

Oleksij Rempel July 26, 2018, 10:57 a.m. UTC | #2

On 26.07.2018 11:49, Jassi Brar wrote:
> On Thu, Jul 26, 2018 at 12:23 PM, Oleksij Rempel
> <[e-mail]> wrote:
> 
>> Each MU has four pairs of rx/tx data register with four rx/tx interrupts
>> which can also be used as a separate channel.
>>
> So the hardware actually supports 4 channels.
> 
>> -- #mbox-cells:  Must be 0. Number of cells in a mailbox
>> +- #mbox-cells:  Must be:
>> +               0 - for single channel mode. i.MX8* SCU protocol specific.
>> +               1 - for multichannel (generic) mode.
>> +
> No, please.
> DT bindings should reflect the real hardware, and not the software
> mode we want the driver to work in.
> Please define mbox-cells=1  and have the i.MX8* platform always ask
> for channel-0.

We spend already some week to publicly talk about this..
The problem is, imx8 has multiple MUs, only some of them communicate
directly with SCU. SCU has some well defined protocol and
MU should be used and configured in some SCU specific way. From NXP
perspective, SCU firmware should not be replaceable and from Linux side
seen as part of the HW.

@Dong, is it correct description?

Jassi Brar July 26, 2018, 11:28 a.m. UTC | #3

On Thu, Jul 26, 2018 at 4:27 PM, Oleksij Rempel <[e-mail]> wrote:
>
>
> On 26.07.2018 11:49, Jassi Brar wrote:
>> On Thu, Jul 26, 2018 at 12:23 PM, Oleksij Rempel
>> <[e-mail]> wrote:
>>
>>> Each MU has four pairs of rx/tx data register with four rx/tx interrupts
>>> which can also be used as a separate channel.
>>>
>> So the hardware actually supports 4 channels.
>>
>>> -- #mbox-cells:  Must be 0. Number of cells in a mailbox
>>> +- #mbox-cells:  Must be:
>>> +               0 - for single channel mode. i.MX8* SCU protocol specific.
>>> +               1 - for multichannel (generic) mode.
>>> +
>> No, please.
>> DT bindings should reflect the real hardware, and not the software
>> mode we want the driver to work in.
>> Please define mbox-cells=1  and have the i.MX8* platform always ask
>> for channel-0.
>
> We spend already some week to publicly talk about this..
>
You should have kept me in the loop.
I know, I am not that important, but for the sake of the protocol you
should have CC'ed the mailbox maintainer :)

> The problem is, imx8 has multiple MUs
>
You mean multiple instances of the MU. Some have 4 channels, some 1.
Is that correct?

>, only some of them communicate
> directly with SCU.
>
OK

> SCU has some well defined protocol and
> MU should be used and configured in some SCU specific way. From NXP
> perspective, SCU firmware should not be replaceable and from Linux side
> seen as part of the HW.
>
Sure. But the behaviour (protocol) should be implemented in the client
driver, not the controller driver.

Aisheng Dong July 26, 2018, 11:51 a.m. UTC | #4

Hi Jassi,

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Thursday, July 26, 2018 5:50 PM
> To: Oleksij Rempel <[e-mail]>
> Cc: Shawn Guo <[e-mail]>; Fabio Estevam
> <[e-mail]>; Rob Herring <[e-mail]>; Mark
> Rutland <[e-mail]>; A.s. Dong <[e-mail]>;
> Vladimir Zapolskiy <[e-mail]>; , Sascha Hauer
> <[e-mail]>; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Thu, Jul 26, 2018 at 12:23 PM, Oleksij Rempel <[e-mail]>
> wrote:
> 
> > Each MU has four pairs of rx/tx data register with four rx/tx
> > interrupts which can also be used as a separate channel.
> >
> So the hardware actually supports 4 channels.
> 
> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> > +- #mbox-cells:  Must be:
> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
> > +               1 - for multichannel (generic) mode.
> > +
> No, please.
> DT bindings should reflect the real hardware, and not the software mode we
> want the driver to work in.
> Please define mbox-cells=1  and have the i.MX8* platform always ask for
> channel-0.

The reality is that MU hardware does not define channels in reference manual.
However, it does have four separate data tx/rx register which can be used
as 'virtual' channels which is supported by this current driver.

See below HW description from the reference manual:
For messaging, the MU has four, 32-bit write-only transmit registers and four, 32-bit
read-only receive registers on the Processor B and Processor A-sides. These registers are
used for sending messages to each other.

Passing short messages
Transmit register(s) can be used to pass short messages
from one to four words in length. For example, when a four-word message is desired,
only one of the registers needs to have its corresponding interrupt enable bit set at the
receiver side; the message’s first three words are written to the registers whose
interrupt is masked, and the fourth word is written to the other register (which
triggers an interrupt at the receiver side).

The reference manual is at here: (Chapter 42 Messaging Unit (MU)
https://www.nxp.com/docs/en/reference-manual/IMX6ULRM.pdf

And SCU firmware does use MU in above way specified by reference manual.
Even from HW point of view, it's still one channel only mailbox.
That's why we change the mbox-cells to 0 accordingly to avoid confusing
the multi channels users supported in this driver.

As per Sasha's request, we're going to support both protocol (SCU and generic M4)
in one mailbox driver, that means we need deal with this two case property.

If we use mbox-cells 1 case for SCU, then the using would be like:
mboxes = <&mailbox 0> which is mixed with the normal multi channel
users. And we also must state clearly in binding doc that for mbox-cells
 1 case for SCU, the cell value must be 0 which is also confusing with
Multi channel users.

How do you suggest for such case?

Regards
Dong Aisheng

Jassi Brar July 26, 2018, 12:26 p.m. UTC | #5

On Thu, Jul 26, 2018 at 5:21 PM, A.s. Dong <[e-mail]> wrote:
> Hi Jassi,
>
>> -----Original Message-----
>> From: Jassi Brar [mailto:[e-mail]]
>> Sent: Thursday, July 26, 2018 5:50 PM
>> To: Oleksij Rempel <[e-mail]>
>> Cc: Shawn Guo <[e-mail]>; Fabio Estevam
>> <[e-mail]>; Rob Herring <[e-mail]>; Mark
>> Rutland <[e-mail]>; A.s. Dong <[e-mail]>;
>> Vladimir Zapolskiy <[e-mail]>; , Sascha Hauer
>> <[e-mail]>; , [e-mail], linux-
>> [e-mail], srv_heupstream <linux-arm-
>> [e-mail]>; Devicetree List <[e-mail]>;
>> dl-linux-imx <[e-mail]>
>> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
>> channel support
>>
>> On Thu, Jul 26, 2018 at 12:23 PM, Oleksij Rempel <[e-mail]>
>> wrote:
>>
>> > Each MU has four pairs of rx/tx data register with four rx/tx
>> > interrupts which can also be used as a separate channel.
>> >
>> So the hardware actually supports 4 channels.
>>
>> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
>> > +- #mbox-cells:  Must be:
>> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
>> > +               1 - for multichannel (generic) mode.
>> > +
>> No, please.
>> DT bindings should reflect the real hardware, and not the software mode we
>> want the driver to work in.
>> Please define mbox-cells=1  and have the i.MX8* platform always ask for
>> channel-0.
>
> The reality is that MU hardware does not define channels in reference manual.
> However, it does have four separate data tx/rx register which can be used
> as 'virtual' channels which is supported by this current driver.
>
> See below HW description from the reference manual:
> For messaging, the MU has four, 32-bit write-only transmit registers and four, 32-bit
> read-only receive registers on the Processor B and Processor A-sides. These registers are
> used for sending messages to each other.
>
> Passing short messages
> Transmit register(s) can be used to pass short messages
> from one to four words in length. For example, when a four-word message is desired,
> only one of the registers needs to have its corresponding interrupt enable bit set at the
> receiver side; the message’s first three words are written to the registers whose
> interrupt is masked, and the fourth word is written to the other register (which
> triggers an interrupt at the receiver side).
>
> The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> https://www.nxp.com/docs/en/reference-manual/IMX6ULRM.pdf
>
Thanks. That makes life much easier.

> And SCU firmware does use MU in above way specified by reference manual.
> Even from HW point of view, it's still one channel only mailbox.
> That's why we change the mbox-cells to 0 accordingly to avoid confusing
> the multi channels users supported in this driver.
>
> As per Sasha's request, we're going to support both protocol (SCU and generic M4)
> in one mailbox driver, that means we need deal with this two case property.
>
> If we use mbox-cells 1 case for SCU, then the using would be like:
> mboxes = <&mailbox 0> which is mixed with the normal multi channel
> users. And we also must state clearly in binding doc that for mbox-cells
>  1 case for SCU, the cell value must be 0 which is also confusing with
> Multi channel users.
>
> How do you suggest for such case?
>
Let me read the spec and try to see/shine the light :)

Cheers!

Aisheng Dong July 26, 2018, 1:34 p.m. UTC | #6

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Thursday, July 26, 2018 8:27 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; Shawn Guo
> <[e-mail]>; Fabio Estevam <[e-mail]>; Rob
> Herring <[e-mail]>; Mark Rutland <[e-mail]>;
> Vladimir Zapolskiy <[e-mail]>; , Sascha Hauer
> <[e-mail]>; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support

[...]

> > And SCU firmware does use MU in above way specified by reference
> manual.
> > Even from HW point of view, it's still one channel only mailbox.
> > That's why we change the mbox-cells to 0 accordingly to avoid
> > confusing the multi channels users supported in this driver.
> >
> > As per Sasha's request, we're going to support both protocol (SCU and
> > generic M4) in one mailbox driver, that means we need deal with this two
> case property.
> >
> > If we use mbox-cells 1 case for SCU, then the using would be like:
> > mboxes = <&mailbox 0> which is mixed with the normal multi channel
> > users. And we also must state clearly in binding doc that for
> > mbox-cells
> >  1 case for SCU, the cell value must be 0 which is also confusing with
> > Multi channel users.
> >
> > How do you suggest for such case?
> >
> Let me read the spec and try to see/shine the light :)

Great, many thanks!

Regards
Dong Aisheng

> 
> Cheers!

Jassi Brar July 26, 2018, 1:49 p.m. UTC | #7

On Thu, Jul 26, 2018 at 5:21 PM, A.s. Dong <[e-mail]> wrote:
>
> The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> https://www.nxp.com/docs/en/reference-manual/IMX6ULRM.pdf
>
You seem to have shared a wrong link. It has
       Chapter 42  Synchronous Audio Interface (SAI)

and finding "Messaging Unit" shows no result.

Aisheng Dong July 26, 2018, 2:13 p.m. UTC | #8

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Thursday, July 26, 2018 9:50 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; Shawn Guo
> <[e-mail]>; Fabio Estevam <[e-mail]>; Rob
> Herring <[e-mail]>; Mark Rutland <[e-mail]>;
> Vladimir Zapolskiy <[e-mail]>; , Sascha Hauer
> <[e-mail]>; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Thu, Jul 26, 2018 at 5:21 PM, A.s. Dong <[e-mail]> wrote:
> >
> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> ww.nxp.com%2Fdocs%2Fen%2Freference-
> manual%2FIMX6ULRM.pdf&amp;data=02%7C01%7Caisheng.dong%40nxp.co
> m%7Ca1e872436611497795bb08d5f2fe9feb%7C686ea1d3bc2b4c6fa92cd99c5c
> 301635%7C0%7C0%7C636682097774333065&amp;sdata=Zqf9kgJrutPtQE8OP
> %2BDu1%2BvULXWSQcgHbszwIWkTNS0%3D&amp;reserved=0
> >
> You seem to have shared a wrong link. It has
>        Chapter 42  Synchronous Audio Interface (SAI)
> 
> and finding "Messaging Unit" shows no result.

Sorry, it should be:
https://www.nxp.com/docs/en/reference-manual/IMX6SXRM.pdf

Regards
Dong Aisheng

Jassi Brar July 26, 2018, 3:44 p.m. UTC | #9

On Thu, Jul 26, 2018 at 5:21 PM, A.s. Dong <[e-mail]> wrote:

>>
>> > Each MU has four pairs of rx/tx data register with four rx/tx
>> > interrupts which can also be used as a separate channel.
>> >
>> So the hardware actually supports 4 channels.
>>
>> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
>> > +- #mbox-cells:  Must be:
>> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
>> > +               1 - for multichannel (generic) mode.
>> > +
>> No, please.
>> DT bindings should reflect the real hardware, and not the software mode we
>> want the driver to work in.
>> Please define mbox-cells=1  and have the i.MX8* platform always ask for
>> channel-0.
>
> The reality is that MU hardware does not define channels in reference manual.
> However, it does have four separate data tx/rx register which can be used
> as 'virtual' channels which is supported by this current driver.
>
> See below HW description from the reference manual:
> For messaging, the MU has four, 32-bit write-only transmit registers and four, 32-bit
> read-only receive registers on the Processor B and Processor A-sides. These registers are
> used for sending messages to each other.
>
For a while please forget the protocol(user) level usage, and consider
only what your h/w is.

MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
(MU also has 4 "doorbell" type channels that it calls GP, but those
are not managed here, so lets not worry atm).

By definition, a mailbox channel is simply a signal, optionally with
data attached. So, MU has 4 TX and 4 RX channels.

The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
channels and set each tx/rx operation to trigger the corresponding
interrupt. This is not my whim, this is how the controller is!

The SCU is poorly implemented as it ignores 3 irqs and club all 4
register together (there are many other cons of this approach but lets
not get into that).
Personally, I would push-back on such a bad design. But if you claim
you have no choice but to support SCU as such, the work around could
be simpler than defining a new "scu mode" altogether.

#mbox-cells:  Must be 3.
                      First cell is 1 for TX and 0 for RX channel
                      Second cell is index of the channel [0,1,2 or 3]
                      Third cell is 1 if the channel triggers an IRQ,
0 if not. That is ACR.RIE/TIE bits are set or not.

Normal clients would always request a channel with irqs enabled.
The SCU client would request all 8 channels -- TX/RX[0,1,2] with irqs
disabled, TX/RX[3] with irqs enabled. And SCU will read/write 4 word
data over 4 rx/tx channels, instead of the virtually concocted one.


> short messages
> Transmit register(s) can be used to pass short messages
> from one to four words in length. For example, when a four-word message is desired,
> only one of the registers needs to have its corresponding interrupt enable bit set at the
> receiver side; the message’s first three words are written to the registers whose
> interrupt is masked, and the fourth word is written to the other register (which
> triggers an interrupt at the receiver side).
>
> The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> https://www.nxp.com/docs/en/reference-manual/IMX6ULRM.pdf
>
> And SCU firmware does use MU in above way specified by reference manual.
> Even from HW point of view, it's still one channel only mailbox.
>
Please realise that any manual is written by a mere mortal afterall.
How the controller works is set in stone, but how the controller can
be used ... is just someones suggestion.

The approach I suggest above, conforms to the api and prevents a
provider dancing to the tunes of a user.

Aisheng Dong July 27, 2018, 4:02 a.m. UTC | #10

Hi Jassi,

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Thursday, July 26, 2018 11:44 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; Shawn Guo
> <[e-mail]>; Fabio Estevam <[e-mail]>; Rob
> Herring <[e-mail]>; Mark Rutland <[e-mail]>;
> Vladimir Zapolskiy <[e-mail]>; , Sascha Hauer
> <[e-mail]>; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Thu, Jul 26, 2018 at 5:21 PM, A.s. Dong <[e-mail]> wrote:
> 
> >>
> >> > Each MU has four pairs of rx/tx data register with four rx/tx
> >> > interrupts which can also be used as a separate channel.
> >> >
> >> So the hardware actually supports 4 channels.
> >>
> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> >> > +- #mbox-cells:  Must be:
> >> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
> >> > +               1 - for multichannel (generic) mode.
> >> > +
> >> No, please.
> >> DT bindings should reflect the real hardware, and not the software
> >> mode we want the driver to work in.
> >> Please define mbox-cells=1  and have the i.MX8* platform always ask
> >> for channel-0.
> >
> > The reality is that MU hardware does not define channels in reference
> manual.
> > However, it does have four separate data tx/rx register which can be
> > used as 'virtual' channels which is supported by this current driver.
> >
> > See below HW description from the reference manual:
> > For messaging, the MU has four, 32-bit write-only transmit registers
> > and four, 32-bit read-only receive registers on the Processor B and
> > Processor A-sides. These registers are used for sending messages to each
> other.
> >
> For a while please forget the protocol(user) level usage, and consider only
> what your h/w is.
> 
> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
> (MU also has 4 "doorbell" type channels that it calls GP, but those are not
> managed here, so lets not worry atm).
> 
> By definition, a mailbox channel is simply a signal, optionally with data
> attached. So, MU has 4 TX and 4 RX channels.
> 
> The MU driver should populate 8 unidirectional (4 Tx and 4 RX) channels and
> set each tx/rx operation to trigger the corresponding interrupt. This is not my
> whim, this is how the controller is!
> 

This looks like reasonable to me, theoretically. 
Just not sure whether it's necessary to support it because we probably will never use
like that in reality, then it might become meaningless complicity introduced
and error prone.

And AFAIK ARM MHU is doing the same way as MU which looks like also unidirectional channel.
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0515b/CHDGBGIF.html
drivers/mailbox/arm_mhu.c

Sascha & Oleksij,
Any idea about this?

> The SCU is poorly implemented as it ignores 3 irqs and club all 4 register
> together (there are many other cons of this approach but lets not get into
> that).
> Personally, I would push-back on such a bad design. But if you claim you have
> no choice but to support SCU as such, the work around could be simpler than
> defining a new "scu mode" altogether.
> 

This is one of the recommended ways designed in HW reference manual and it
allows to send frame information up to 4 words without using shared memory.
SCU just follows it, so it's hard to believe it's a bad design.

> #mbox-cells:  Must be 3.
>                       First cell is 1 for TX and 0 for RX channel
>                       Second cell is index of the channel [0,1,2 or 3]
>                       Third cell is 1 if the channel triggers an IRQ,
> 0 if not. That is ACR.RIE/TIE bits are set or not.
> 
> Normal clients would always request a channel with irqs enabled.
> The SCU client would request all 8 channels -- TX/RX[0,1,2] with irqs disabled,
> TX/RX[3] with irqs enabled. And SCU will read/write 4 word data over 4 rx/tx
> channels, instead of the virtually concocted one.
> 

It may work If SCU protocol data length is fixed to 4 words. However, it's length
is flexible for different SVC service. That means this binding won't work for SCU.
And it will introduce much complexities during the implementation.

Instead, we're using polling mode for both TX/RX and the data size is stored in
the msg header and sending msgs using all 4 data tx/rx registers as a channel fifo.

> 
> > short messages
> > Transmit register(s) can be used to pass short messages from one to
> > four words in length. For example, when a four-word message is
> > desired, only one of the registers needs to have its corresponding
> > interrupt enable bit set at the receiver side; the message’s first
> > three words are written to the registers whose interrupt is masked,
> > and the fourth word is written to the other register (which triggers an
> interrupt at the receiver side).
> >
> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> ww
> > .nxp.com%2Fdocs%2Fen%2Freference-
> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
> >
> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
> 86ea1
> >
> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
> a=54rE
> >
> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
> >
> > And SCU firmware does use MU in above way specified by reference
> manual.
> > Even from HW point of view, it's still one channel only mailbox.
> >
> Please realise that any manual is written by a mere mortal afterall.
> How the controller works is set in stone, but how the controller can be
> used ... is just someones suggestion.
> 
> The approach I suggest above, conforms to the api and prevents a provider
> dancing to the tunes of a user.

First of all, really appreciate for your suggestions. 
It may be hard to find a common binding with the same mbox-cells.
It looks like we just need a property is distinguish how the Mailbox is used
In one channel or multi channel mode.

Directly checking mbox-cells seems the most easy way and it is already Acked
by Rob. Can't this way be Okay to you?

Regards
Dong Aisheng

Jassi Brar July 27, 2018, 4:55 a.m. UTC | #11

On Fri, Jul 27, 2018 at 9:32 AM, A.s. Dong <[e-mail]> wrote:
>>
>> >>
>> >> > Each MU has four pairs of rx/tx data register with four rx/tx
>> >> > interrupts which can also be used as a separate channel.
>> >> >
>> >> So the hardware actually supports 4 channels.
>> >>
>> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
>> >> > +- #mbox-cells:  Must be:
>> >> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
>> >> > +               1 - for multichannel (generic) mode.
>> >> > +
>> >> No, please.
>> >> DT bindings should reflect the real hardware, and not the software
>> >> mode we want the driver to work in.
>> >> Please define mbox-cells=1  and have the i.MX8* platform always ask
>> >> for channel-0.
>> >
>> > The reality is that MU hardware does not define channels in reference
>> manual.
>> > However, it does have four separate data tx/rx register which can be
>> > used as 'virtual' channels which is supported by this current driver.
>> >
>> > See below HW description from the reference manual:
>> > For messaging, the MU has four, 32-bit write-only transmit registers
>> > and four, 32-bit read-only receive registers on the Processor B and
>> > Processor A-sides. These registers are used for sending messages to each
>> other.
>> >
>> For a while please forget the protocol(user) level usage, and consider only
>> what your h/w is.
>>
>> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
>> (MU also has 4 "doorbell" type channels that it calls GP, but those are not
>> managed here, so lets not worry atm).
>>
>> By definition, a mailbox channel is simply a signal, optionally with data
>> attached. So, MU has 4 TX and 4 RX channels.
>>
>> The MU driver should populate 8 unidirectional (4 Tx and 4 RX) channels and
>> set each tx/rx operation to trigger the corresponding interrupt. This is not my
>> whim, this is how the controller is!
>>
>
> This looks like reasonable to me, theoretically.
> Just not sure whether it's necessary to support it because we probably will never use
> like that in reality, then it might become meaningless complicity introduced
> and error prone.
>
It _is_ necessary to write controller driver independent of client drivers.


> And AFAIK ARM MHU is doing the same way as MU which looks like also unidirectional channel.
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0515b/CHDGBGIF.html
> drivers/mailbox/arm_mhu.c
>
MHU driver is doing exactly what the data sheet says. I know because I
wrote the driver.


>> The SCU is poorly implemented as it ignores 3 irqs and club all 4 register
>> together (there are many other cons of this approach but lets not get into
>> that).
>> Personally, I would push-back on such a bad design. But if you claim you have
>> no choice but to support SCU as such, the work around could be simpler than
>> defining a new "scu mode" altogether.
>>
>
> This is one of the recommended ways designed in HW reference manual and it
> allows to send frame information up to 4 words without using shared memory.
> SCU just follows it, so it's hard to believe it's a bad design.
>
>> #mbox-cells:  Must be 3.
>>                       First cell is 1 for TX and 0 for RX channel
>>                       Second cell is index of the channel [0,1,2 or 3]
>>                       Third cell is 1 if the channel triggers an IRQ,
>> 0 if not. That is ACR.RIE/TIE bits are set or not.
>>
>> Normal clients would always request a channel with irqs enabled.
>> The SCU client would request all 8 channels -- TX/RX[0,1,2] with irqs disabled,
>> TX/RX[3] with irqs enabled. And SCU will read/write 4 word data over 4 rx/tx
>> channels, instead of the virtually concocted one.
>>
>
> It may work If SCU protocol data length is fixed to 4 words. However, it's length
> is flexible for different SVC service. That means this binding won't work for SCU.
> And it will introduce much complexities during the implementation.
>
> Instead, we're using polling mode for both TX/RX and the data size is stored in
> the msg header and sending msgs using all 4 data tx/rx registers as a channel fifo.
>
You first give me the "Passing short messages" usecase (quite a bad
one) and ask how it can work. When I give you a solution, you
effectively say "well, my usecase isn't even that". I feel I just
wasted my time.


>>
>> > short messages
>> > Transmit register(s) can be used to pass short messages from one to
>> > four words in length. For example, when a four-word message is
>> > desired, only one of the registers needs to have its corresponding
>> > interrupt enable bit set at the receiver side; the message’s first
>> > three words are written to the registers whose interrupt is masked,
>> > and the fourth word is written to the other register (which triggers an
>> interrupt at the receiver side).
>> >
>> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
>> >
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>> ww
>> > .nxp.com%2Fdocs%2Fen%2Freference-
>> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
>> >
>> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
>> 86ea1
>> >
>> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
>> a=54rE
>> >
>> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
>> >
>> > And SCU firmware does use MU in above way specified by reference
>> manual.
>> > Even from HW point of view, it's still one channel only mailbox.
>> >
>> Please realise that any manual is written by a mere mortal afterall.
>> How the controller works is set in stone, but how the controller can be
>> used ... is just someones suggestion.
>>
>> The approach I suggest above, conforms to the api and prevents a provider
>> dancing to the tunes of a user.
>
> First of all, really appreciate for your suggestions.
> It may be hard to find a common binding with the same mbox-cells.
> It looks like we just need a property is distinguish how the Mailbox is used
> In one channel or multi channel mode.
>
I get the idea you were ready to see the code merged in the coming
window and be done with it. And now it feels lazy to change the code.
I am sorry, but I can't allow controller drivers "emulating" some mode
for a client driver. That is moving a part of client code into the
controller driver.


> Directly checking mbox-cells seems the most easy way and it is already Acked
> by Rob. Can't this way be Okay to you?
>
Rob is indeed far better than I am. But he also has to look into 100
other subsystems, whereas I am only looking into mailbox. I have time
to dig into your datasheets. I believe Rob will point out anything
wrong I suggest.

BTW, the submitted driver doesn't even support the SCU usecase. Why
the bindings?

Aisheng Dong July 27, 2018, 6 a.m. UTC | #12

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Friday, July 27, 2018 12:56 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; , [e-mail],
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Fri, Jul 27, 2018 at 9:32 AM, A.s. Dong <[e-mail]> wrote:
> >>
> >> >>
> >> >> > Each MU has four pairs of rx/tx data register with four rx/tx
> >> >> > interrupts which can also be used as a separate channel.
> >> >> >
> >> >> So the hardware actually supports 4 channels.
> >> >>
> >> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> >> >> > +- #mbox-cells:  Must be:
> >> >> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
> >> >> > +               1 - for multichannel (generic) mode.
> >> >> > +
> >> >> No, please.
> >> >> DT bindings should reflect the real hardware, and not the software
> >> >> mode we want the driver to work in.
> >> >> Please define mbox-cells=1  and have the i.MX8* platform always
> >> >> ask for channel-0.
> >> >
> >> > The reality is that MU hardware does not define channels in
> >> > reference
> >> manual.
> >> > However, it does have four separate data tx/rx register which can
> >> > be used as 'virtual' channels which is supported by this current driver.
> >> >
> >> > See below HW description from the reference manual:
> >> > For messaging, the MU has four, 32-bit write-only transmit
> >> > registers and four, 32-bit read-only receive registers on the
> >> > Processor B and Processor A-sides. These registers are used for
> >> > sending messages to each
> >> other.
> >> >
> >> For a while please forget the protocol(user) level usage, and
> >> consider only what your h/w is.
> >>
> >> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
> >> (MU also has 4 "doorbell" type channels that it calls GP, but those
> >> are not managed here, so lets not worry atm).
> >>
> >> By definition, a mailbox channel is simply a signal, optionally with
> >> data attached. So, MU has 4 TX and 4 RX channels.
> >>
> >> The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
> >> channels and set each tx/rx operation to trigger the corresponding
> >> interrupt. This is not my whim, this is how the controller is!
> >>
> >
> > This looks like reasonable to me, theoretically.
> > Just not sure whether it's necessary to support it because we probably
> > will never use like that in reality, then it might become meaningless
> > complicity introduced and error prone.
> >
> It _is_ necessary to write controller driver independent of client drivers.
> 

Yes, that's true. What if we think we're writing driver against HW capabilities
which support 4 pair of channel links(tx/rx)? 
It looks like independent of client drivers and may make life easily.
Does it make sense?

> 
> > And AFAIK ARM MHU is doing the same way as MU which looks like also
> unidirectional channel.
> >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfo
> >
> center.arm.com%2Fhelp%2Findex.jsp%3Ftopic%3D%2Fcom.arm.doc.ddi0515
> b%2F
> >
> CHDGBGIF.html&amp;data=02%7C01%7Caisheng.dong%40nxp.com%7Cb728
> 16362983
> >
> 4208f9e908d5f37d3e00%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%
> 7C6366
> >
> 82641593785009&amp;sdata=VsG0oXxEObPRwT5KVss2eZthSTMTR2%2BMrv
> PqhDUpPGU
> > %3D&amp;reserved=0
> > drivers/mailbox/arm_mhu.c
> >
> MHU driver is doing exactly what the data sheet says. I know because I wrote
> the driver.
> 

Hmm... Maybe I missed something, but seems no from what I see:
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0922g/CCHHGIAH.html

Let's see the data sheet:
2.1.1. Physical channels
The CSS MHU peripheral provides physical channels that are used for communication between
the SCP and the AP.  These channels are called physical channels because their implementation
is fixed in hardware.  Each physical channel is unidirectional (SCP to AP or AP to SCP).

A physical channel comprises:
A 32-bit STAT (STATUS) register:
A 32-bit SET register:
A 32-bit CLEAR register:

The driver seems composes them into a pair of links with both tx and rx physical channel.
drivers/mailbox/arm_mhu.c
#define MHU_CHANS       3

struct mhu_link {
        unsigned irq;
        void __iomem *tx_reg;
        void __iomem *rx_reg;
};

for (i = 0; i < MHU_CHANS; i++) {
        mhu->chan[i].con_priv = &mhu->mlink[i];
        mhu->mlink[i].irq = adev->irq[i];
        mhu->mlink[i].rx_reg = mhu->base + mhu_reg[i];
        mhu->mlink[i].tx_reg = mhu->mlink[i].rx_reg + TX_REG_OFFSET;
}
And the driver totally supports 3 links lp, hp and sec. Each links actually
are using two unidirectional physical channels. Am I correct?

> 
> >> The SCU is poorly implemented as it ignores 3 irqs and club all 4
> >> register together (there are many other cons of this approach but
> >> lets not get into that).
> >> Personally, I would push-back on such a bad design. But if you claim
> >> you have no choice but to support SCU as such, the work around could
> >> be simpler than defining a new "scu mode" altogether.
> >>
> >
> > This is one of the recommended ways designed in HW reference manual
> > and it allows to send frame information up to 4 words without using shared
> memory.
> > SCU just follows it, so it's hard to believe it's a bad design.
> >
> >> #mbox-cells:  Must be 3.
> >>                       First cell is 1 for TX and 0 for RX channel
> >>                       Second cell is index of the channel [0,1,2 or 3]
> >>                       Third cell is 1 if the channel triggers an IRQ,
> >> 0 if not. That is ACR.RIE/TIE bits are set or not.
> >>
> >> Normal clients would always request a channel with irqs enabled.
> >> The SCU client would request all 8 channels -- TX/RX[0,1,2] with irqs
> >> disabled, TX/RX[3] with irqs enabled. And SCU will read/write 4 word
> >> data over 4 rx/tx channels, instead of the virtually concocted one.
> >>
> >
> > It may work If SCU protocol data length is fixed to 4 words. However,
> > it's length is flexible for different SVC service. That means this binding
> won't work for SCU.
> > And it will introduce much complexities during the implementation.
> >
> > Instead, we're using polling mode for both TX/RX and the data size is
> > stored in the msg header and sending msgs using all 4 data tx/rx registers
> as a channel fifo.
> >
> You first give me the "Passing short messages" usecase (quite a bad
> one) and ask how it can work. When I give you a solution, you effectively say
> "well, my usecase isn't even that". I feel I just wasted my time.
> 

Sorry for may not clear, "Passing short message' usecase is to tell how the
HW is working on one channel mode sending up to 4 words in one time
As specified in reference manual.

SCU does work that way, the only difference is it's using polling mode
rather than interrupt driven.  The point is the data size may be different
for each msg, so we can't simply know which data register interrupt
should be enable from static data defined in device tree.

> 
> >>
> >> > short messages
> >> > Transmit register(s) can be used to pass short messages from one to
> >> > four words in length. For example, when a four-word message is
> >> > desired, only one of the registers needs to have its corresponding
> >> > interrupt enable bit set at the receiver side; the message’s first
> >> > three words are written to the registers whose interrupt is masked,
> >> > and the fourth word is written to the other register (which
> >> > triggers an
> >> interrupt at the receiver side).
> >> >
> >> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> >> >
> >>
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> >> ww
> >> > .nxp.com%2Fdocs%2Fen%2Freference-
> >> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
> >> >
> >>
> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
> >> 86ea1
> >> >
> >>
> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
> >> a=54rE
> >> >
> >>
> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
> >> >
> >> > And SCU firmware does use MU in above way specified by reference
> >> manual.
> >> > Even from HW point of view, it's still one channel only mailbox.
> >> >
> >> Please realise that any manual is written by a mere mortal afterall.
> >> How the controller works is set in stone, but how the controller can
> >> be used ... is just someones suggestion.
> >>
> >> The approach I suggest above, conforms to the api and prevents a
> >> provider dancing to the tunes of a user.
> >
> > First of all, really appreciate for your suggestions.
> > It may be hard to find a common binding with the same mbox-cells.
> > It looks like we just need a property is distinguish how the Mailbox
> > is used In one channel or multi channel mode.
> >
> I get the idea you were ready to see the code merged in the coming window
> and be done with it. And now it feels lazy to change the code.

For me, I'm glad to change if we have a clear better solution.
And I do appreciate your suggestion and review time.

Why I'm a bit hesitate now is because your suggestion may not work for SCU,
(see above explanation), but it does work for generic M4 case.
But we' re now going to support both protocol in one mailbox driver.
Any suggestion on how to treat them properly if change the binding?

> I am sorry, but I can't allow controller drivers "emulating" some mode for a
> client driver. That is moving a part of client code into the controller driver.
> 

Okay, let's figure out it first.
Would you be more specific on what "emulating" did you mean in controller driver?
Sending up to 4 words capability in one channel mode as specified in reference manual?
That's I'm a bit confusing because I thought it's HW capability independent of client driver.

Or anything else?

> 
> > Directly checking mbox-cells seems the most easy way and it is already
> > Acked by Rob. Can't this way be Okay to you?
> >
> Rob is indeed far better than I am. But he also has to look into 100 other
> subsystems, whereas I am only looking into mailbox. I have time to dig into
> your datasheets. I believe Rob will point out anything wrong I suggest.
> 

Yes, you're in the fair enough authority to judge it. Thanks for your effort.

> BTW, the submitted driver doesn't even support the SCU usecase. Why the
> bindings?

Because that patch is firstly Acked by Rob. Others are reworked and ready to be sent
out against this patch series. But it seems we still have unresolved issues now as you
pointed out. We can first resolve them. Or do you need me to send out for your reference?

Regards
Dong Aisheng

Jassi Brar July 27, 2018, 6:46 a.m. UTC | #13

On Fri, Jul 27, 2018 at 11:30 AM, A.s. Dong <[e-mail]> wrote:
>> -----Original Message-----
>> From: Jassi Brar [mailto:[e-mail]]
>> Sent: Friday, July 27, 2018 12:56 PM
>> To: A.s. Dong <[e-mail]>
>> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
>> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
>> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
>> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
>> <[e-mail]>; , [e-mail],
>> [e-mail], srv_heupstream <linux-arm-
>> [e-mail]>; Devicetree List <[e-mail]>;
>> dl-linux-imx <[e-mail]>
>> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
>> channel support
>>
>> On Fri, Jul 27, 2018 at 9:32 AM, A.s. Dong <[e-mail]> wrote:
>> >>
>> >> >>
>> >> >> > Each MU has four pairs of rx/tx data register with four rx/tx
>> >> >> > interrupts which can also be used as a separate channel.
>> >> >> >
>> >> >> So the hardware actually supports 4 channels.
>> >> >>
>> >> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
>> >> >> > +- #mbox-cells:  Must be:
>> >> >> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
>> >> >> > +               1 - for multichannel (generic) mode.
>> >> >> > +
>> >> >> No, please.
>> >> >> DT bindings should reflect the real hardware, and not the software
>> >> >> mode we want the driver to work in.
>> >> >> Please define mbox-cells=1  and have the i.MX8* platform always
>> >> >> ask for channel-0.
>> >> >
>> >> > The reality is that MU hardware does not define channels in
>> >> > reference
>> >> manual.
>> >> > However, it does have four separate data tx/rx register which can
>> >> > be used as 'virtual' channels which is supported by this current driver.
>> >> >
>> >> > See below HW description from the reference manual:
>> >> > For messaging, the MU has four, 32-bit write-only transmit
>> >> > registers and four, 32-bit read-only receive registers on the
>> >> > Processor B and Processor A-sides. These registers are used for
>> >> > sending messages to each
>> >> other.
>> >> >
>> >> For a while please forget the protocol(user) level usage, and
>> >> consider only what your h/w is.
>> >>
>> >> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
>> >> (MU also has 4 "doorbell" type channels that it calls GP, but those
>> >> are not managed here, so lets not worry atm).
>> >>
>> >> By definition, a mailbox channel is simply a signal, optionally with
>> >> data attached. So, MU has 4 TX and 4 RX channels.
>> >>
>> >> The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
>> >> channels and set each tx/rx operation to trigger the corresponding
>> >> interrupt. This is not my whim, this is how the controller is!
>> >>
>> >
>> > This looks like reasonable to me, theoretically.
>> > Just not sure whether it's necessary to support it because we probably
>> > will never use like that in reality, then it might become meaningless
>> > complicity introduced and error prone.
>> >
>> It _is_ necessary to write controller driver independent of client drivers.
>>
>
> Yes, that's true. What if we think we're writing driver against HW capabilities
> which support 4 pair of channel links(tx/rx)?
> It looks like independent of client drivers and may make life easily.
> Does it make sense?
>
No, that would be emulation.
Why doesn't a usb device controller (udc) driver emulate FSG/HID etc,
by "thinking" it has a hardware backed storage/keyboard? It doesn't
because that is the job of upper protocol layer.


>>
>> > And AFAIK ARM MHU is doing the same way as MU which looks like also
>> unidirectional channel.
>> >
>> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfo
>> >
>> center.arm.com%2Fhelp%2Findex.jsp%3Ftopic%3D%2Fcom.arm.doc.ddi0515
>> b%2F
>> >
>> CHDGBGIF.html&amp;data=02%7C01%7Caisheng.dong%40nxp.com%7Cb728
>> 16362983
>> >
>> 4208f9e908d5f37d3e00%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%
>> 7C6366
>> >
>> 82641593785009&amp;sdata=VsG0oXxEObPRwT5KVss2eZthSTMTR2%2BMrv
>> PqhDUpPGU
>> > %3D&amp;reserved=0
>> > drivers/mailbox/arm_mhu.c
>> >
>> MHU driver is doing exactly what the data sheet says. I know because I wrote
>> the driver.
>>
>
> Hmm... Maybe I missed something, but seems no from what I see:
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0922g/CCHHGIAH.html
>
That is not the MHU datasheet.

Read Section 3.6 at page-3-41 of
http://infocenter.arm.com/help/topic/com.arm.doc.ddi0515b/DDI0515B_juno_arm_development_platform_soc_trm.pdf

This is another reason I can't let a bad code (emulation code)
through, because people start looking for examples to justify their
implementation rather than fixing it.


>>
>> >> The SCU is poorly implemented as it ignores 3 irqs and club all 4
>> >> register together (there are many other cons of this approach but
>> >> lets not get into that).
>> >> Personally, I would push-back on such a bad design. But if you claim
>> >> you have no choice but to support SCU as such, the work around could
>> >> be simpler than defining a new "scu mode" altogether.
>> >>
>> >
>> > This is one of the recommended ways designed in HW reference manual
>> > and it allows to send frame information up to 4 words without using shared
>> memory.
>> > SCU just follows it, so it's hard to believe it's a bad design.
>> >
>> >> #mbox-cells:  Must be 3.
>> >>                       First cell is 1 for TX and 0 for RX channel
>> >>                       Second cell is index of the channel [0,1,2 or 3]
>> >>                       Third cell is 1 if the channel triggers an IRQ,
>> >> 0 if not. That is ACR.RIE/TIE bits are set or not.
>> >>
>> >> Normal clients would always request a channel with irqs enabled.
>> >> The SCU client would request all 8 channels -- TX/RX[0,1,2] with irqs
>> >> disabled, TX/RX[3] with irqs enabled. And SCU will read/write 4 word
>> >> data over 4 rx/tx channels, instead of the virtually concocted one.
>> >>
>> >
>> > It may work If SCU protocol data length is fixed to 4 words. However,
>> > it's length is flexible for different SVC service. That means this binding
>> won't work for SCU.
>> > And it will introduce much complexities during the implementation.
>> >
>> > Instead, we're using polling mode for both TX/RX and the data size is
>> > stored in the msg header and sending msgs using all 4 data tx/rx registers
>> as a channel fifo.
>> >
>> You first give me the "Passing short messages" usecase (quite a bad
>> one) and ask how it can work. When I give you a solution, you effectively say
>> "well, my usecase isn't even that". I feel I just wasted my time.
>>
>
> Sorry for may not clear, "Passing short message' usecase is to tell how the
> HW is working on one channel mode sending up to 4 words in one time
> As specified in reference manual.
>
> SCU does work that way, the only difference is it's using polling mode
> rather than interrupt driven.  The point is the data size may be different
> for each msg, so we can't simply know which data register interrupt
> should be enable from static data defined in device tree.
>
And you think passing variable data through registers is a better idea
than passing packets via shared-memory?


>>
>> >>
>> >> > short messages
>> >> > Transmit register(s) can be used to pass short messages from one to
>> >> > four words in length. For example, when a four-word message is
>> >> > desired, only one of the registers needs to have its corresponding
>> >> > interrupt enable bit set at the receiver side; the message’s first
>> >> > three words are written to the registers whose interrupt is masked,
>> >> > and the fourth word is written to the other register (which
>> >> > triggers an
>> >> interrupt at the receiver side).
>> >> >
>> >> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
>> >> >
>> >>
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>> >> ww
>> >> > .nxp.com%2Fdocs%2Fen%2Freference-
>> >> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
>> >> >
>> >>
>> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
>> >> 86ea1
>> >> >
>> >>
>> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
>> >> a=54rE
>> >> >
>> >>
>> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
>> >> >
>> >> > And SCU firmware does use MU in above way specified by reference
>> >> manual.
>> >> > Even from HW point of view, it's still one channel only mailbox.
>> >> >
>> >> Please realise that any manual is written by a mere mortal afterall.
>> >> How the controller works is set in stone, but how the controller can
>> >> be used ... is just someones suggestion.
>> >>
>> >> The approach I suggest above, conforms to the api and prevents a
>> >> provider dancing to the tunes of a user.
>> >
>> > First of all, really appreciate for your suggestions.
>> > It may be hard to find a common binding with the same mbox-cells.
>> > It looks like we just need a property is distinguish how the Mailbox
>> > is used In one channel or multi channel mode.
>> >
>> I get the idea you were ready to see the code merged in the coming window
>> and be done with it. And now it feels lazy to change the code.
>
> For me, I'm glad to change if we have a clear better solution.
> And I do appreciate your suggestion and review time.
>
> Why I'm a bit hesitate now is because your suggestion may not work for SCU,
> (see above explanation), but it does work for generic M4 case.
> But we' re now going to support both protocol in one mailbox driver.
> Any suggestion on how to treat them properly if change the binding?
>
In your last post you said "This looks like reasonable to me, theoretically"
My suggestion is the same because I don't see why it won't work.


>> I am sorry, but I can't allow controller drivers "emulating" some mode for a
>> client driver. That is moving a part of client code into the controller driver.
>>
>
> Okay, let's figure out it first.
> Would you be more specific on what "emulating" did you mean in controller driver?
> Sending up to 4 words capability in one channel mode as specified in reference manual?
> That's I'm a bit confusing because I thought it's HW capability independent of client driver.
>
> Or anything else?
>
Emulation means pretending something that we are not.

The hardware has 8 unidirectional channels. But your protocol (SCU
implementation) assumes there is one _virtual_ channel that has 4
registers and 1/0 irq --- which is not true. Instead of fixing the
assumption in your protocol or emulating the virtual channel in the
protocol level (user of a MU), you want to add code in the controller
driver that ignores interrupts and club the 4 independent channels
together.

There is no end to protocols and their kinky assumptions, adding
"xyz-mode" support for each protocol isn't scalable.


>>
>> > Directly checking mbox-cells seems the most easy way and it is already
>> > Acked by Rob. Can't this way be Okay to you?
>> >
>> Rob is indeed far better than I am. But he also has to look into 100 other
>> subsystems, whereas I am only looking into mailbox. I have time to dig into
>> your datasheets. I believe Rob will point out anything wrong I suggest.
>>
>
> Yes, you're in the fair enough authority to judge it. Thanks for your effort.
>
>> BTW, the submitted driver doesn't even support the SCU usecase. Why the
>> bindings?
>
> Because that patch is firstly Acked by Rob. Others are reworked and ready to be sent
> out against this patch series. But it seems we still have unresolved issues now as you
> pointed out. We can first resolve them. Or do you need me to send out for your reference?
>
I am sure Rob took the best decision at the time with whatever
information provided to him.
Now, after reading the datasheet, we have the option to avoid
implementing consumer code in the provider driver.

Thanks.

Aisheng Dong July 27, 2018, 8:42 a.m. UTC | #14

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Friday, July 27, 2018 2:47 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; , [e-mail],
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Fri, Jul 27, 2018 at 11:30 AM, A.s. Dong <[e-mail]> wrote:
> >> -----Original Message-----
> >> From: Jassi Brar [mailto:[e-mail]]
> >> Sent: Friday, July 27, 2018 12:56 PM
> >> To: A.s. Dong <[e-mail]>
> >> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> >> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> >> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> >> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> >> <[e-mail]>; ,
> >> [e-mail],
> >> [e-mail], srv_heupstream <linux-arm-
> >> [e-mail]>; Devicetree List
> >> <[e-mail]>; dl-linux-imx <[e-mail]>
> >> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic
> >> MU channel support
> >>
> >> On Fri, Jul 27, 2018 at 9:32 AM, A.s. Dong <[e-mail]> wrote:
> >> >>
> >> >> >>
> >> >> >> > Each MU has four pairs of rx/tx data register with four rx/tx
> >> >> >> > interrupts which can also be used as a separate channel.
> >> >> >> >
> >> >> >> So the hardware actually supports 4 channels.
> >> >> >>
> >> >> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> >> >> >> > +- #mbox-cells:  Must be:
> >> >> >> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
> >> >> >> > +               1 - for multichannel (generic) mode.
> >> >> >> > +
> >> >> >> No, please.
> >> >> >> DT bindings should reflect the real hardware, and not the
> >> >> >> software mode we want the driver to work in.
> >> >> >> Please define mbox-cells=1  and have the i.MX8* platform always
> >> >> >> ask for channel-0.
> >> >> >
> >> >> > The reality is that MU hardware does not define channels in
> >> >> > reference
> >> >> manual.
> >> >> > However, it does have four separate data tx/rx register which
> >> >> > can be used as 'virtual' channels which is supported by this current
> driver.
> >> >> >
> >> >> > See below HW description from the reference manual:
> >> >> > For messaging, the MU has four, 32-bit write-only transmit
> >> >> > registers and four, 32-bit read-only receive registers on the
> >> >> > Processor B and Processor A-sides. These registers are used for
> >> >> > sending messages to each
> >> >> other.
> >> >> >
> >> >> For a while please forget the protocol(user) level usage, and
> >> >> consider only what your h/w is.
> >> >>
> >> >> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
> >> >> (MU also has 4 "doorbell" type channels that it calls GP, but
> >> >> those are not managed here, so lets not worry atm).
> >> >>
> >> >> By definition, a mailbox channel is simply a signal, optionally
> >> >> with data attached. So, MU has 4 TX and 4 RX channels.
> >> >>
> >> >> The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
> >> >> channels and set each tx/rx operation to trigger the corresponding
> >> >> interrupt. This is not my whim, this is how the controller is!
> >> >>
> >> >
> >> > This looks like reasonable to me, theoretically.
> >> > Just not sure whether it's necessary to support it because we
> >> > probably will never use like that in reality, then it might become
> >> > meaningless complicity introduced and error prone.
> >> >
> >> It _is_ necessary to write controller driver independent of client drivers.
> >>
> >
> > Yes, that's true. What if we think we're writing driver against HW
> > capabilities which support 4 pair of channel links(tx/rx)?
> > It looks like independent of client drivers and may make life easily.
> > Does it make sense?
> >
> No, that would be emulation.
> Why doesn't a usb device controller (udc) driver emulate FSG/HID etc, by
> "thinking" it has a hardware backed storage/keyboard? It doesn't because
> that is the job of upper protocol layer.
> 

Sorry I'm not quite familiar with USB device.
IMHO the HW supports many capabilities, but it does not mean
we need support them all. For MU case, a pair of channel link capable of both tx/rx
seems like a better using. It's irrelevant of client drivers. It's simply that HW supports
different kind of modes (one channel mode, one link mode, separate per register mode)
and we just support link mode.

Another reason is I doubt that we may never use per register mode in a different register pair
in the future. For example:
AP:
node {
	...
	// cell 0 meaning: 0: tx 1: rx	cell1 meaning: channel id
	Mboxes = <&mbox 0 1 &mbox 1 2>
	Mbox-names = "tx", "rx";
>

M4:
node {
	...
	Mboxes = <&mbox 0 2 &mbox 1 1>
	Mbox-names = "tx", "rx";
>
This make things complicated and error prone as I said before.

But that's just my understanding and may overlook something, if you still think
we should do exactly as above, I will not against it because it does work for M4 case.

Then the left question is how we handle SCU case?

> >>
> >> > And AFAIK ARM MHU is doing the same way as MU which looks like also
> >> unidirectional channel.
> >> >
> >>
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finf
> >> o
> >> >
> >>
> center.arm.com%2Fhelp%2Findex.jsp%3Ftopic%3D%2Fcom.arm.doc.ddi0515
> >> b%2F
> >> >
> >>
> CHDGBGIF.html&amp;data=02%7C01%7Caisheng.dong%40nxp.com%7Cb728
> >> 16362983
> >> >
> >>
> 4208f9e908d5f37d3e00%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%
> >> 7C6366
> >> >
> >>
> 82641593785009&amp;sdata=VsG0oXxEObPRwT5KVss2eZthSTMTR2%2BMrv
> >> PqhDUpPGU
> >> > %3D&amp;reserved=0
> >> > drivers/mailbox/arm_mhu.c
> >> >
> >> MHU driver is doing exactly what the data sheet says. I know because
> >> I wrote the driver.
> >>
> >
> > Hmm... Maybe I missed something, but seems no from what I see:
> >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfo
> >
> center.arm.com%2Fhelp%2Findex.jsp%3Ftopic%3D%2Fcom.arm.doc.dui0922
> g%2F
> >
> CCHHGIAH.html&amp;data=02%7C01%7Caisheng.dong%40nxp.com%7Cd2b1
> 23b89dae
> >
> 4a00cb7108d5f38cb611%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%
> 7C6366
> >
> 82708036553137&amp;sdata=4SMUpH%2FO9MWArC%2BjbPy%2BrbNAqUla
> o9IezKUi7UX
> > gIyQ%3D&amp;reserved=0
> >
> That is not the MHU datasheet.
> 
> Read Section 3.6 at page-3-41 of
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfo
> center.arm.com%2Fhelp%2Ftopic%2Fcom.arm.doc.ddi0515b%2FDDI0515B_j
> uno_arm_development_platform_soc_trm.pdf&amp;data=02%7C01%7Cais
> heng.dong%40nxp.com%7Cd2b123b89dae4a00cb7108d5f38cb611%7C686ea1
> d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C636682708036553137&amp;sdat
> a=eUd75VnwEs44ZoVV0n7R1G3UfAH%2B%2BpeoGns7Mq5UBN8%3D&amp;
> reserved=0
> 
> This is another reason I can't let a bad code (emulation code) through,
> because people start looking for examples to justify their implementation
> rather than fixing it.
> 

I'm a bit confusing....
The section 3.6 you pointed is the MHU register description. It does not conflict
with what I see from ARM doc center that each physical channel is unidirectional.

See below:
Chan 1:
0x000 SCP_INTR_L_STAT
0x008 SCP_INTR_L_SET
0x010 SCP _INTR_L_CLEAR

Chan 2:
0x020 SCP_INTR_H_STAT
0x028 SCP_INTR_H_SET
0x030 SCP _INTR_H_CLEAR

Chan 3:
0x100 CPU_INTR_L_STAT
0x108 CPU_INTR_L_SET
0x110 CPU_INTR_L_CLEAR

Chan 4:
0x120 CPU_INTR_H_STAT
0x128 CPU_INTR_H_SET
0x130 CPU_INTR_H_CLEAR

Chan 5:
0x200 SCP_INTR_S_STAT
0x208 SCP_INTR_S_SET
0x210 SCP _INTR_S_CLEAR

Chan 6:
0x300 CPU_INTR_S_STAT
0x308 CPU_INTR_S_SET
0x310 CPU_INTR_S_CLEAR

And the driver compose them into 3 channel links (lp, hp and sec).
Am I wrong?

> 
> >>
> >> >> The SCU is poorly implemented as it ignores 3 irqs and club all 4
> >> >> register together (there are many other cons of this approach but
> >> >> lets not get into that).
> >> >> Personally, I would push-back on such a bad design. But if you
> >> >> claim you have no choice but to support SCU as such, the work
> >> >> around could be simpler than defining a new "scu mode" altogether.
> >> >>
> >> >
> >> > This is one of the recommended ways designed in HW reference
> manual
> >> > and it allows to send frame information up to 4 words without using
> >> > shared
> >> memory.
> >> > SCU just follows it, so it's hard to believe it's a bad design.
> >> >
> >> >> #mbox-cells:  Must be 3.
> >> >>                       First cell is 1 for TX and 0 for RX channel
> >> >>                       Second cell is index of the channel [0,1,2 or 3]
> >> >>                       Third cell is 1 if the channel triggers an
> >> >> IRQ,
> >> >> 0 if not. That is ACR.RIE/TIE bits are set or not.
> >> >>
> >> >> Normal clients would always request a channel with irqs enabled.
> >> >> The SCU client would request all 8 channels -- TX/RX[0,1,2] with
> >> >> irqs disabled, TX/RX[3] with irqs enabled. And SCU will read/write
> >> >> 4 word data over 4 rx/tx channels, instead of the virtually concocted
> one.
> >> >>
> >> >
> >> > It may work If SCU protocol data length is fixed to 4 words.
> >> > However, it's length is flexible for different SVC service. That
> >> > means this binding
> >> won't work for SCU.
> >> > And it will introduce much complexities during the implementation.
> >> >
> >> > Instead, we're using polling mode for both TX/RX and the data size
> >> > is stored in the msg header and sending msgs using all 4 data tx/rx
> >> > registers
> >> as a channel fifo.
> >> >
> >> You first give me the "Passing short messages" usecase (quite a bad
> >> one) and ask how it can work. When I give you a solution, you
> >> effectively say "well, my usecase isn't even that". I feel I just wasted my
> time.
> >>
> >
> > Sorry for may not clear, "Passing short message' usecase is to tell
> > how the HW is working on one channel mode sending up to 4 words in one
> > time As specified in reference manual.
> >
> > SCU does work that way, the only difference is it's using polling mode
> > rather than interrupt driven.  The point is the data size may be
> > different for each msg, so we can't simply know which data register
> > interrupt should be enable from static data defined in device tree.
> >
> And you think passing variable data through registers is a better idea than
> passing packets via shared-memory?
> 

You got me. :-)
I've no idea about which one is better. The problem is SCU firmware is already
there passing packets through data registers, we have no way to change it.

> 
> >>
> >> >>
> >> >> > short messages
> >> >> > Transmit register(s) can be used to pass short messages from one
> >> >> > to four words in length. For example, when a four-word message
> >> >> > is desired, only one of the registers needs to have its
> >> >> > corresponding interrupt enable bit set at the receiver side; the
> >> >> > message’s first three words are written to the registers whose
> >> >> > interrupt is masked, and the fourth word is written to the other
> >> >> > register (which triggers an
> >> >> interrupt at the receiver side).
> >> >> >
> >> >> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> >> >> >
> >> >>
> >>
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> >> >> ww
> >> >> > .nxp.com%2Fdocs%2Fen%2Freference-
> >> >> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
> >> >> >
> >> >>
> >>
> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
> >> >> 86ea1
> >> >> >
> >> >>
> >>
> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
> >> >> a=54rE
> >> >> >
> >> >>
> >>
> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
> >> >> >
> >> >> > And SCU firmware does use MU in above way specified by reference
> >> >> manual.
> >> >> > Even from HW point of view, it's still one channel only mailbox.
> >> >> >
> >> >> Please realise that any manual is written by a mere mortal afterall.
> >> >> How the controller works is set in stone, but how the controller
> >> >> can be used ... is just someones suggestion.
> >> >>
> >> >> The approach I suggest above, conforms to the api and prevents a
> >> >> provider dancing to the tunes of a user.
> >> >
> >> > First of all, really appreciate for your suggestions.
> >> > It may be hard to find a common binding with the same mbox-cells.
> >> > It looks like we just need a property is distinguish how the
> >> > Mailbox is used In one channel or multi channel mode.
> >> >
> >> I get the idea you were ready to see the code merged in the coming
> >> window and be done with it. And now it feels lazy to change the code.
> >
> > For me, I'm glad to change if we have a clear better solution.
> > And I do appreciate your suggestion and review time.
> >
> > Why I'm a bit hesitate now is because your suggestion may not work for
> > SCU, (see above explanation), but it does work for generic M4 case.
> > But we' re now going to support both protocol in one mailbox driver.
> > Any suggestion on how to treat them properly if change the binding?
> >
> In your last post you said "This looks like reasonable to me, theoretically"
> My suggestion is the same because I don't see why it won't work.
> 

Sorry for not clear. I mean it's reasonable for M4 generic using case.
But not for SCU case.

> 
> >> I am sorry, but I can't allow controller drivers "emulating" some
> >> mode for a client driver. That is moving a part of client code into the
> controller driver.
> >>
> >
> > Okay, let's figure out it first.
> > Would you be more specific on what "emulating" did you mean in
> controller driver?
> > Sending up to 4 words capability in one channel mode as specified in
> reference manual?
> > That's I'm a bit confusing because I thought it's HW capability independent
> of client driver.
> >
> > Or anything else?
> >
> Emulation means pretending something that we are not.
> 
> The hardware has 8 unidirectional channels. But your protocol (SCU
> implementation) assumes there is one _virtual_ channel that has 4 registers
> and 1/0 irq --- which is not true. Instead of fixing the assumption in your
> protocol or emulating the virtual channel in the protocol level (user of a MU),
> you want to add code in the controller driver that ignores interrupts and club
> the 4 independent channels together.
> 

This stucks me. This is how the hardware is designed  and suggested to use
in hardware reference manual. And now you're telling me this is wrong and
we should not use the design in reference manual...

> There is no end to protocols and their kinky assumptions, adding "xyz-mode"
> support for each protocol isn't scalable.
> 

This is also how Sascha suggested me to move to mailbox to support
both protocols and handle them differently in mailbox driver.

Honestly I was hesitated to do that before because I doubt the value
we can gain if switching to mailbox besides the unnecessary complexity
introduced and performance downgrading (extra execution of a few
unnecessary code in mailbox API and It's even worse if use 8 separate
channels for SCU, comparing to original only tens of lines of library API way),
but Sascha insisted...

The mailbox itself is somehow protocol specific (doorbell, signal, data packet and etc)
It's hard for us to find a common way to support two totally different protocols.
So the controller needs to know different protocol it is transferring.

> 
> >>
> >> > Directly checking mbox-cells seems the most easy way and it is
> >> > already Acked by Rob. Can't this way be Okay to you?
> >> >
> >> Rob is indeed far better than I am. But he also has to look into 100
> >> other subsystems, whereas I am only looking into mailbox. I have time
> >> to dig into your datasheets. I believe Rob will point out anything wrong I
> suggest.
> >>
> >
> > Yes, you're in the fair enough authority to judge it. Thanks for your effort.
> >
> >> BTW, the submitted driver doesn't even support the SCU usecase. Why
> >> the bindings?
> >
> > Because that patch is firstly Acked by Rob. Others are reworked and
> > ready to be sent out against this patch series. But it seems we still
> > have unresolved issues now as you pointed out. We can first resolve them.
> Or do you need me to send out for your reference?
> >
> I am sure Rob took the best decision at the time with whatever information
> provided to him.
> Now, after reading the datasheet, we have the option to avoid implementing
> consumer code in the provider driver.

I have one doubt, irrelevant of SCU protocol, from the datasheet, the hardware
does support transfer message up to 4 words in one channel mode and it is the
recommended way in datasheet for less than 4 words frame transferring,
why switching to mailbox framework, we can't use it now?
And makes us lose the HW capability.

If that, what's meaning for us to switch to mailbox framework?

Regards
Dong Aisheng

> 
> Thanks.

Aisheng Dong July 27, 2018, 9:05 a.m. UTC | #15

> -----Original Message-----
> From: A.s. Dong
> Sent: Friday, July 27, 2018 4:42 PM
> To: Jassi Brar <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; , [e-mail],
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: RE: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> > -----Original Message-----
> > From: Jassi Brar [mailto:[e-mail]]
> > Sent: Friday, July 27, 2018 2:47 PM
> > To: A.s. Dong <[e-mail]>
> > Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> > <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> > Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> > Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> > <[e-mail]>; ,
> > [e-mail],
> > [e-mail], srv_heupstream <linux-arm-
> > [e-mail]>; Devicetree List
> > <[e-mail]>; dl-linux-imx <[e-mail]>
> > Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic
> > MU channel support
> >
> > On Fri, Jul 27, 2018 at 11:30 AM, A.s. Dong <[e-mail]> wrote:
> > >> -----Original Message-----
> > >> From: Jassi Brar [mailto:[e-mail]]
> > >> Sent: Friday, July 27, 2018 12:56 PM
> > >> To: A.s. Dong <[e-mail]>
> > >> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> > >> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> > >> Estevam <[e-mail]>; Rob Herring
> <[e-mail]>;
> > >> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> > >> <[e-mail]>; ,
> > >> [e-mail],
> > >> [e-mail], srv_heupstream <linux-arm-
> > >> [e-mail]>; Devicetree List
> > >> <[e-mail]>; dl-linux-imx <[e-mail]>
> > >> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add
> > >> generic MU channel support
> > >>
> > >> On Fri, Jul 27, 2018 at 9:32 AM, A.s. Dong <[e-mail]>
> wrote:
> > >> >>
> > >> >> >>
> > >> >> >> > Each MU has four pairs of rx/tx data register with four
> > >> >> >> > rx/tx interrupts which can also be used as a separate channel.
> > >> >> >> >
> > >> >> >> So the hardware actually supports 4 channels.
> > >> >> >>
> > >> >> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> > >> >> >> > +- #mbox-cells:  Must be:
> > >> >> >> > +               0 - for single channel mode. i.MX8* SCU protocol
> specific.
> > >> >> >> > +               1 - for multichannel (generic) mode.
> > >> >> >> > +
> > >> >> >> No, please.
> > >> >> >> DT bindings should reflect the real hardware, and not the
> > >> >> >> software mode we want the driver to work in.
> > >> >> >> Please define mbox-cells=1  and have the i.MX8* platform
> > >> >> >> always ask for channel-0.
> > >> >> >
> > >> >> > The reality is that MU hardware does not define channels in
> > >> >> > reference
> > >> >> manual.
> > >> >> > However, it does have four separate data tx/rx register which
> > >> >> > can be used as 'virtual' channels which is supported by this
> > >> >> > current
> > driver.
> > >> >> >
> > >> >> > See below HW description from the reference manual:
> > >> >> > For messaging, the MU has four, 32-bit write-only transmit
> > >> >> > registers and four, 32-bit read-only receive registers on the
> > >> >> > Processor B and Processor A-sides. These registers are used
> > >> >> > for sending messages to each
> > >> >> other.
> > >> >> >
> > >> >> For a while please forget the protocol(user) level usage, and
> > >> >> consider only what your h/w is.
> > >> >>
> > >> >> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
> > >> >> (MU also has 4 "doorbell" type channels that it calls GP, but
> > >> >> those are not managed here, so lets not worry atm).
> > >> >>
> > >> >> By definition, a mailbox channel is simply a signal, optionally
> > >> >> with data attached. So, MU has 4 TX and 4 RX channels.
> > >> >>
> > >> >> The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
> > >> >> channels and set each tx/rx operation to trigger the
> > >> >> corresponding interrupt. This is not my whim, this is how the
> controller is!
> > >> >>
> > >> >
> > >> > This looks like reasonable to me, theoretically.
> > >> > Just not sure whether it's necessary to support it because we
> > >> > probably will never use like that in reality, then it might
> > >> > become meaningless complicity introduced and error prone.
> > >> >
> > >> It _is_ necessary to write controller driver independent of client drivers.
> > >>
> > >
> > > Yes, that's true. What if we think we're writing driver against HW
> > > capabilities which support 4 pair of channel links(tx/rx)?
> > > It looks like independent of client drivers and may make life easily.
> > > Does it make sense?
> > >
> > No, that would be emulation.
> > Why doesn't a usb device controller (udc) driver emulate FSG/HID etc,
> > by "thinking" it has a hardware backed storage/keyboard? It doesn't
> > because that is the job of upper protocol layer.
> >
> 
> Sorry I'm not quite familiar with USB device.
> IMHO the HW supports many capabilities, but it does not mean we need
> support them all. For MU case, a pair of channel link capable of both tx/rx
> seems like a better using. It's irrelevant of client drivers. It's simply that HW
> supports different kind of modes (one channel mode, one link mode,
> separate per register mode) and we just support link mode.
> 
> Another reason is I doubt that we may never use per register mode in a
> different register pair in the future. For example:
> AP:
> node {
> 	...
> 	// cell 0 meaning: 0: tx 1: rx	cell1 meaning: channel id
> 	Mboxes = <&mbox 0 1 &mbox 1 2>
> 	Mbox-names = "tx", "rx";
> >
> 
> M4:
> node {
> 	...
> 	Mboxes = <&mbox 0 2 &mbox 1 1>
> 	Mbox-names = "tx", "rx";
> >
> This make things complicated and error prone as I said before.
> 
> But that's just my understanding and may overlook something, if you still
> think we should do exactly as above, I will not against it because it does work
> for M4 case.
> 
> Then the left question is how we handle SCU case?
> 
> > >>
> > >> > And AFAIK ARM MHU is doing the same way as MU which looks like
> > >> > also
> > >> unidirectional channel.
> > >> >
> > >>
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finf
> > >> o
> > >> >
> > >>
> >
> center.arm.com%2Fhelp%2Findex.jsp%3Ftopic%3D%2Fcom.arm.doc.ddi0515
> > >> b%2F
> > >> >
> > >>
> >
> CHDGBGIF.html&amp;data=02%7C01%7Caisheng.dong%40nxp.com%7Cb728
> > >> 16362983
> > >> >
> > >>
> > 4208f9e908d5f37d3e00%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%
> > >> 7C6366
> > >> >
> > >>
> >
> 82641593785009&amp;sdata=VsG0oXxEObPRwT5KVss2eZthSTMTR2%2BMrv
> > >> PqhDUpPGU
> > >> > %3D&amp;reserved=0
> > >> > drivers/mailbox/arm_mhu.c
> > >> >
> > >> MHU driver is doing exactly what the data sheet says. I know
> > >> because I wrote the driver.
> > >>
> > >
> > > Hmm... Maybe I missed something, but seems no from what I see:
> > >
> >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfo
> > >
> >
> center.arm.com%2Fhelp%2Findex.jsp%3Ftopic%3D%2Fcom.arm.doc.dui0922
> > g%2F
> > >
> >
> CCHHGIAH.html&amp;data=02%7C01%7Caisheng.dong%40nxp.com%7Cd2b1
> > 23b89dae
> > >
> > 4a00cb7108d5f38cb611%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%
> > 7C6366
> > >
> >
> 82708036553137&amp;sdata=4SMUpH%2FO9MWArC%2BjbPy%2BrbNAqUla
> > o9IezKUi7UX
> > > gIyQ%3D&amp;reserved=0
> > >
> > That is not the MHU datasheet.
> >
> > Read Section 3.6 at page-3-41 of
> >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfo
> >
> center.arm.com%2Fhelp%2Ftopic%2Fcom.arm.doc.ddi0515b%2FDDI0515B_j
> >
> uno_arm_development_platform_soc_trm.pdf&amp;data=02%7C01%7Cais
> >
> heng.dong%40nxp.com%7Cd2b123b89dae4a00cb7108d5f38cb611%7C686ea1
> >
> d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C636682708036553137&amp;sdat
> >
> a=eUd75VnwEs44ZoVV0n7R1G3UfAH%2B%2BpeoGns7Mq5UBN8%3D&amp;
> > reserved=0
> >
> > This is another reason I can't let a bad code (emulation code)
> > through, because people start looking for examples to justify their
> > implementation rather than fixing it.
> >
> 
> I'm a bit confusing....
> The section 3.6 you pointed is the MHU register description. It does not
> conflict with what I see from ARM doc center that each physical channel is
> unidirectional.
> 
> See below:
> Chan 1:
> 0x000 SCP_INTR_L_STAT
> 0x008 SCP_INTR_L_SET
> 0x010 SCP _INTR_L_CLEAR
> 
> Chan 2:
> 0x020 SCP_INTR_H_STAT
> 0x028 SCP_INTR_H_SET
> 0x030 SCP _INTR_H_CLEAR
> 
> Chan 3:
> 0x100 CPU_INTR_L_STAT
> 0x108 CPU_INTR_L_SET
> 0x110 CPU_INTR_L_CLEAR
> 
> Chan 4:
> 0x120 CPU_INTR_H_STAT
> 0x128 CPU_INTR_H_SET
> 0x130 CPU_INTR_H_CLEAR
> 
> Chan 5:
> 0x200 SCP_INTR_S_STAT
> 0x208 SCP_INTR_S_SET
> 0x210 SCP _INTR_S_CLEAR
> 
> Chan 6:
> 0x300 CPU_INTR_S_STAT
> 0x308 CPU_INTR_S_SET
> 0x310 CPU_INTR_S_CLEAR
> 
> And the driver compose them into 3 channel links (lp, hp and sec).
> Am I wrong?
> 
> >
> > >>
> > >> >> The SCU is poorly implemented as it ignores 3 irqs and club all
> > >> >> 4 register together (there are many other cons of this approach
> > >> >> but lets not get into that).
> > >> >> Personally, I would push-back on such a bad design. But if you
> > >> >> claim you have no choice but to support SCU as such, the work
> > >> >> around could be simpler than defining a new "scu mode" altogether.
> > >> >>
> > >> >
> > >> > This is one of the recommended ways designed in HW reference
> > manual
> > >> > and it allows to send frame information up to 4 words without
> > >> > using shared
> > >> memory.
> > >> > SCU just follows it, so it's hard to believe it's a bad design.
> > >> >
> > >> >> #mbox-cells:  Must be 3.
> > >> >>                       First cell is 1 for TX and 0 for RX channel
> > >> >>                       Second cell is index of the channel [0,1,2 or 3]
> > >> >>                       Third cell is 1 if the channel triggers an
> > >> >> IRQ,
> > >> >> 0 if not. That is ACR.RIE/TIE bits are set or not.
> > >> >>
> > >> >> Normal clients would always request a channel with irqs enabled.
> > >> >> The SCU client would request all 8 channels -- TX/RX[0,1,2] with
> > >> >> irqs disabled, TX/RX[3] with irqs enabled. And SCU will
> > >> >> read/write
> > >> >> 4 word data over 4 rx/tx channels, instead of the virtually
> > >> >> concocted
> > one.
> > >> >>
> > >> >
> > >> > It may work If SCU protocol data length is fixed to 4 words.
> > >> > However, it's length is flexible for different SVC service. That
> > >> > means this binding
> > >> won't work for SCU.
> > >> > And it will introduce much complexities during the implementation.
> > >> >
> > >> > Instead, we're using polling mode for both TX/RX and the data
> > >> > size is stored in the msg header and sending msgs using all 4
> > >> > data tx/rx registers
> > >> as a channel fifo.
> > >> >
> > >> You first give me the "Passing short messages" usecase (quite a bad
> > >> one) and ask how it can work. When I give you a solution, you
> > >> effectively say "well, my usecase isn't even that". I feel I just
> > >> wasted my
> > time.
> > >>
> > >
> > > Sorry for may not clear, "Passing short message' usecase is to tell
> > > how the HW is working on one channel mode sending up to 4 words in
> > > one time As specified in reference manual.
> > >
> > > SCU does work that way, the only difference is it's using polling
> > > mode rather than interrupt driven.  The point is the data size may
> > > be different for each msg, so we can't simply know which data
> > > register interrupt should be enable from static data defined in device tree.
> > >
> > And you think passing variable data through registers is a better idea
> > than passing packets via shared-memory?
> >
> 
> You got me. :-)
> I've no idea about which one is better. The problem is SCU firmware is
> already there passing packets through data registers, we have no way to
> change it.
> 
> >
> > >>
> > >> >>
> > >> >> > short messages
> > >> >> > Transmit register(s) can be used to pass short messages from
> > >> >> > one to four words in length. For example, when a four-word
> > >> >> > message is desired, only one of the registers needs to have
> > >> >> > its corresponding interrupt enable bit set at the receiver
> > >> >> > side; the message’s first three words are written to the
> > >> >> > registers whose interrupt is masked, and the fourth word is
> > >> >> > written to the other register (which triggers an
> > >> >> interrupt at the receiver side).
> > >> >> >
> > >> >> > The reference manual is at here: (Chapter 42 Messaging Unit
> > >> >> > (MU)
> > >> >> >
> > >> >>
> > >>
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> > >> >> ww
> > >> >> > .nxp.com%2Fdocs%2Fen%2Freference-
> > >> >> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
> > >> >> >
> > >> >>
> > >>
> >
> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
> > >> >> 86ea1
> > >> >> >
> > >> >>
> > >>
> >
> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
> > >> >> a=54rE
> > >> >> >
> > >> >>
> > >>
> >
> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
> > >> >> >
> > >> >> > And SCU firmware does use MU in above way specified by
> > >> >> > reference
> > >> >> manual.
> > >> >> > Even from HW point of view, it's still one channel only mailbox.
> > >> >> >
> > >> >> Please realise that any manual is written by a mere mortal afterall.
> > >> >> How the controller works is set in stone, but how the controller
> > >> >> can be used ... is just someones suggestion.
> > >> >>
> > >> >> The approach I suggest above, conforms to the api and prevents a
> > >> >> provider dancing to the tunes of a user.
> > >> >
> > >> > First of all, really appreciate for your suggestions.
> > >> > It may be hard to find a common binding with the same mbox-cells.
> > >> > It looks like we just need a property is distinguish how the
> > >> > Mailbox is used In one channel or multi channel mode.
> > >> >
> > >> I get the idea you were ready to see the code merged in the coming
> > >> window and be done with it. And now it feels lazy to change the code.
> > >
> > > For me, I'm glad to change if we have a clear better solution.
> > > And I do appreciate your suggestion and review time.
> > >
> > > Why I'm a bit hesitate now is because your suggestion may not work
> > > for SCU, (see above explanation), but it does work for generic M4 case.
> > > But we' re now going to support both protocol in one mailbox driver.
> > > Any suggestion on how to treat them properly if change the binding?
> > >
> > In your last post you said "This looks like reasonable to me, theoretically"
> > My suggestion is the same because I don't see why it won't work.
> >
> 
> Sorry for not clear. I mean it's reasonable for M4 generic using case.
> But not for SCU case.
> 
> >
> > >> I am sorry, but I can't allow controller drivers "emulating" some
> > >> mode for a client driver. That is moving a part of client code into
> > >> the
> > controller driver.
> > >>
> > >
> > > Okay, let's figure out it first.
> > > Would you be more specific on what "emulating" did you mean in
> > controller driver?
> > > Sending up to 4 words capability in one channel mode as specified in
> > reference manual?
> > > That's I'm a bit confusing because I thought it's HW capability
> > > independent
> > of client driver.
> > >
> > > Or anything else?
> > >
> > Emulation means pretending something that we are not.
> >
> > The hardware has 8 unidirectional channels. But your protocol (SCU
> > implementation) assumes there is one _virtual_ channel that has 4
> > registers and 1/0 irq --- which is not true. Instead of fixing the
> > assumption in your protocol or emulating the virtual channel in the
> > protocol level (user of a MU), you want to add code in the controller
> > driver that ignores interrupts and club the 4 independent channels
> together.
> >
> 
> This stucks me. This is how the hardware is designed  and suggested to use in
> hardware reference manual. And now you're telling me this is wrong and we
> should not use the design in reference manual...
> 
> > There is no end to protocols and their kinky assumptions, adding "xyz-
> mode"
> > support for each protocol isn't scalable.
> >
> 
> This is also how Sascha suggested me to move to mailbox to support both
> protocols and handle them differently in mailbox driver.
> 
> Honestly I was hesitated to do that before because I doubt the value we can
> gain if switching to mailbox besides the unnecessary complexity introduced
> and performance downgrading (extra execution of a few unnecessary code
> in mailbox API and It's even worse if use 8 separate channels for SCU,
> comparing to original only tens of lines of library API way), but Sascha
> insisted...
> 
> The mailbox itself is somehow protocol specific (doorbell, signal, data packet
> and etc) It's hard for us to find a common way to support two totally
> different protocols.
> So the controller needs to know different protocol it is transferring.
> 
> >
> > >>
> > >> > Directly checking mbox-cells seems the most easy way and it is
> > >> > already Acked by Rob. Can't this way be Okay to you?
> > >> >
> > >> Rob is indeed far better than I am. But he also has to look into
> > >> 100 other subsystems, whereas I am only looking into mailbox. I
> > >> have time to dig into your datasheets. I believe Rob will point out
> > >> anything wrong I
> > suggest.
> > >>
> > >
> > > Yes, you're in the fair enough authority to judge it. Thanks for your effort.
> > >
> > >> BTW, the submitted driver doesn't even support the SCU usecase. Why
> > >> the bindings?
> > >
> > > Because that patch is firstly Acked by Rob. Others are reworked and
> > > ready to be sent out against this patch series. But it seems we
> > > still have unresolved issues now as you pointed out. We can first resolve
> them.
> > Or do you need me to send out for your reference?
> > >
> > I am sure Rob took the best decision at the time with whatever
> > information provided to him.
> > Now, after reading the datasheet, we have the option to avoid
> > implementing consumer code in the provider driver.
> 
> I have one doubt, irrelevant of SCU protocol, from the datasheet, the
> hardware does support transfer message up to 4 words in one channel mode
> and it is the recommended way in datasheet for less than 4 words frame
> transferring, why switching to mailbox framework, we can't use it now?
> And makes us lose the HW capability.
> 

See TI mailbox does in a quite similar way as SCU.
The difference is TI mailbox use TX/Rx buffer while i.MX MU uses 4 TX/RX regsiters.
And TI mailbox support send up to 8 words in TX/RX buffer in one time while
i.MX MU supports Send up to 4 words in one time.

truct ti_msgmgr_message {
        size_t len;
        u8 *buf;
};
drivers/mailbox/ti-msgmgr.c: ti_msgmgr_send_data()

Regards
Dong Aisheng

> If that, what's meaning for us to switch to mailbox framework?
> 
> Regards
> Dong Aisheng
> 
> >
> > Thanks.

Jassi Brar July 28, 2018, 1:09 p.m. UTC | #16

On Fri, Jul 27, 2018 at 2:12 PM, A.s. Dong <[e-mail]> wrote:


>> >> >>
>> >> >> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
>> >> >> (MU also has 4 "doorbell" type channels that it calls GP, but
>> >> >> those are not managed here, so lets not worry atm).
>> >> >>
>> >> >> By definition, a mailbox channel is simply a signal, optionally
>> >> >> with data attached. So, MU has 4 TX and 4 RX channels.
>> >> >>
>> >> >> The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
>> >> >> channels and set each tx/rx operation to trigger the corresponding
>> >> >> interrupt. This is not my whim, this is how the controller is!
>> >> >>
>> >> >
>> >> > This looks like reasonable to me, theoretically.
>> >> > Just not sure whether it's necessary to support it because we
>> >> > probably will never use like that in reality, then it might become
>> >> > meaningless complicity introduced and error prone.
>> >> >
>> >> It _is_ necessary to write controller driver independent of client drivers.
>> >>
>> >
>> > Yes, that's true. What if we think we're writing driver against HW
>> > capabilities which support 4 pair of channel links(tx/rx)?
>> > It looks like independent of client drivers and may make life easily.
>> > Does it make sense?
>> >
>> No, that would be emulation.
>> Why doesn't a usb device controller (udc) driver emulate FSG/HID etc, by
>> "thinking" it has a hardware backed storage/keyboard? It doesn't because
>> that is the job of upper protocol layer.
>>
>
> Sorry I'm not quite familiar with USB device.
>
Which subsystem are you familiar with? Every stack is designed like
that -- controller drivers reflect the physical hardware, any
emulation/protocol is implemented in the user layer.


> Another reason is I doubt that we may never use per register mode in a different register pair
> in the future.
>
You may never use, but if the driver reflects the controller as
precisely as it is, linux could support any usecase that some
baremetal code could support on your platform.
For example, you could support "normal" and "scu" mode over the 3-cell
mechanism I suggested. But your original proposal/assumption of
"scu-mode", failed immediately, hence Oleksij had to imlpement the
"normal" mode.


> For example:
> AP:
> node {
>         ...
>         // cell 0 meaning: 0: tx 1: rx  cell1 meaning: channel id
>         Mboxes = <&mbox 0 1 &mbox 1 2>
>         Mbox-names = "tx", "rx";
>>
>
> M4:
> node {
>         ...
>         Mboxes = <&mbox 0 2 &mbox 1 1>
>         Mbox-names = "tx", "rx";
>>
> This make things complicated and error prone as I said before.
>
I don't see how.


> But that's just my understanding and may overlook something, if you still think
> we should do exactly as above, I will not against it because it does work for M4 case.
>
Cool, done.


> Then the left question is how we handle SCU case?
>
Please point me to the code that you worry might not work.


>
> I'm a bit confusing....
> The section 3.6 you pointed is the MHU register description. It does not conflict
> with what I see from ARM doc center that each physical channel is unidirectional.
>
> See below:
> Chan 1:
> 0x000 SCP_INTR_L_STAT
> 0x008 SCP_INTR_L_SET
> 0x010 SCP _INTR_L_CLEAR
>
> Chan 2:
> 0x020 SCP_INTR_H_STAT
> 0x028 SCP_INTR_H_SET
> 0x030 SCP _INTR_H_CLEAR
>
> Chan 3:
> 0x100 CPU_INTR_L_STAT
> 0x108 CPU_INTR_L_SET
> 0x110 CPU_INTR_L_CLEAR
>
> Chan 4:
> 0x120 CPU_INTR_H_STAT
> 0x128 CPU_INTR_H_SET
> 0x130 CPU_INTR_H_CLEAR
>
> Chan 5:
> 0x200 SCP_INTR_S_STAT
> 0x208 SCP_INTR_S_SET
> 0x210 SCP _INTR_S_CLEAR
>
> Chan 6:
> 0x300 CPU_INTR_S_STAT
> 0x308 CPU_INTR_S_SET
> 0x310 CPU_INTR_S_CLEAR
>
> And the driver compose them into 3 channel links (lp, hp and sec).
> Am I wrong?
>
The RX and TX are not independent of each other, their priorities are
tied together in hardware. So it makes more sense (as compared to
declaring them independent) to club them together as one channel.
Whatever example you take - MHU or TI - you won't find a driver that
implements a virtual mode like "scu-mode".


>> >
>> > Sorry for may not clear, "Passing short message' usecase is to tell
>> > how the HW is working on one channel mode sending up to 4 words in one
>> > time As specified in reference manual.
>> >
>> > SCU does work that way, the only difference is it's using polling mode
>> > rather than interrupt driven.  The point is the data size may be
>> > different for each msg, so we can't simply know which data register
>> > interrupt should be enable from static data defined in device tree.
>> >
>> And you think passing variable data through registers is a better idea than
>> passing packets via shared-memory?
>>
>
> You got me. :-)
> I've no idea about which one is better.
>
Passing data directly via controller makes sense only when your
protocol defines fixed length packets that fit into registers (usually
FIFOs) or there is no shared memory between the processors.
Otherwise, you write a data packet at some located in shmem, and
provide its start and size along with the code of expected operation
on it, over the mailbox. Please refer to "Passing frame information"
para of page-2743 of your manual.
That is another reason the SCU implementation is broken - it has
variable length packets and yet use registers to pass them around.


> The problem is SCU firmware is already
> there passing packets through data registers, we have no way to change it.
>
OK. But what is SCU exactly? Part of ATF? Do you get some binary from
third party? If the last, you shouldn't be paying hsit for such a
design.


>>
>> The hardware has 8 unidirectional channels. But your protocol (SCU
>> implementation) assumes there is one _virtual_ channel that has 4 registers
>> and 1/0 irq --- which is not true. Instead of fixing the assumption in your
>> protocol or emulating the virtual channel in the protocol level (user of a MU),
>> you want to add code in the controller driver that ignores interrupts and club
>> the 4 independent channels together.
>>
>
> This stucks me. This is how the hardware is designed  and suggested to use
> in hardware reference manual. And now you're telling me this is wrong and
> we should not use the design in reference manual...
>
What you call "suggested to use in hardware reference manual" is just
1/5 examples of usecase.
And SCU is not even doing that correctly (it uses variable length
packet, unlike the fixed 4-words suggestion).

A manual can suggest multiple ways of implementing a usecase. It is
our job to chose the best one.

Oleksij Rempel July 30, 2018, 7:35 a.m. UTC | #17

Hi Aishen, Jassie,

i'm lost in this discussion. Please, as soon as I need to add some
changes to my patches, notify me. Preferably on top for email.

On 28.07.2018 15:09, Jassi Brar wrote:
> On Fri, Jul 27, 2018 at 2:12 PM, A.s. Dong <[e-mail]> wrote:
> 
> 
>>>>>>>
>>>>>>> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
>>>>>>> (MU also has 4 "doorbell" type channels that it calls GP, but
>>>>>>> those are not managed here, so lets not worry atm).
>>>>>>>
>>>>>>> By definition, a mailbox channel is simply a signal, optionally
>>>>>>> with data attached. So, MU has 4 TX and 4 RX channels.
>>>>>>>
>>>>>>> The MU driver should populate 8 unidirectional (4 Tx and 4 RX)
>>>>>>> channels and set each tx/rx operation to trigger the corresponding
>>>>>>> interrupt. This is not my whim, this is how the controller is!
>>>>>>>
>>>>>>
>>>>>> This looks like reasonable to me, theoretically.
>>>>>> Just not sure whether it's necessary to support it because we
>>>>>> probably will never use like that in reality, then it might become
>>>>>> meaningless complicity introduced and error prone.
>>>>>>
>>>>> It _is_ necessary to write controller driver independent of client drivers.
>>>>>
>>>>
>>>> Yes, that's true. What if we think we're writing driver against HW
>>>> capabilities which support 4 pair of channel links(tx/rx)?
>>>> It looks like independent of client drivers and may make life easily.
>>>> Does it make sense?
>>>>
>>> No, that would be emulation.
>>> Why doesn't a usb device controller (udc) driver emulate FSG/HID etc, by
>>> "thinking" it has a hardware backed storage/keyboard? It doesn't because
>>> that is the job of upper protocol layer.
>>>
>>
>> Sorry I'm not quite familiar with USB device.
>>
> Which subsystem are you familiar with? Every stack is designed like
> that -- controller drivers reflect the physical hardware, any
> emulation/protocol is implemented in the user layer.
> 
> 
>> Another reason is I doubt that we may never use per register mode in a different register pair
>> in the future.
>>
> You may never use, but if the driver reflects the controller as
> precisely as it is, linux could support any usecase that some
> baremetal code could support on your platform.
> For example, you could support "normal" and "scu" mode over the 3-cell
> mechanism I suggested. But your original proposal/assumption of
> "scu-mode", failed immediately, hence Oleksij had to imlpement the
> "normal" mode.
> 
> 
>> For example:
>> AP:
>> node {
>>         ...
>>         // cell 0 meaning: 0: tx 1: rx  cell1 meaning: channel id
>>         Mboxes = <&mbox 0 1 &mbox 1 2>
>>         Mbox-names = "tx", "rx";
>>>
>>
>> M4:
>> node {
>>         ...
>>         Mboxes = <&mbox 0 2 &mbox 1 1>
>>         Mbox-names = "tx", "rx";
>>>
>> This make things complicated and error prone as I said before.
>>
> I don't see how.
> 
> 
>> But that's just my understanding and may overlook something, if you still think
>> we should do exactly as above, I will not against it because it does work for M4 case.
>>
> Cool, done.
> 
> 
>> Then the left question is how we handle SCU case?
>>
> Please point me to the code that you worry might not work.
> 
> 
>>
>> I'm a bit confusing....
>> The section 3.6 you pointed is the MHU register description. It does not conflict
>> with what I see from ARM doc center that each physical channel is unidirectional.
>>
>> See below:
>> Chan 1:
>> 0x000 SCP_INTR_L_STAT
>> 0x008 SCP_INTR_L_SET
>> 0x010 SCP _INTR_L_CLEAR
>>
>> Chan 2:
>> 0x020 SCP_INTR_H_STAT
>> 0x028 SCP_INTR_H_SET
>> 0x030 SCP _INTR_H_CLEAR
>>
>> Chan 3:
>> 0x100 CPU_INTR_L_STAT
>> 0x108 CPU_INTR_L_SET
>> 0x110 CPU_INTR_L_CLEAR
>>
>> Chan 4:
>> 0x120 CPU_INTR_H_STAT
>> 0x128 CPU_INTR_H_SET
>> 0x130 CPU_INTR_H_CLEAR
>>
>> Chan 5:
>> 0x200 SCP_INTR_S_STAT
>> 0x208 SCP_INTR_S_SET
>> 0x210 SCP _INTR_S_CLEAR
>>
>> Chan 6:
>> 0x300 CPU_INTR_S_STAT
>> 0x308 CPU_INTR_S_SET
>> 0x310 CPU_INTR_S_CLEAR
>>
>> And the driver compose them into 3 channel links (lp, hp and sec).
>> Am I wrong?
>>
> The RX and TX are not independent of each other, their priorities are
> tied together in hardware. So it makes more sense (as compared to
> declaring them independent) to club them together as one channel.
> Whatever example you take - MHU or TI - you won't find a driver that
> implements a virtual mode like "scu-mode".
> 
> 
>>>>
>>>> Sorry for may not clear, "Passing short message' usecase is to tell
>>>> how the HW is working on one channel mode sending up to 4 words in one
>>>> time As specified in reference manual.
>>>>
>>>> SCU does work that way, the only difference is it's using polling mode
>>>> rather than interrupt driven.  The point is the data size may be
>>>> different for each msg, so we can't simply know which data register
>>>> interrupt should be enable from static data defined in device tree.
>>>>
>>> And you think passing variable data through registers is a better idea than
>>> passing packets via shared-memory?
>>>
>>
>> You got me. :-)
>> I've no idea about which one is better.
>>
> Passing data directly via controller makes sense only when your
> protocol defines fixed length packets that fit into registers (usually
> FIFOs) or there is no shared memory between the processors.
> Otherwise, you write a data packet at some located in shmem, and
> provide its start and size along with the code of expected operation
> on it, over the mailbox. Please refer to "Passing frame information"
> para of page-2743 of your manual.
> That is another reason the SCU implementation is broken - it has
> variable length packets and yet use registers to pass them around.
> 
> 
>> The problem is SCU firmware is already
>> there passing packets through data registers, we have no way to change it.
>>
> OK. But what is SCU exactly? Part of ATF? Do you get some binary from
> third party? If the last, you shouldn't be paying hsit for such a
> design.
> 
> 
>>>
>>> The hardware has 8 unidirectional channels. But your protocol (SCU
>>> implementation) assumes there is one _virtual_ channel that has 4 registers
>>> and 1/0 irq --- which is not true. Instead of fixing the assumption in your
>>> protocol or emulating the virtual channel in the protocol level (user of a MU),
>>> you want to add code in the controller driver that ignores interrupts and club
>>> the 4 independent channels together.
>>>
>>
>> This stucks me. This is how the hardware is designed  and suggested to use
>> in hardware reference manual. And now you're telling me this is wrong and
>> we should not use the design in reference manual...
>>
> What you call "suggested to use in hardware reference manual" is just
> 1/5 examples of usecase.
> And SCU is not even doing that correctly (it uses variable length
> packet, unlike the fixed 4-words suggestion).
> 
> A manual can suggest multiple ways of implementing a usecase. It is
> our job to chose the best one.
>

Aisheng Dong July 30, 2018, 8:29 a.m. UTC | #18

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Saturday, July 28, 2018 9:10 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; , [e-mail],
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 

[...]

> >> >
> >> > Sorry for may not clear, "Passing short message' usecase is to tell
> >> > how the HW is working on one channel mode sending up to 4 words in
> >> > one time As specified in reference manual.
> >> >
> >> > SCU does work that way, the only difference is it's using polling
> >> > mode rather than interrupt driven.  The point is the data size may
> >> > be different for each msg, so we can't simply know which data
> >> > register interrupt should be enable from static data defined in device
> tree.
> >> >
> >> And you think passing variable data through registers is a better
> >> idea than passing packets via shared-memory?
> >>
> >
> > You got me. :-)
> > I've no idea about which one is better.
> >
> Passing data directly via controller makes sense only when your protocol
> defines fixed length packets that fit into registers (usually
> FIFOs) or there is no shared memory between the processors.

Don't have to be fixed length packets. Every data packets less than 4 words
can be transferred in one time by controller perfectly well as it has 4 data registers.
No shared memory need.

> Otherwise, you write a data packet at some located in shmem, and provide
> its start and size along with the code of expected operation on it, over the
> mailbox. Please refer to "Passing frame information"
> para of page-2743 of your manual.
> That is another reason the SCU implementation is broken - it has variable
> length packets and yet use registers to pass them around.
> 

" Passing frame information" is used for long message transfer via shared memory.
But almost all the SCU data frames are less than 4 words. (only quite a few of them,
2~3, over 4 words) For these types of messages, it's obviously more efficient to transfer
them in "Passing short message' way as the MU hardware supports well.

So I believe our SCU firmware has already done the best choice in a most efficient
way (register write is obviously faster than memory copies). Thus, no shared memory
need to be used for SCU protocol case.

> 
> > The problem is SCU firmware is already there passing packets through
> > data registers, we have no way to change it.
> >
> OK. But what is SCU exactly? Part of ATF? Do you get some binary from third
> party? If the last, you shouldn't be paying hsit for such a design.
> 

The System Controller Firmware (SCFW) is a low-level system function
which runs on a dedicated Cortex-M core to provide power, clock, and
resource management. It exists on some i.MX8 processors. e.g. i.MX8QM
(QM, QP), and i.MX8QX (QXP, DX). 

It's developed by NXP internally.

> 
> >>
> >> The hardware has 8 unidirectional channels. But your protocol (SCU
> >> implementation) assumes there is one _virtual_ channel that has 4
> >> registers and 1/0 irq --- which is not true. Instead of fixing the
> >> assumption in your protocol or emulating the virtual channel in the
> >> protocol level (user of a MU), you want to add code in the controller
> >> driver that ignores interrupts and club the 4 independent channels
> together.
> >>
> >
> > This stucks me. This is how the hardware is designed  and suggested to
> > use in hardware reference manual. And now you're telling me this is
> > wrong and we should not use the design in reference manual...
> >
> What you call "suggested to use in hardware reference manual" is just
> 1/5 examples of usecase.
> And SCU is not even doing that correctly (it uses variable length packet,
> unlike the fixed 4-words suggestion).

SCU is doing well. Please see my above reply.

> 
> A manual can suggest multiple ways of implementing a usecase. It is our job
> to chose the best one.

Well, great. That is also our orientation.
Because we thought the way " Passing short message " is the most suitable way for
SCU and is the most efficient way as the most SCU Messages are less than 4 words
and MU hardware supports transfer them well. We could not think out a reason to
not use it. That's why we choose that.

A simple objection that it uses variable length packet seems not quite convincible.
Unless we have more convincible reasons...

How do you think?

For example, what's the real benefits we can get if we drop the HW capability of
"passing Short messages" for SCU in one channel way and use separate 8 Tx/Rx channel
Instead to send them each time? What about the Performance downgrading? 
Is it valuable comparing to the complexity introduced?

Regards
Dong Aisheng

Aisheng Dong July 30, 2018, 8:39 a.m. UTC | #19

Hi Oleksij & Sascha,

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Thursday, July 26, 2018 11:44 PM
> To: A.s. Dong <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; Shawn Guo
> <[e-mail]>; Fabio Estevam <[e-mail]>; Rob
> Herring <[e-mail]>; Mark Rutland <[e-mail]>;
> Vladimir Zapolskiy <[e-mail]>; , Sascha Hauer
> <[e-mail]>; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Thu, Jul 26, 2018 at 5:21 PM, A.s. Dong <[e-mail]> wrote:
> 
> >>
> >> > Each MU has four pairs of rx/tx data register with four rx/tx
> >> > interrupts which can also be used as a separate channel.
> >> >
> >> So the hardware actually supports 4 channels.
> >>
> >> > -- #mbox-cells:  Must be 0. Number of cells in a mailbox
> >> > +- #mbox-cells:  Must be:
> >> > +               0 - for single channel mode. i.MX8* SCU protocol specific.
> >> > +               1 - for multichannel (generic) mode.
> >> > +
> >> No, please.
> >> DT bindings should reflect the real hardware, and not the software
> >> mode we want the driver to work in.
> >> Please define mbox-cells=1  and have the i.MX8* platform always ask
> >> for channel-0.
> >
> > The reality is that MU hardware does not define channels in reference
> manual.
> > However, it does have four separate data tx/rx register which can be
> > used as 'virtual' channels which is supported by this current driver.
> >
> > See below HW description from the reference manual:
> > For messaging, the MU has four, 32-bit write-only transmit registers
> > and four, 32-bit read-only receive registers on the Processor B and
> > Processor A-sides. These registers are used for sending messages to each
> other.
> >
> For a while please forget the protocol(user) level usage, and consider only
> what your h/w is.
> 
> MU has 4 pairs of TX_Reg + TX_IRQ, and 4 pairs of RX_Reg + RX_IRQ.
> (MU also has 4 "doorbell" type channels that it calls GP, but those are not
> managed here, so lets not worry atm).
> 
> By definition, a mailbox channel is simply a signal, optionally with data
> attached. So, MU has 4 TX and 4 RX channels.
> 
> The MU driver should populate 8 unidirectional (4 Tx and 4 RX) channels and
> set each tx/rx operation to trigger the corresponding interrupt. This is not my
> whim, this is how the controller is!
> 
> The SCU is poorly implemented as it ignores 3 irqs and club all 4 register
> together (there are many other cons of this approach but lets not get into
> that).
> Personally, I would push-back on such a bad design. But if you claim you have
> no choice but to support SCU as such, the work around could be simpler than
> defining a new "scu mode" altogether.
> 
> #mbox-cells:  Must be 3.
>                       First cell is 1 for TX and 0 for RX channel
>                       Second cell is index of the channel [0,1,2 or 3]
>                       Third cell is 1 if the channel triggers an IRQ,
> 0 if not. That is ACR.RIE/TIE bits are set or not.
> 
> Normal clients would always request a channel with irqs enabled.
> The SCU client would request all 8 channels -- TX/RX[0,1,2] with irqs disabled,
> TX/RX[3] with irqs enabled. And SCU will read/write 4 word data over 4 rx/tx
> channels, instead of the virtually concocted one.
> 

Sascha & Oleksij,
Do you have any comments on this new approach suggested by Jassi?
This looks like may work for M4 case.

But I'm still discussing with Jassi on whether it's suitable for SCU as well?
As that means we drop the MU Hardware capability to send multi words
in one time but send one word on each channel one time which seems like
less efficiency and more complexity introduced for SCU case.

Regards
Dong Aisheng

> 
> > short messages
> > Transmit register(s) can be used to pass short messages from one to
> > four words in length. For example, when a four-word message is
> > desired, only one of the registers needs to have its corresponding
> > interrupt enable bit set at the receiver side; the message’s first
> > three words are written to the registers whose interrupt is masked,
> > and the fourth word is written to the other register (which triggers an
> interrupt at the receiver side).
> >
> > The reference manual is at here: (Chapter 42 Messaging Unit (MU)
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> ww
> > .nxp.com%2Fdocs%2Fen%2Freference-
> manual%2FIMX6ULRM.pdf&amp;data=02%7C0
> >
> 1%7Caisheng.dong%40nxp.com%7Cef349d7c9caf46c4260008d5f30e9ef5%7C6
> 86ea1
> >
> d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636682166494860332&amp;sdat
> a=54rE
> >
> iDm%2BGD6EY8NE64ck1LXVGoUCtWYrHXPoWqcujUo%3D&amp;reserved=0
> >
> > And SCU firmware does use MU in above way specified by reference
> manual.
> > Even from HW point of view, it's still one channel only mailbox.
> >
> Please realise that any manual is written by a mere mortal afterall.
> How the controller works is set in stone, but how the controller can be
> used ... is just someones suggestion.
> 
> The approach I suggest above, conforms to the api and prevents a provider
> dancing to the tunes of a user.

Aisheng Dong July 30, 2018, 8:42 a.m. UTC | #20

Hi Oleksij,

> -----Original Message-----
> From: Oleksij Rempel [mailto:[e-mail]]
> Sent: Monday, July 30, 2018 3:36 PM
> To: Jassi Brar <[e-mail]>; A.s. Dong
> <[e-mail]>
> Cc: , Sascha Hauer <[e-mail]>; Shawn Guo
> <[e-mail]>; Fabio Estevam <[e-mail]>; Rob
> Herring <[e-mail]>; Mark Rutland <[e-mail]>;
> Vladimir Zapolskiy <[e-mail]>; ", linux-arm-
> kernel"@lists.infradead.org; [e-mail];
> srv_heupstream <[e-mail]>; Devicetree List
> <[e-mail]>; dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> Hi Aishen, Jassie,
> 
> i'm lost in this discussion. Please, as soon as I need to add some changes to
> my patches, notify me. Preferably on top for email.
> 

I just send you an email of Jassie's early suggestion.

According to the suggestion, the binding needs to be changed into 8 separate
TX/RX channels. You can quickly start from there.

Regards
Dong Aisheng

Jassi Brar July 30, 2018, 1:04 p.m. UTC | #21

On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]> wrote:
> Hi Aishen, Jassie,
>
> i'm lost in this discussion. Please, as soon as I need to add some
> changes to my patches, notify me. Preferably on top for email.
>
I am ok with however you choose to implement, 8 unidirectional or 4
bidirectional channels or whatever.

We just can't have protocol specific s/w modes in the controller drivers.

The best solution is to fix the SCU firmware. If that is _really_
impossible, I provided a solution (3 cells work around). If you have a
better idea please feel free to propose and implement that.

It will also help if you could share the user code of "scu-mode". If
there is no such code (and we know the driver doesn't respect the
"scu-mode" property) why do we even have that binding? Maybe drop it.

thnx.

Aisheng Dong July 30, 2018, 2:14 p.m. UTC | #22

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Monday, July 30, 2018 9:05 PM
> To: Oleksij Rempel <[e-mail]>
> Cc: A.s. Dong <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; ", linux-arm-
> kernel"@lists.infradead.org; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-
> [e-mail]>; srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]>
> wrote:
> > Hi Aishen, Jassie,
> >
> > i'm lost in this discussion. Please, as soon as I need to add some
> > changes to my patches, notify me. Preferably on top for email.
> >
> I am ok with however you choose to implement, 8 unidirectional or 4
> bidirectional channels or whatever.
> 
> We just can't have protocol specific s/w modes in the controller drivers.
> 

Do you think implement "Passing short messages" defined in reference
manual in controller driver is protocol specific?
That seems like our most argument before.

> The best solution is to fix the SCU firmware. If that is _really_ impossible, I
> provided a solution (3 cells work around). If you have a better idea please
> feel free to propose and implement that.
> 

That's out of our control.
SCU firmware is not using Linux and there's no mailbox framework.
It's simply follow "passing short messages" way to send multi words with 4 data
Registers.
 
> It will also help if you could share the user code of "scu-mode". If there is no
> such code (and we know the driver doesn't respect the "scu-mode" property)
> why do we even have that binding? Maybe drop it.

You can refer to this patch for the user code of "scu-mode".
[RFC PATCH v6 2/2] mailbox: imx-mailbox: add scu protocol support
https://www.spinics.net/lists/arm-kernel/msg665135.html

And I do admit that there're two protocol specific bits in the driver.
1. Unlike TI message header, SCU size is in the second u8.
Ti message header:
struct ti_msgmgr_message {
        size_t len;
        u8 *buf;
};
Controller get the size from the first u32.

SCU message header:
struct sc_rpc_msg {
        uint8_t ver;
        uint8_t size;
        uint8_t svc; 
        uint8_t func;
};
Controller get the size from the second u8.

And a full SCU message is like:
struct imx_sc_msg_req_set_clock_rate {
        struct sc_rpc_msg hdr;
        u32 rate;
        u16 resource;
        u8 clk;
} __packed;

2. There's no synchronous read function in mailbox framework,
So we use mbox_client_peek_data to read data which can't return
the read data. In order to save a mem copy from read data to the
data buffer, the controller driver saves the tx data buffer in itself. 
(Anyway to improve? Maybe we should implement a synchronous
mbox_client_read_data()?)

And for SCU IPC client, it's something like:
int sc_call_rpc(sc_ipc_t ipc, void *msg, bool no_resp)
{
        ...
        mbox_send_message(sc_ipc->chan, msg);

        if (!no_resp) 
                ret = mbox_client_peek_data(sc_ipc->chan);

        mbox_client_txdone(sc_ipc->chan, ret);
        ...
}

int sc_ipc_open(sc_ipc_t *ipc, struct device *dev)
{
        sc_ipc->cl.dev = dev; 
        sc_ipc->cl.tx_block = false;
        sc_ipc->cl.knows_txdone = true;

        sc_ipc->chan = mbox_request_channel(&sc_ipc->cl, 0);
        ...
}

Any better ideas?

Regards
Dong Aisheng

> 
> thnx.

Aisheng Dong July 30, 2018, 2:17 p.m. UTC | #23

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Monday, July 30, 2018 9:05 PM
> To: Oleksij Rempel <[e-mail]>
> Cc: A.s. Dong <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; ", linux-arm-
> kernel"@lists.infradead.org; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-
> [e-mail]>; srv_heupstream <linux-arm-
> [e-mail]>; Devicetree List <[e-mail]>;
> dl-linux-imx <[e-mail]>
> Subject: Re: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]>
> wrote:
> > Hi Aishen, Jassie,
> >
> > i'm lost in this discussion. Please, as soon as I need to add some
> > changes to my patches, notify me. Preferably on top for email.
> >
> I am ok with however you choose to implement, 8 unidirectional or 4
> bidirectional channels or whatever.
> 
> We just can't have protocol specific s/w modes in the controller drivers.
> 
> The best solution is to fix the SCU firmware. If that is _really_ impossible, I
> provided a solution (3 cells work around). If you have a better idea please
> feel free to propose and implement that.
> 
> It will also help if you could share the user code of "scu-mode". If there is no
> such code (and we know the driver doesn't respect the "scu-mode" property)
> why do we even have that binding? Maybe drop it.

The initial idea is using mbox-cells value to distinguish which MU mode is used
(single chan or multi chans).

Regards
Dong Aisheng

> 
> thnx.

Aisheng Dong July 30, 2018, 2:27 p.m. UTC | #24

[...]
> > It will also help if you could share the user code of "scu-mode". If
> > there is no such code (and we know the driver doesn't respect the
> > "scu-mode" property) why do we even have that binding? Maybe drop it.
> 
> You can refer to this patch for the user code of "scu-mode".
> [RFC PATCH v6 2/2] mailbox: imx-mailbox: add scu protocol support
> https://www.spinics.net/lists/arm-kernel/msg665135.html
> 
> And I do admit that there're two protocol specific bits in the driver.
> 1. Unlike TI message header, SCU size is in the second u8.
> Ti message header:
> struct ti_msgmgr_message {
>         size_t len;
>         u8 *buf;
> };
> Controller get the size from the first u32.
> 
> SCU message header:
> struct sc_rpc_msg {
>         uint8_t ver;
>         uint8_t size;
>         uint8_t svc;
>         uint8_t func;
> };
> Controller get the size from the second u8.
> 
> And a full SCU message is like:
> struct imx_sc_msg_req_set_clock_rate {
>         struct sc_rpc_msg hdr;
>         u32 rate;
>         u16 resource;
>         u8 clk;
> } __packed;
> 
> 2. There's no synchronous read function in mailbox framework, So we use
> mbox_client_peek_data to read data which can't return the read data. In
> order to save a mem copy from read data to the data buffer, the controller
> driver saves the tx data buffer in itself.
> (Anyway to improve? Maybe we should implement a synchronous
> mbox_client_read_data()?)
> 
> And for SCU IPC client, it's something like:
> int sc_call_rpc(sc_ipc_t ipc, void *msg, bool no_resp) {
>         ...
>         mbox_send_message(sc_ipc->chan, msg);
> 
>         if (!no_resp)
>                 ret = mbox_client_peek_data(sc_ipc->chan);
> 
>         mbox_client_txdone(sc_ipc->chan, ret);
>         ...
> }
> 
> int sc_ipc_open(sc_ipc_t *ipc, struct device *dev) {
>         sc_ipc->cl.dev = dev;
>         sc_ipc->cl.tx_block = false;
>         sc_ipc->cl.knows_txdone = true;
> 
>         sc_ipc->chan = mbox_request_channel(&sc_ipc->cl, 0);
>         ...
> }
> 
> Any better ideas?
> 

And this the original implementation used by NXP internally without mailbox.
As it's quite simple and suppose more easy to help you understand the using of SCU.

void mu_send_msg(struct mu_priv *priv, uint32_t index, uint32_t msg)
{
        ...
        /* Wait TX register to be empty. */ 
        while (!(readl_relaxed(priv->base + MU_ASR) & mask))
                ;
        writel_relaxed(msg, priv->base + MU_ATR0  + (index * 4));
}

/*
 * Wait to receive message from the other core.
 */
void mu_receive_msg(struct mu_priv *priv, uint32_t index, uint32_t *msg)
{
        ...
        /* Wait RX register to be full. */
        while (!(readl_relaxed(priv->base + MU_ASR) & mask))
                ;
        *msg = readl_relaxed(priv->base + MU_ARR0 + (index * 4));
}

static void sc_ipc_read(struct sc_ipc *sc_ipc, void *data)
{
        sc_rpc_msg_t *msg = (sc_rpc_msg_t *)data;
 
        /* Read first word */
        mu_receive_msg(sc_ipc->mu, 0, (uint32_t *)msg);
        count++;
 
        /* Read remaining words */
        while (count < msg->size) {
                mu_receive_msg(sc_ipc->mu, count % MU_RR_COUNT,
                               &(msg->DATA.u32[count - 1]));
                count++;
        }
}

static void sc_ipc_write(struct sc_ipc *sc_ipc, void *data)
{
        sc_rpc_msg_t *msg = (sc_rpc_msg_t *)data;
        uint8_t count = 0;

        /* Write first word */
        mu_send_msg(sc_ipc->mu, 0, *((uint32_t *)msg));
        count++;

        while (count < msg->size) {
                mu_send_msg(sc_ipc->mu, count % MU_TR_COUNT,
                            msg->DATA.u32[count - 1]);
                count++;
        }
}

void sc_call_rpc(sc_ipc_t ipc, sc_rpc_msg_t *msg, bool no_resp)
{
        sc_ipc_write(sc_ipc, msg);
        if (!no_resp)
                sc_ipc_read(sc_ipc, msg);
}

For details, see:
https://www.spinics.net/lists/arm-kernel/msg659138.html

Regards
Dong Aisheng

> Regards
> Dong Aisheng
> 
> >
> > thnx.

Oleksij Rempel July 30, 2018, 2:44 p.m. UTC | #25

On Mon, Jul 30, 2018 at 06:34:56PM +0530, Jassi Brar wrote:
> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]> wrote:
> > Hi Aishen, Jassie,
> >
> > i'm lost in this discussion. Please, as soon as I need to add some
> > changes to my patches, notify me. Preferably on top for email.
> >
> I am ok with however you choose to implement, 8 unidirectional or 4
> bidirectional channels or whatever.
> 
> We just can't have protocol specific s/w modes in the controller drivers.
> 
> The best solution is to fix the SCU firmware. If that is _really_
> impossible, I provided a solution (3 cells work around). If you have a
> better idea please feel free to propose and implement that.
> 
> It will also help if you could share the user code of "scu-mode". If
> there is no such code (and we know the driver doesn't respect the
> "scu-mode" property) why do we even have that binding? Maybe drop it.

Tomorrow I have a time slot to address your generic iMX MU suggestions.
So, what is better, uni- or bi-directional channels? Should I implement
*all* (4TX+FIFO, 4RX+FIFO, 4TX-simple, 4RX-simple) channels in this run?
If yes, how should it be reflected in DT?
- 1 cell: &mu [0-15]
- 2 cells: &mu [0-7] [TX,RX]
- 3 cells: &mu [0-4] [FIFO,NOFIFO] [TX,RX]

Jassi Brar July 30, 2018, 3:02 p.m. UTC | #26

On Mon, Jul 30, 2018 at 8:14 PM, Oleksij Rempel <[e-mail]> wrote:
> On Mon, Jul 30, 2018 at 06:34:56PM +0530, Jassi Brar wrote:
>> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]> wrote:
>> > Hi Aishen, Jassie,
>> >
>> > i'm lost in this discussion. Please, as soon as I need to add some
>> > changes to my patches, notify me. Preferably on top for email.
>> >
>> I am ok with however you choose to implement, 8 unidirectional or 4
>> bidirectional channels or whatever.
>>
>> We just can't have protocol specific s/w modes in the controller drivers.
>>
>> The best solution is to fix the SCU firmware. If that is _really_
>> impossible, I provided a solution (3 cells work around). If you have a
>> better idea please feel free to propose and implement that.
>>
>> It will also help if you could share the user code of "scu-mode". If
>> there is no such code (and we know the driver doesn't respect the
>> "scu-mode" property) why do we even have that binding? Maybe drop it.
>
> Tomorrow I have a time slot to address your generic iMX MU suggestions.
> So, what is better, uni- or bi-directional channels?
>
The datasheet indicates there are 4 tx and 4 rx channels. So 8
uni-directional channels (which allow more fine-grained/efficient
resource allocation btw).

> Should I implement
> *all* (4TX+FIFO, 4RX+FIFO, 4TX-simple, 4RX-simple) channels in this run?
>
From datasheet, each of 8 channels should be defined as signal+data
i.e, IRQ + TX/RX_Reg.
The rest 4 GP channels are doorbells (irq only).

So we can have 2-cells.
    First cell is 0->Tx, 1->RX, 2->Doorbell
    Second cell is index of the channel {0,3}

Now you may implement only RX+TX, and leave 'doorbell' out for future.
Thats ok, because we wouldn't have to change bindings then.

However, if SCU (in its current form) must be supported. We may need
to add the third cell (irq enable or not) or some better way, right
now.

> If yes, how should it be reflected in DT?
> - 1 cell: &mu [0-15]
> - 2 cells: &mu [0-7] [TX,RX]
> - 3 cells: &mu [0-4] [FIFO,NOFIFO] [TX,RX]
>
It may be possible for a device binding to have variable cells, but I
haven't seen any. And I am not sure that would look neat either.

Aisheng Dong July 30, 2018, 3:36 p.m. UTC | #27

Hi Jassi,

> > If yes, how should it be reflected in DT?
> > - 1 cell: &mu [0-15]
> > - 2 cells: &mu [0-7] [TX,RX]
> > - 3 cells: &mu [0-4] [FIFO,NOFIFO] [TX,RX]
> >
> It may be possible for a device binding to have variable cells, but I haven't
> seen any. And I am not sure that would look neat either.

Please refer to 
Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt
- #interrupt-cells : Specifies the number of cells needed to encode an
  interrupt source. Must be a single cell with a value of at least 3.
  If the system requires describing PPI affinity, then the value must
  be at least 4.

Regards
Dong Aisheng

Jassi Brar July 30, 2018, 4:18 p.m. UTC | #28

On Mon, Jul 30, 2018 at 8:32 PM, Jassi Brar <[e-mail]> wrote:
> On Mon, Jul 30, 2018 at 8:14 PM, Oleksij Rempel <[e-mail]> wrote:
>> On Mon, Jul 30, 2018 at 06:34:56PM +0530, Jassi Brar wrote:
>>> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]> wrote:
>>> > Hi Aishen, Jassie,
>>> >
>>> > i'm lost in this discussion. Please, as soon as I need to add some
>>> > changes to my patches, notify me. Preferably on top for email.
>>> >
>>> I am ok with however you choose to implement, 8 unidirectional or 4
>>> bidirectional channels or whatever.
>>>
>>> We just can't have protocol specific s/w modes in the controller drivers.
>>>
>>> The best solution is to fix the SCU firmware. If that is _really_
>>> impossible, I provided a solution (3 cells work around). If you have a
>>> better idea please feel free to propose and implement that.
>>>
>>> It will also help if you could share the user code of "scu-mode". If
>>> there is no such code (and we know the driver doesn't respect the
>>> "scu-mode" property) why do we even have that binding? Maybe drop it.
>>
>> Tomorrow I have a time slot to address your generic iMX MU suggestions.
>> So, what is better, uni- or bi-directional channels?
>>
> The datasheet indicates there are 4 tx and 4 rx channels. So 8
> uni-directional channels (which allow more fine-grained/efficient
> resource allocation btw).
>
>> Should I implement
>> *all* (4TX+FIFO, 4RX+FIFO, 4TX-simple, 4RX-simple) channels in this run?
>>
> From datasheet, each of 8 channels should be defined as signal+data
> i.e, IRQ + TX/RX_Reg.
> The rest 4 GP channels are doorbells (irq only).
>
> So we can have 2-cells.
>     First cell is 0->Tx, 1->RX, 2->Doorbell
>     Second cell is index of the channel {0,3}
>
> Now you may implement only RX+TX, and leave 'doorbell' out for future.
> Thats ok, because we wouldn't have to change bindings then.
>
> However, if SCU (in its current form) must be supported. We may need
> to add the third cell (irq enable or not) or some better way, right
> now.
>
Looking at imx_mu_scu_send_data(), which simply polls on the tx, I
think we don't even need third cell for scu client. A simple 2-cell, 8
uni-dir channel setup should work.
If I see the scu client driver, I could confirm how it would work.

So, lets please do the 8 uni-directional, 2 cells implementation.

Regards.

Oleksij Rempel July 30, 2018, 4:49 p.m. UTC | #29

On Mon, Jul 30, 2018 at 09:48:35PM +0530, Jassi Brar wrote:
> On Mon, Jul 30, 2018 at 8:32 PM, Jassi Brar <[e-mail]> wrote:
> > On Mon, Jul 30, 2018 at 8:14 PM, Oleksij Rempel <[e-mail]> wrote:
> >> On Mon, Jul 30, 2018 at 06:34:56PM +0530, Jassi Brar wrote:
> >>> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]> wrote:
> >>> > Hi Aishen, Jassie,
> >>> >
> >>> > i'm lost in this discussion. Please, as soon as I need to add some
> >>> > changes to my patches, notify me. Preferably on top for email.
> >>> >
> >>> I am ok with however you choose to implement, 8 unidirectional or 4
> >>> bidirectional channels or whatever.
> >>>
> >>> We just can't have protocol specific s/w modes in the controller drivers.
> >>>
> >>> The best solution is to fix the SCU firmware. If that is _really_
> >>> impossible, I provided a solution (3 cells work around). If you have a
> >>> better idea please feel free to propose and implement that.
> >>>
> >>> It will also help if you could share the user code of "scu-mode". If
> >>> there is no such code (and we know the driver doesn't respect the
> >>> "scu-mode" property) why do we even have that binding? Maybe drop it.
> >>
> >> Tomorrow I have a time slot to address your generic iMX MU suggestions.
> >> So, what is better, uni- or bi-directional channels?
> >>
> > The datasheet indicates there are 4 tx and 4 rx channels. So 8
> > uni-directional channels (which allow more fine-grained/efficient
> > resource allocation btw).
> >
> >> Should I implement
> >> *all* (4TX+FIFO, 4RX+FIFO, 4TX-simple, 4RX-simple) channels in this run?
> >>
> > From datasheet, each of 8 channels should be defined as signal+data
> > i.e, IRQ + TX/RX_Reg.
> > The rest 4 GP channels are doorbells (irq only).
> >
> > So we can have 2-cells.
> >     First cell is 0->Tx, 1->RX, 2->Doorbell
> >     Second cell is index of the channel {0,3}
> >
> > Now you may implement only RX+TX, and leave 'doorbell' out for future.
> > Thats ok, because we wouldn't have to change bindings then.
> >
> > However, if SCU (in its current form) must be supported. We may need
> > to add the third cell (irq enable or not) or some better way, right
> > now.
> >
> Looking at imx_mu_scu_send_data(), which simply polls on the tx, I
> think we don't even need third cell for scu client. A simple 2-cell, 8
> uni-dir channel setup should work.
> If I see the scu client driver, I could confirm how it would work.
> 
> So, lets please do the 8 uni-directional, 2 cells implementation.

Just in case.. to avoid confusion (or add more...). There are 4 GIP (General Interrupt Request n
Pending bits) or RX-doorbell and 4 GIR (General Interrupt Request bits)
or TX-doorbell.

Jassi Brar July 31, 2018, 2:51 a.m. UTC | #30

On Mon, Jul 30, 2018 at 10:19 PM, Oleksij Rempel
<[e-mail]> wrote:
> On Mon, Jul 30, 2018 at 09:48:35PM +0530, Jassi Brar wrote:
>> On Mon, Jul 30, 2018 at 8:32 PM, Jassi Brar <[e-mail]> wrote:
>> > On Mon, Jul 30, 2018 at 8:14 PM, Oleksij Rempel <[e-mail]> wrote:
>> >> On Mon, Jul 30, 2018 at 06:34:56PM +0530, Jassi Brar wrote:
>> >>> On Mon, Jul 30, 2018 at 1:05 PM, Oleksij Rempel <[e-mail]> wrote:
>> >>> > Hi Aishen, Jassie,
>> >>> >
>> >>> > i'm lost in this discussion. Please, as soon as I need to add some
>> >>> > changes to my patches, notify me. Preferably on top for email.
>> >>> >
>> >>> I am ok with however you choose to implement, 8 unidirectional or 4
>> >>> bidirectional channels or whatever.
>> >>>
>> >>> We just can't have protocol specific s/w modes in the controller drivers.
>> >>>
>> >>> The best solution is to fix the SCU firmware. If that is _really_
>> >>> impossible, I provided a solution (3 cells work around). If you have a
>> >>> better idea please feel free to propose and implement that.
>> >>>
>> >>> It will also help if you could share the user code of "scu-mode". If
>> >>> there is no such code (and we know the driver doesn't respect the
>> >>> "scu-mode" property) why do we even have that binding? Maybe drop it.
>> >>
>> >> Tomorrow I have a time slot to address your generic iMX MU suggestions.
>> >> So, what is better, uni- or bi-directional channels?
>> >>
>> > The datasheet indicates there are 4 tx and 4 rx channels. So 8
>> > uni-directional channels (which allow more fine-grained/efficient
>> > resource allocation btw).
>> >
>> >> Should I implement
>> >> *all* (4TX+FIFO, 4RX+FIFO, 4TX-simple, 4RX-simple) channels in this run?
>> >>
>> > From datasheet, each of 8 channels should be defined as signal+data
>> > i.e, IRQ + TX/RX_Reg.
>> > The rest 4 GP channels are doorbells (irq only).
>> >
>> > So we can have 2-cells.
>> >     First cell is 0->Tx, 1->RX, 2->Doorbell
>> >     Second cell is index of the channel {0,3}
>> >
>> > Now you may implement only RX+TX, and leave 'doorbell' out for future.
>> > Thats ok, because we wouldn't have to change bindings then.
>> >
>> > However, if SCU (in its current form) must be supported. We may need
>> > to add the third cell (irq enable or not) or some better way, right
>> > now.
>> >
>> Looking at imx_mu_scu_send_data(), which simply polls on the tx, I
>> think we don't even need third cell for scu client. A simple 2-cell, 8
>> uni-dir channel setup should work.
>> If I see the scu client driver, I could confirm how it would work.
>>
>> So, lets please do the 8 uni-directional, 2 cells implementation.
>
> Just in case.. to avoid confusion (or add more...). There are 4 GIP (General Interrupt Request n
> Pending bits) or RX-doorbell and 4 GIR (General Interrupt Request bits)
> or TX-doorbell.
>
Ok. But the approach remains the same.

thanks.

Aisheng Dong July 31, 2018, 7:21 a.m. UTC | #31

Hi Jassi,

> > However, if SCU (in its current form) must be supported. We may need
> > to add the third cell (irq enable or not) or some better way, right
> > now.
> >
> Looking at imx_mu_scu_send_data(), which simply polls on the tx, I think we
> don't even need third cell for scu client. A simple 2-cell, 8 uni-dir channel
> setup should work.

How would you suggest to use for SCU in 2 -cells?

> If I see the scu client driver, I could confirm how it would work.
> 

I've already given a general overview from here:
https://www.spinics.net/lists/arm-kernel/msg669202.html

Do you need me to send the full scu client driver patch for the reference?
If yes, please feel free to let me know.

Regards
Dong Aisheng

> So, lets please do the 8 uni-directional, 2 cells implementation.
> 
> Regards.

Jassi Brar July 31, 2018, 10:15 a.m. UTC | #32

On Tue, Jul 31, 2018 at 12:51 PM, A.s. Dong <[e-mail]> wrote:
> Hi Jassi,
>
>> > However, if SCU (in its current form) must be supported. We may need
>> > to add the third cell (irq enable or not) or some better way, right
>> > now.
>> >
>> Looking at imx_mu_scu_send_data(), which simply polls on the tx, I think we
>> don't even need third cell for scu client. A simple 2-cell, 8 uni-dir channel
>> setup should work.
>
> How would you suggest to use for SCU in 2 -cells?
>
>> If I see the scu client driver, I could confirm how it would work.
>>
>
> I've already given a general overview from here:
> https://www.spinics.net/lists/arm-kernel/msg669202.html
>
No, that is controller side code.

> Do you need me to send the full scu client driver patch for the reference?
> If yes, please feel free to let me know.
>
Yes, that'll make it clearer.

Jassi Brar July 31, 2018, 12:42 p.m. UTC | #33

On Tue, Jul 31, 2018 at 3:45 PM, Jassi Brar <[e-mail]> wrote:
> On Tue, Jul 31, 2018 at 12:51 PM, A.s. Dong <[e-mail]> wrote:
>> Hi Jassi,
>>
>>> > However, if SCU (in its current form) must be supported. We may need
>>> > to add the third cell (irq enable or not) or some better way, right
>>> > now.
>>> >
>>> Looking at imx_mu_scu_send_data(), which simply polls on the tx, I think we
>>> don't even need third cell for scu client. A simple 2-cell, 8 uni-dir channel
>>> setup should work.
>>
>> How would you suggest to use for SCU in 2 -cells?
>>
>>> If I see the scu client driver, I could confirm how it would work.
>>>
>>
>> I've already given a general overview from here:
>> https://www.spinics.net/lists/arm-kernel/msg669202.html
>>
> No, that is controller side code.
>
>> Do you need me to send the full scu client driver patch for the reference?
>> If yes, please feel free to let me know.
>>
> Yes, that'll make it clearer.
>
Thanks for sharing the patch offline.

I don't see why it can't be adapted to work with the simple 2-cell
implementation. Instead of one virtual, ask for 4 physical channels
and send data in u32 chunks sequentially over them.
In fact, that should make it even better by removing the tight-loop
(zero delay!!) polling in send_data() -- MU does support interrupts,
why not use them to avoid polling?  If you find it hard, I can make a
patch on top of your client patchset, once Oleksij is done with the MU
driver.

I have some more suggestions for the client driver. But since I don't
have to live with them (you do), I can't demand you make those
changes. If you like, please feel free to cc me on the next revision
of the client driver.

Aisheng Dong Aug. 2, 2018, 9:24 a.m. UTC | #34

Hi Jassi,

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Tuesday, July 31, 2018 8:42 PM
> To: A.s. Dong <[e-mail]>

[...]

> >> Do you need me to send the full scu client driver patch for the reference?
> >> If yes, please feel free to let me know.
> >>
> > Yes, that'll make it clearer.
> >
> Thanks for sharing the patch offline.
> 
> I don't see why it can't be adapted to work with the simple 2-cell
> implementation. Instead of one virtual, ask for 4 physical channels and send
> data in u32 chunks sequentially over them.
> In fact, that should make it even better by removing the tight-loop (zero
> delay!!) polling in send_data() -- MU does support interrupts, why not use
> them to avoid polling?  If you find it hard, I can make a patch on top of your
> client patchset, once Oleksij is done with the MU driver.
> 

As you insist, I will try to implement multi channels mode, then we can do a
clear comparison to see which one is better for SCU mode rather than assumptions.

First of all,  using interrupt for each chan will likely to terribly slow down the SCU
Message handling speed. SCU Messages usually are handled by SCU firmware very
quickly in a few microseconds. (< 10us). That's why we use polling mode for SCU
message handling.

If we break down a SCU message into a few separate MU channel interrupts driven,
There will be a lot of unnecessary and meaningless interrupt handling.
And the whole flow may look like:

Sending Word 0 ->
Chan 0 interrupt -> (meaningless)
Sending Word 1 ->
Chan1 interrupt -> (meaningless)
Sending Word 2 ->
Chan2 interrupt -> (meaningless)
Sending Word 3 ->
Chan3 interrupt-> (meaningless)

Sending Word 4 on Chan 0 again ->
Wait for Chan 0 interrupt (meaningless) ->
...

Then waiting for Rx:

Chan 0 Rx interrupt ->
Read Word 0   ->
(Then we know response size here)

Chan 1 Rx interrupt ->
Read Word 1 ->
Chan 2 Rx interrupt ->
Read Word 2 ->
Chan 3 Rx interrupt ->
Read Word 3 ->

(If msg size > 4 words)
Chan 0 Rx interrupt ->
Read Word 4

...
And we must use a complicated way to map the received data of each channel
into one SCU msg buffer and need handle the sequence properly for size > 4
words case

Is this what you intended?

Per my understanding, there're several issues for this follow:
1) Many unnecessary and meaningless interrupts hanllding of Tx and RX
2) Performance downgrading
    A 4 word SCU MSG may need 8 interrupts to handle which terribly slow
    down the speed. (Interrupt latency will be terrible if many)
3) Complex RX flow in order to align with framework design

Any comments about these issues?

Generally, from what I see, the mailbox framework really is not designed for SCU type
devices that handling msg sending or receiving via multiple channels.  Including the
ring buffer supported for handling several msgs on the same channel and rich
completion mechanism, none of them are suitable for SCU MU in multiple channel mode.

What we really actually needed is a quite simple way to handle MSG Tx and Tx
synchronously.

int sc_call_rpc(sc_ipc_t ipc, void *msg, bool no_resp)
{       
        ...
        ret = mbox_send_message(sc_ipc->chan, msg);
        if (!no_resp) 
                ret = mbox_receive_message(sc_ipc->chan, msg);
        ...
        return ret;
}

For me, I don't think the above way driven by interrupt is suitable for SCU.
But I will still keep investigating if any better way to handle it in multi chan
mode as you wish.

How would you suggest for such situation in order to support SCU properly?

> I have some more suggestions for the client driver. But since I don't have to
> live with them (you do), I can't demand you make those changes. If you like,
> please feel free to cc me on the next revision of the client driver.

I certainly appreciate any clear and valuable improvement suggestions.
Just feel free to let me know if any better idea.

Regards
Dong Aisheng

Aisheng Dong Aug. 9, 2018, 2:22 a.m. UTC | #35

Hi Jassi,

Do you have any comments about this reply?

Sascha,
What's your option?

Regards
Dong Aisheng

> -----Original Message-----
> From: A.s. Dong
> Sent: Thursday, August 2, 2018 5:24 PM
> To: 'Jassi Brar' <[e-mail]>
> Cc: Oleksij Rempel <[e-mail]>; , Sascha Hauer
> <[e-mail]>; Shawn Guo <[e-mail]>; Fabio
> Estevam <[e-mail]>; Rob Herring <[e-mail]>;
> Mark Rutland <[e-mail]>; Vladimir Zapolskiy
> <[e-mail]>; , [e-mail],
> [e-mail], srv_heupstream <linux-arm-
> [e-mail]>; , [e-mail], linux-
> [e-mail], srv_heupstream <linux-
> [e-mail]>; Devicetree List
> <[e-mail]>; dl-linux-imx <[e-mail]>
> Subject: RE: [PATCH v7 3/6] dt-bindings: mailbox: imx-mu: add generic MU
> channel support
> 
> Hi Jassi,
> 
> > -----Original Message-----
> > From: Jassi Brar [mailto:[e-mail]]
> > Sent: Tuesday, July 31, 2018 8:42 PM
> > To: A.s. Dong <[e-mail]>
> 
> [...]
> 
> > >> Do you need me to send the full scu client driver patch for the reference?
> > >> If yes, please feel free to let me know.
> > >>
> > > Yes, that'll make it clearer.
> > >
> > Thanks for sharing the patch offline.
> >
> > I don't see why it can't be adapted to work with the simple 2-cell
> > implementation. Instead of one virtual, ask for 4 physical channels
> > and send data in u32 chunks sequentially over them.
> > In fact, that should make it even better by removing the tight-loop
> > (zero
> > delay!!) polling in send_data() -- MU does support interrupts, why not
> > use them to avoid polling?  If you find it hard, I can make a patch on
> > top of your client patchset, once Oleksij is done with the MU driver.
> >
> 
> As you insist, I will try to implement multi channels mode, then we can do a
> clear comparison to see which one is better for SCU mode rather than
> assumptions.
> 
> First of all,  using interrupt for each chan will likely to terribly slow down the
> SCU Message handling speed. SCU Messages usually are handled by SCU
> firmware very quickly in a few microseconds. (< 10us). That's why we use
> polling mode for SCU message handling.
> 
> If we break down a SCU message into a few separate MU channel interrupts
> driven, There will be a lot of unnecessary and meaningless interrupt handling.
> And the whole flow may look like:
> 
> Sending Word 0 ->
> Chan 0 interrupt -> (meaningless)
> Sending Word 1 ->
> Chan1 interrupt -> (meaningless)
> Sending Word 2 ->
> Chan2 interrupt -> (meaningless)
> Sending Word 3 ->
> Chan3 interrupt-> (meaningless)
> 
> Sending Word 4 on Chan 0 again ->
> Wait for Chan 0 interrupt (meaningless) -> ...
> 
> Then waiting for Rx:
> 
> Chan 0 Rx interrupt ->
> Read Word 0   ->
> (Then we know response size here)
> 
> Chan 1 Rx interrupt ->
> Read Word 1 ->
> Chan 2 Rx interrupt ->
> Read Word 2 ->
> Chan 3 Rx interrupt ->
> Read Word 3 ->
> 
> (If msg size > 4 words)
> Chan 0 Rx interrupt ->
> Read Word 4
> 
> ...
> And we must use a complicated way to map the received data of each
> channel into one SCU msg buffer and need handle the sequence properly for
> size > 4 words case
> 
> Is this what you intended?
> 
> Per my understanding, there're several issues for this follow:
> 1) Many unnecessary and meaningless interrupts hanllding of Tx and RX
> 2) Performance downgrading
>     A 4 word SCU MSG may need 8 interrupts to handle which terribly slow
>     down the speed. (Interrupt latency will be terrible if many)
> 3) Complex RX flow in order to align with framework design
> 
> Any comments about these issues?
> 
> Generally, from what I see, the mailbox framework really is not designed for
> SCU type devices that handling msg sending or receiving via multiple
> channels.  Including the ring buffer supported for handling several msgs on
> the same channel and rich completion mechanism, none of them are suitable
> for SCU MU in multiple channel mode.
> 
> What we really actually needed is a quite simple way to handle MSG Tx and
> Tx synchronously.
> 
> int sc_call_rpc(sc_ipc_t ipc, void *msg, bool no_resp)
> {
>         ...
>         ret = mbox_send_message(sc_ipc->chan, msg);
>         if (!no_resp)
>                 ret = mbox_receive_message(sc_ipc->chan, msg);
>         ...
>         return ret;
> }
> 
> For me, I don't think the above way driven by interrupt is suitable for SCU.
> But I will still keep investigating if any better way to handle it in multi chan
> mode as you wish.
> 
> How would you suggest for such situation in order to support SCU properly?
> 
> > I have some more suggestions for the client driver. But since I don't
> > have to live with them (you do), I can't demand you make those
> > changes. If you like, please feel free to cc me on the next revision of the
> client driver.
> 
> I certainly appreciate any clear and valuable improvement suggestions.
> Just feel free to let me know if any better idea.
> 
> Regards
> Dong Aisheng

Jassi Brar Aug. 9, 2018, 2:55 a.m. UTC | #36

On Thu, Aug 9, 2018 at 7:52 AM, A.s. Dong <[e-mail]> wrote:
> Hi Jassi,
>
> Do you have any comments about this reply?
>
I did read your post. And my opinion isn't changed. If mailbox is new
to you, please try to see how other subsystems works, especially SPI,
I2C and DMA.

Thanks.

Aisheng Dong Aug. 9, 2018, 6:45 a.m. UTC | #37

> -----Original Message-----
> From: Jassi Brar [mailto:[e-mail]]
> Sent: Thursday, August 9, 2018 10:56 AM
[...]
> On Thu, Aug 9, 2018 at 7:52 AM, A.s. Dong <[e-mail]> wrote:
> > Hi Jassi,
> >
> > Do you have any comments about this reply?
> >
> I did read your post. And my opinion isn't changed. If mailbox is new to you,
> please try to see how other subsystems works, especially SPI, I2C and DMA.
> 

Okay.
What's your comments about the known issues if doing that way?
1) Many unnecessary and meaningless interrupts handling of Tx and RX
2) Performance downgrading
    A 4 word SCU MSG may need 8 interrupts to handle which terribly slow
    down the speed. (Interrupt latency will be terrible if many). SCU msg
    are mostly handled within 10 us.
3) Complex RX flow in order to align with framework design

Are you referring to them as not a real issue?
Or even they're issues/limitations, you would still prefer multi channels way
In order to be consistent with mailbox subsystem by sacrificing the
performance drop?

And to be clear, the working flow you want is exactly as follows, right?
Sending Word 0 ->
Chan 0 interrupt -> (meaningless)
Sending Word 1 ->
Chan1 interrupt -> (meaningless)
Sending Word 2 ->
Chan2 interrupt -> (meaningless)
Sending Word 3 ->
Chan3 interrupt-> (meaningless)

Sending Word 4 on Chan 0 again ->
Wait for Chan 0 interrupt (meaningless) -> ...

Then waiting for Rx:

Chan 0 Rx interrupt ->
Read Word 0   ->
(Then we know response size here)

Chan 1 Rx interrupt ->
Read Word 1 ->
Chan 2 Rx interrupt ->
Read Word 2 ->
Chan 3 Rx interrupt ->
Read Word 3 ->

(If msg size > 4 words)
Chan 0 Rx interrupt ->
Read Word 4

Regards
Dong Aisheng

> Thanks.

Patch

diff --git a/Documentation/devicetree/bindings/mailbox/fsl,mu.txt b/Documentation/devicetree/bindings/mailbox/fsl,mu.txt
index 90e4905dfc69..113d6ab931ef 100644
--- a/Documentation/devicetree/bindings/mailbox/fsl,mu.txt
+++ b/Documentation/devicetree/bindings/mailbox/fsl,mu.txt
@@ -22,7 +22,14 @@  Required properties:
 - reg :		Should contain the registers location and length
 - interrupts :	Interrupt number. The interrupt specifier format depends
 		on the interrupt controller parent.
-- #mbox-cells:  Must be 0. Number of cells in a mailbox
+- #mbox-cells:  Must be:
+		0 - for single channel mode. i.MX8* SCU protocol specific.
+		1 - for multichannel (generic) mode.
+
+Optional properties:
+-------------------
+- clocks :	phandle to the input clock.
+- fsl,mu-side-b : Should be set for side B MU.
 
 Examples:
 --------