usb: dwc3: gadget: Support Multi-Stream Transfer

Commit Message

Thinh Nguyen Nov. 30, 2021, 3:21 a.m. UTC

Synopsys introduced a new enhancement to DWC_usb32 called Multi-Stream
Transfer (MST) to improve bulk streams performance for SuperSpeed and
SuperSpeed Plus. This enhancement allows the controller to look ahead
and process multiple bulk streams.

Previously, to initiate a bulk stream transfer, the driver has to issue
Start Transfer command and wait for the stream to complete before
initiating a new stream. As a result, the controller does not process
TRBs beyond a single stream. With the enhancement, as long as there are
new requests, the dwc3 driver can keep preparing new TRBs and the
controller can keep caching and processing them without waiting for the
transfer completion.

The programming flow is similar to regular bulk endpoint with a few
additional rules:

1) Chained TRBs of the same stream must have a matching stream ID
2) The last TRB of a stream must have CHN=0
3) All the TRBs with LST=0 must have CSP=1

Depends on the application and usage, internal tests show significant
performance improvement in UASP transfers with this enhancement.

Signed-off-by: Thinh Nguyen <[e-mail]>
---
 drivers/usb/dwc3/core.h   |  9 +++++++++
 drivers/usb/dwc3/gadget.c | 26 +++++++++++++++++++++-----
 2 files changed, 30 insertions(+), 5 deletions(-)


base-commit: 4d012040161cba054208555300d9fdf2b8925c34

Comments

Felipe Balbi Nov. 30, 2021, 7:23 a.m. UTC | #1

Hi Thinh,

Thinh Nguyen <[e-mail]> writes:
> Synopsys introduced a new enhancement to DWC_usb32 called Multi-Stream
> Transfer (MST) to improve bulk streams performance for SuperSpeed and
> SuperSpeed Plus. This enhancement allows the controller to look ahead
> and process multiple bulk streams.

Finally!! This is a great improvement to the core :-)

> Previously, to initiate a bulk stream transfer, the driver has to issue
> Start Transfer command and wait for the stream to complete before
> initiating a new stream. As a result, the controller does not process
> TRBs beyond a single stream. With the enhancement, as long as there are
> new requests, the dwc3 driver can keep preparing new TRBs and the
> controller can keep caching and processing them without waiting for the
> transfer completion.
>
> The programming flow is similar to regular bulk endpoint with a few
> additional rules:
>
> 1) Chained TRBs of the same stream must have a matching stream ID
> 2) The last TRB of a stream must have CHN=0
> 3) All the TRBs with LST=0 must have CSP=1
>
> Depends on the application and usage, internal tests show significant
> performance improvement in UASP transfers with this enhancement.

Does this mean that we can now issue Start Transfer for each of the
streams on the same endpoint? Should we start having per-stream TRBs,
then? Maybe we don't need a full 256 TRBs for each stream, perhaps 16 to
32 TRBs for each stream ought to be enough?

Thinh Nguyen Nov. 30, 2021, 10:45 p.m. UTC | #2

Hi Felipe,

Felipe Balbi wrote:
> 
> Hi Thinh,
> 
> Thinh Nguyen <[e-mail]> writes:
>> Synopsys introduced a new enhancement to DWC_usb32 called Multi-Stream
>> Transfer (MST) to improve bulk streams performance for SuperSpeed and
>> SuperSpeed Plus. This enhancement allows the controller to look ahead
>> and process multiple bulk streams.
> 
> Finally!! This is a great improvement to the core :-)
> 
>> Previously, to initiate a bulk stream transfer, the driver has to issue
>> Start Transfer command and wait for the stream to complete before
>> initiating a new stream. As a result, the controller does not process
>> TRBs beyond a single stream. With the enhancement, as long as there are
>> new requests, the dwc3 driver can keep preparing new TRBs and the
>> controller can keep caching and processing them without waiting for the
>> transfer completion.
>>
>> The programming flow is similar to regular bulk endpoint with a few
>> additional rules:
>>
>> 1) Chained TRBs of the same stream must have a matching stream ID
>> 2) The last TRB of a stream must have CHN=0
>> 3) All the TRBs with LST=0 must have CSP=1
>>
>> Depends on the application and usage, internal tests show significant
>> performance improvement in UASP transfers with this enhancement.
> 
> Does this mean that we can now issue Start Transfer for each of the
> streams on the same endpoint? Should we start having per-stream TRBs,
> then? Maybe we don't need a full 256 TRBs for each stream, perhaps 16 to
> 32 TRBs for each stream ought to be enough?
> 

Sorry, I may not be clear in the description. Just like previously,
before we can issue a Start Transfer command, we need to make sure that
the endpoint transfer is completed or ended first. That hasn't changed.
However, previously, while the endpoint is started, all the TRBs need to
be of the same stream id.

The enhancement here allows the endpoint to remain "started" with
multiple streams of different IDs. The driver can add new streams/TRBs
with Update Transfer command.

MST is mainly for device initiated stream. We don't maintain multiple
TRB rings (one of for each stream) as xHCI driver for each active
stream. For device initiated stream, the device will have control of the
stream/transfer order. The order will be the order of the stream
requests queued by the function driver.

The dwc3 driver can indicate the last TRB of a stream ID with CHN=0.
This will tell the controller that it needs to switch/initiate stream on
the next TRB's stream ID.

In short, we will still keep the 256-TRB ring per endpoint. To keep the
driver behavior/implementation simple and consistent with regular bulk,
we will only need to issue Start Transfer at the beginning and continue
to prepare new TRBs from different streams using the same TRB ring. We
just need to follow the rules noted in the commit message.

BR,
Thinh

Patch

diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index 5c491d0a19d7..e1cc3f7398fb 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -153,6 +153,7 @@ 
 #define DWC3_DGCMDPAR		0xc710
 #define DWC3_DGCMD		0xc714
 #define DWC3_DALEPENA		0xc720
+#define DWC3_DCFG1		0xc740 /* DWC_usb32 only */
 
 #define DWC3_DEP_BASE(n)	(0xc800 + ((n) * 0x10))
 #define DWC3_DEPCMDPAR2		0x00
@@ -382,6 +383,7 @@ 
 
 /* Global HWPARAMS9 Register */
 #define DWC3_GHWPARAMS9_DEV_TXF_FLUSH_BYPASS	BIT(0)
+#define DWC3_GHWPARAMS9_DEV_MST			BIT(1)
 
 /* Global Frame Length Adjustment Register */
 #define DWC3_GFLADJ_30MHZ_SDBND_SEL		BIT(7)
@@ -558,6 +560,9 @@ 
 /* The EP number goes 0..31 so ep0 is always out and ep1 is always in */
 #define DWC3_DALEPENA_EP(n)		BIT(n)
 
+/* DWC_usb32 DCFG1 config */
+#define DWC3_DCFG1_DIS_MST_ENH		BIT(1)
+
 #define DWC3_DEPCMD_TYPE_CONTROL	0
 #define DWC3_DEPCMD_TYPE_ISOC		1
 #define DWC3_DEPCMD_TYPE_BULK		2
@@ -888,6 +893,10 @@  struct dwc3_hwparams {
 /* HWPARAMS7 */
 #define DWC3_RAM1_DEPTH(n)	((n) & 0xffff)
 
+/* HWPARAMS9 */
+#define DWC3_MST_CAPABLE(p)	(!!((p)->hwparams9 &		\
+			DWC3_GHWPARAMS9_DEV_MST))
+
 /**
  * struct dwc3_request - representation of a transfer request
  * @request: struct usb_request to be transferred
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 7e3db00e9759..3c5b60539142 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -1260,12 +1260,17 @@  static void __dwc3_prepare_one_trb(struct dwc3_ep *dep, struct dwc3_trb *trb,
 			trb->ctrl |= DWC3_TRB_CTRL_ISP_IMI;
 	}
 
+	/* All TRBs setup for MST must set CSP=1 when LST=0 */
+	if (dep->stream_capable && DWC3_MST_CAPABLE(&dwc->hwparams))
+		trb->ctrl |= DWC3_TRB_CTRL_CSP;
+
 	if ((!no_interrupt && !chain) || must_interrupt)
 		trb->ctrl |= DWC3_TRB_CTRL_IOC;
 
 	if (chain)
 		trb->ctrl |= DWC3_TRB_CTRL_CHN;
-	else if (dep->stream_capable && is_last)
+	else if (dep->stream_capable && is_last &&
+		 !DWC3_MST_CAPABLE(&dwc->hwparams))
 		trb->ctrl |= DWC3_TRB_CTRL_LST;
 
 	if (usb_endpoint_xfer_bulk(dep->endpoint.desc) && dep->stream_capable)
@@ -1513,7 +1518,8 @@  static int dwc3_prepare_trbs(struct dwc3_ep *dep)
 		 * burst capability may try to read and use TRBs beyond the
 		 * active transfer instead of stopping.
 		 */
-		if (dep->stream_capable && req->request.is_last)
+		if (dep->stream_capable && req->request.is_last &&
+		    !DWC3_MST_CAPABLE(&dep->dwc->hwparams))
 			return ret;
 	}
 
@@ -1546,7 +1552,8 @@  static int dwc3_prepare_trbs(struct dwc3_ep *dep)
 		 * burst capability may try to read and use TRBs beyond the
 		 * active transfer instead of stopping.
 		 */
-		if (dep->stream_capable && req->request.is_last)
+		if (dep->stream_capable && req->request.is_last &&
+		    !DWC3_MST_CAPABLE(&dwc->hwparams))
 			return ret;
 	}
 
@@ -1623,7 +1630,8 @@  static int __dwc3_gadget_kick_transfer(struct dwc3_ep *dep)
 		return ret;
 	}
 
-	if (dep->stream_capable && req->request.is_last)
+	if (dep->stream_capable && req->request.is_last &&
+	    !DWC3_MST_CAPABLE(&dep->dwc->hwparams))
 		dep->flags |= DWC3_EP_WAIT_TRANSFER_COMPLETE;
 
 	return 0;
@@ -2638,6 +2646,13 @@  static int __dwc3_gadget_start(struct dwc3 *dwc)
 	reg |= DWC3_DCFG_IGNSTRMPP;
 	dwc3_writel(dwc->regs, DWC3_DCFG, reg);
 
+	/* Enable MST by default if the device is capable of MST */
+	if (DWC3_MST_CAPABLE(&dwc->hwparams)) {
+		reg = dwc3_readl(dwc->regs, DWC3_DCFG1);
+		reg &= ~DWC3_DCFG1_DIS_MST_ENH;
+		dwc3_writel(dwc->regs, DWC3_DCFG1, reg);
+	}
+
 	/* Start with SuperSpeed Default */
 	dwc3_gadget_ep0_desc.wMaxPacketSize = cpu_to_le16(512);
 
@@ -3437,7 +3452,8 @@  static void dwc3_gadget_endpoint_stream_event(struct dwc3_ep *dep,
 	case DEPEVT_STREAM_NOSTREAM:
 		if ((dep->flags & DWC3_EP_IGNORE_NEXT_NOSTREAM) ||
 		    !(dep->flags & DWC3_EP_FORCE_RESTART_STREAM) ||
-		    !(dep->flags & DWC3_EP_WAIT_TRANSFER_COMPLETE))
+		    (!DWC3_MST_CAPABLE(&dwc->hwparams) &&
+		     !(dep->flags & DWC3_EP_WAIT_TRANSFER_COMPLETE)))
 			break;
 
 		/*