From patchwork Fri May 31 15:07:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amelie Delaunay X-Patchwork-Id: 13681804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CFF19C25B75 for ; Fri, 31 May 2024 15:09:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wVZ2hwOZFV5yB6P/qYvqIGWqWlkMSQWM5BfWD2QS3n0=; b=NFOjxHRWufvgqg 36/siW89BWcLQYWG/jzO8AmL+DMQMihZSxulv+DzhLY46W0fizCSrHRQGgVExI5BDlfmsCIJvMlY6 Ske33uYqzaWq2J4tzEIrk/Zef7oGwzWpGb3DiVOIukBNuda4trwGKjyfIGT3TkhbII1B5Ii9cM9Ck 2dR9iigxz/YG8YRZExqQepOnME0lfiJp3UPnsEpYSIOIbmZRELmE0y03TQP1RcyrIBVZDfrSMBZio 7CexrzqY0meT2TS5VbMEG9RJxC57ZttPng4Wq8+vT3kvfpjdAVG7DK4DvxPUaKrq4G3IooCXLjBVD zTzhpiYfTRHAtSYWA4zw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sD3sr-0000000AcTH-2Nso; Fri, 31 May 2024 15:09:33 +0000 Received: from mx08-00178001.pphosted.com ([91.207.212.93] helo=mx07-00178001.pphosted.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sD3sg-0000000AcNj-0Q4y for linux-arm-kernel@lists.infradead.org; Fri, 31 May 2024 15:09:24 +0000 Received: from pps.filterd (m0046661.ppops.net [127.0.0.1]) by mx07-00178001.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 44VDSXoS007210; Fri, 31 May 2024 17:09:13 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foss.st.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=selector1; bh= 3BomsPzw+rrfxUkd5voZleggY17cgPEfFVxoaFaa784=; b=zohP+ML+5o0jJkKU Z/QkGTiLyjiIWT0nqTpcLfDwVfLsAw1nhOzwFp1eqNFBxG0H+upg/CvaLMlHxOsk Lgr1iSifoKzGcY+g4crU/ASTUq3PnLs7PJH6bNKfjvyWR9urWsJkRhX4s3e0gXCH DLz3cImybOTwAQA0REa+8bVmDJ3Be9hfNuFcE5s9AB+eBvnP/Coy+Bat7dIjpMpf hCpEI9ogADxcc1xq9ci0cLpghfw1ZZj66VSR/O4JwFfCWqwIBi6lEcw1GJi1bw+K PfNOIxLir+o70CYfhiKOXZkRD0Jwn0R+RNWKpU102JBSFDwJHzgIx6/dV0jAQSJK 7nJ3LQ== Received: from beta.dmz-ap.st.com (beta.dmz-ap.st.com [138.198.100.35]) by mx07-00178001.pphosted.com (PPS) with ESMTPS id 3yb9ykd73f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 31 May 2024 17:09:13 +0200 (MEST) Received: from euls16034.sgp.st.com (euls16034.sgp.st.com [10.75.44.20]) by beta.dmz-ap.st.com (STMicroelectronics) with ESMTP id 3B33B4004B; Fri, 31 May 2024 17:09:09 +0200 (CEST) Received: from Webmail-eu.st.com (shfdag1node3.st.com [10.75.129.71]) by euls16034.sgp.st.com (STMicroelectronics) with ESMTP id 694DD220B65; Fri, 31 May 2024 17:08:23 +0200 (CEST) Received: from localhost (10.252.27.179) by SHFDAG1NODE3.st.com (10.75.129.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 31 May 2024 17:08:23 +0200 From: Amelie Delaunay To: Vinod Koul , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Maxime Coquelin , Alexandre Torgue CC: , , , , , , Amelie Delaunay Subject: [PATCH v4 09/12] dmaengine: stm32-dma3: improve residue granularity Date: Fri, 31 May 2024 17:07:09 +0200 Message-ID: <20240531150712.2503554-10-amelie.delaunay@foss.st.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240531150712.2503554-1-amelie.delaunay@foss.st.com> References: <20240531150712.2503554-1-amelie.delaunay@foss.st.com> MIME-Version: 1.0 X-Originating-IP: [10.252.27.179] X-ClientProxiedBy: EQNCAS1NODE4.st.com (10.75.129.82) To SHFDAG1NODE3.st.com (10.75.129.71) X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16 definitions=2024-05-31_11,2024-05-30_01,2024-05-17_01 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240531_080922_840175_95B2C135 X-CRM114-Status: GOOD ( 28.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement own device_tx_status ops to compute the residue with a finer granularity, up to bytes. STM32 DMA3 has a bitfield, BNDT, in CxTR1 register which reflects the number of bytes read from the source. It also has a bitfield, FIFOL, in CxSR register which reflects the FIFO level in units of programmed destination data width. The channel is briefly suspended to get a coherent snapshot of registers. It is possible to correct the fifo level when packing/unpacking is enabled with destination increment. Signed-off-by: Amelie Delaunay --- drivers/dma/stm32/stm32-dma3.c | 165 ++++++++++++++++++++++++++++++++- 1 file changed, 163 insertions(+), 2 deletions(-) diff --git a/drivers/dma/stm32/stm32-dma3.c b/drivers/dma/stm32/stm32-dma3.c index 5032d74d6288..9249b64a4455 100644 --- a/drivers/dma/stm32/stm32-dma3.c +++ b/drivers/dma/stm32/stm32-dma3.c @@ -808,6 +808,134 @@ static void stm32_dma3_chan_reset(struct stm32_dma3_chan *chan) writel_relaxed(ccr |= CCR_RESET, ddata->base + STM32_DMA3_CCR(chan->id)); } +static int stm32_dma3_chan_get_curr_hwdesc(struct stm32_dma3_swdesc *swdesc, u32 cllr, u32 *residue) +{ + u32 i, lli_offset, next_lli_offset = cllr & CLLR_LA; + + /* If cllr is null, it means it is either the last or single item */ + if (!cllr) + return swdesc->lli_size - 1; + + /* In cyclic mode, go fast and first check we are not on the last item */ + if (swdesc->cyclic && next_lli_offset == (swdesc->lli[0].hwdesc_addr & CLLR_LA)) + return swdesc->lli_size - 1; + + /* As transfer is in progress, look backward from the last item */ + for (i = swdesc->lli_size - 1; i > 0; i--) { + *residue += FIELD_GET(CBR1_BNDT, swdesc->lli[i].hwdesc->cbr1); + lli_offset = swdesc->lli[i].hwdesc_addr & CLLR_LA; + if (lli_offset == next_lli_offset) + return i - 1; + } + + return -EINVAL; +} + +static void stm32_dma3_chan_set_residue(struct stm32_dma3_chan *chan, + struct stm32_dma3_swdesc *swdesc, + struct dma_tx_state *txstate) +{ + struct stm32_dma3_ddata *ddata = to_stm32_dma3_ddata(chan); + struct device *dev = chan2dev(chan); + struct stm32_dma3_hwdesc *hwdesc; + u32 residue, curr_lli, csr, cdar, cbr1, cllr, bndt, fifol; + bool pack_unpack; + int ret; + + csr = readl_relaxed(ddata->base + STM32_DMA3_CSR(chan->id)); + if (!(csr & CSR_IDLEF) && chan->dma_status != DMA_PAUSED) { + /* Suspend current transfer to read registers for a snapshot */ + writel_relaxed(swdesc->ccr | CCR_SUSP, ddata->base + STM32_DMA3_CCR(chan->id)); + ret = readl_relaxed_poll_timeout_atomic(ddata->base + STM32_DMA3_CSR(chan->id), csr, + csr & (CSR_SUSPF | CSR_IDLEF), 1, 10); + + if (ret || ((csr & CSR_TCF) && (csr & CSR_IDLEF))) { + writel_relaxed(CFCR_SUSPF, ddata->base + STM32_DMA3_CFCR(chan->id)); + writel_relaxed(swdesc->ccr, ddata->base + STM32_DMA3_CCR(chan->id)); + if (ret) + dev_err(dev, "Channel suspension timeout, csr=%08x\n", csr); + } + } + + /* If channel is still active (CSR_IDLEF is not set), can't get a reliable residue */ + if (!(csr & CSR_IDLEF)) + dev_warn(dev, "Can't get residue: channel still active, csr=%08x\n", csr); + + /* + * If channel is not suspended, but Idle and Transfer Complete are set, + * linked-list is over, no residue + */ + if (!(csr & CSR_SUSPF) && (csr & CSR_TCF) && (csr & CSR_IDLEF)) + return; + + /* Read registers to have a snapshot */ + cllr = readl_relaxed(ddata->base + STM32_DMA3_CLLR(chan->id)); + cbr1 = readl_relaxed(ddata->base + STM32_DMA3_CBR1(chan->id)); + cdar = readl_relaxed(ddata->base + STM32_DMA3_CDAR(chan->id)); + + /* Resume current transfer */ + if (csr & CSR_SUSPF) { + writel_relaxed(CFCR_SUSPF, ddata->base + STM32_DMA3_CFCR(chan->id)); + writel_relaxed(swdesc->ccr, ddata->base + STM32_DMA3_CCR(chan->id)); + } + + /* Add current BNDT */ + bndt = FIELD_GET(CBR1_BNDT, cbr1); + residue = bndt; + + /* Get current hwdesc and cumulate residue of pending hwdesc BNDT */ + ret = stm32_dma3_chan_get_curr_hwdesc(swdesc, cllr, &residue); + if (ret < 0) { + dev_err(chan2dev(chan), "Can't get residue: current hwdesc not found\n"); + return; + } + curr_lli = ret; + + /* Read current FIFO level - in units of programmed destination data width */ + hwdesc = swdesc->lli[curr_lli].hwdesc; + fifol = FIELD_GET(CSR_FIFOL, csr) * (1 << FIELD_GET(CTR1_DDW_LOG2, hwdesc->ctr1)); + /* If the FIFO contains as many bytes as its size, it can't contain more */ + if (fifol == (1 << (chan->fifo_size + 1))) + goto skip_fifol_update; + + /* + * In case of PACKING (Destination burst length > Source burst length) or UNPACKING + * (Source burst length > Destination burst length), bytes could be pending in the FIFO + * (to be packed up to Destination burst length or unpacked into Destination burst length + * chunks). + * BNDT is not reliable, as it reflects the number of bytes read from the source but not the + * number of bytes written to the destination. + * FIFOL is also not sufficient, because it reflects the number of available write beats in + * units of Destination data width but not the bytes not yet packed or unpacked. + * In case of Destination increment DINC, it is possible to compute the number of bytes in + * the FIFO: + * fifol_in_bytes = bytes_read - bytes_written. + */ + pack_unpack = !!(FIELD_GET(CTR1_PAM, hwdesc->ctr1) == CTR1_PAM_PACK_UNPACK); + if (pack_unpack && (hwdesc->ctr1 & CTR1_DINC)) { + int bytes_read = FIELD_GET(CBR1_BNDT, hwdesc->cbr1) - bndt; + int bytes_written = cdar - hwdesc->cdar; + + if (bytes_read > 0) + fifol = bytes_read - bytes_written; + } + +skip_fifol_update: + if (fifol) { + dev_dbg(chan2dev(chan), "%u byte(s) in the FIFO\n", fifol); + dma_set_in_flight_bytes(txstate, fifol); + /* + * Residue is already accurate for DMA_MEM_TO_DEV as BNDT reflects data read from + * the source memory buffer, so just need to add fifol to residue in case of + * DMA_DEV_TO_MEM transfer because these bytes are not yet written in destination + * memory buffer. + */ + if (chan->dma_config.direction == DMA_DEV_TO_MEM) + residue += fifol; + } + dma_set_residue(txstate, residue); +} + static int stm32_dma3_chan_stop(struct stm32_dma3_chan *chan) { struct stm32_dma3_ddata *ddata = to_stm32_dma3_ddata(chan); @@ -1310,6 +1438,39 @@ static void stm32_dma3_synchronize(struct dma_chan *c) vchan_synchronize(&chan->vchan); } +static enum dma_status stm32_dma3_tx_status(struct dma_chan *c, dma_cookie_t cookie, + struct dma_tx_state *txstate) +{ + struct stm32_dma3_chan *chan = to_stm32_dma3_chan(c); + struct stm32_dma3_swdesc *swdesc = NULL; + enum dma_status status; + unsigned long flags; + struct virt_dma_desc *vd; + + status = dma_cookie_status(c, cookie, txstate); + if (status == DMA_COMPLETE) + return status; + + if (!txstate) + return chan->dma_status; + + spin_lock_irqsave(&chan->vchan.lock, flags); + + vd = vchan_find_desc(&chan->vchan, cookie); + if (vd) + swdesc = to_stm32_dma3_swdesc(vd); + else if (chan->swdesc && chan->swdesc->vdesc.tx.cookie == cookie) + swdesc = chan->swdesc; + + /* Get residue/in_flight_bytes only if a transfer is currently running (swdesc != NULL) */ + if (swdesc) + stm32_dma3_chan_set_residue(chan, swdesc, txstate); + + spin_unlock_irqrestore(&chan->vchan.lock, flags); + + return chan->dma_status; +} + static void stm32_dma3_issue_pending(struct dma_chan *c) { struct stm32_dma3_chan *chan = to_stm32_dma3_chan(c); @@ -1506,7 +1667,7 @@ static int stm32_dma3_probe(struct platform_device *pdev) dma_dev->descriptor_reuse = true; dma_dev->max_sg_burst = STM32_DMA3_MAX_SEG_SIZE; - dma_dev->residue_granularity = DMA_RESIDUE_GRANULARITY_DESCRIPTOR; + dma_dev->residue_granularity = DMA_RESIDUE_GRANULARITY_BURST; dma_dev->device_alloc_chan_resources = stm32_dma3_alloc_chan_resources; dma_dev->device_free_chan_resources = stm32_dma3_free_chan_resources; dma_dev->device_prep_dma_memcpy = stm32_dma3_prep_dma_memcpy; @@ -1518,7 +1679,7 @@ static int stm32_dma3_probe(struct platform_device *pdev) dma_dev->device_resume = stm32_dma3_resume; dma_dev->device_terminate_all = stm32_dma3_terminate_all; dma_dev->device_synchronize = stm32_dma3_synchronize; - dma_dev->device_tx_status = dma_cookie_status; + dma_dev->device_tx_status = stm32_dma3_tx_status; dma_dev->device_issue_pending = stm32_dma3_issue_pending; /* if dma_channels is not modified, get it from hwcfgr1 */