From patchwork Fri Jun 10 15:12:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12877699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 07CF2C43334 for ; Fri, 10 Jun 2022 15:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YjJ4py8vM5C6BicF+20LAoA4eVzOaThE8cy2papmaY4=; b=JJTA0qbnGoXnsO 4jPlnBbV/14JeTKq8Hr7Qior+Rc6lnJ4vHPx1ijwX9RS+jkxtF+Bv6thdGWVZoL58sVkpWIp/1cLR QxmbXlChBUTRe2GdwJnVbJ8Hs+TpPkPPteVGxn8LYb6Re6xIp264OK+idgkAJb07Egrgser9ZVdoo ARuQDcOjpbAqEEuisnKDhMZ0feqkkptT22rUkaYbqeXVMoqLpzgY3f7PEjL7T240y6AQ4mMBNPQQv Ns/lwObR90fUsgM7TEor7tzHlMZLHLvfMqx+TmEoxg1WqCjRdp5G2HCJ4HTK5b0gn5jBjceCIGrke Ui5NVbyZT6j5Co5AtJ5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzgK0-008i8k-Np; Fri, 10 Jun 2022 15:13:12 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzgJo-008i61-Uz for linux-arm-kernel@lists.infradead.org; Fri, 10 Jun 2022 15:13:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4FE7361F71; Fri, 10 Jun 2022 15:13:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 251D6C341C0; Fri, 10 Jun 2022 15:12:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654873979; bh=/N4kt8tL+7p6YrFOAl4Ta619t84QRyVreBncJO9IHaA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oW0O7dtjGOZ4Kgw1KnlEd60hohaAmJItETFUBcdckHmjbPqJhxtit6o2ifYuWaNpL n78zcSqaSlKIyhjtEZkShLNHak5dzge+3Jc75lsVKnrAEN32kzS4XU4jiZ4FQa8ua9 mcGb/lIVtPRDAOLUo4KurOl1w9bij1U5r1UkHbWHAW7fFiFT1emFIZLp03Y36EA1Xy xbV55eJABbVxS8+qKeZlbi0M6bKbp3OBIG1ZUfkYY/b/vwnE7O1Gv7ja1cA4nfOnMv 7MmSghTa/cFEACowJkm04GWzofrZJouWTRqdae4/cLLY1vut5r5ZcJKaJgxeXdMqQH W/P5ot7rro5/w== From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: Will Deacon , Ard Biesheuvel , Christoph Hellwig , Catalin Marinas , Robin Murphy , Russell King , stable@vger.kernel.org Subject: [PATCH 1/2] arm64: mm: Don't invalidate FROM_DEVICE buffers at start of DMA transfer Date: Fri, 10 Jun 2022 16:12:27 +0100 Message-Id: <20220610151228.4562-2-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220610151228.4562-1-will@kernel.org> References: <20220610151228.4562-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220610_081301_074657_60EFCAE6 X-CRM114-Status: GOOD ( 12.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Invalidating the buffer memory in arch_sync_dma_for_device() for FROM_DEVICE transfers When using the streaming DMA API to map a buffer prior to inbound non-coherent DMA (i.e. DMA_FROM_DEVICE), we invalidate any dirty CPU cachelines so that they will not be written back during the transfer and corrupt the buffer contents written by the DMA. This, however, poses two potential problems: (1) If the DMA transfer does not write to every byte in the buffer, then the unwritten bytes will contain stale data once the transfer has completed. (2) If the buffer has a virtual alias in userspace, then stale data may be visible via this alias during the period between performing the cache invalidation and the DMA writes landing in memory. Address both of these issues by cleaning (aka writing-back) the dirty lines in arch_sync_dma_for_device(DMA_FROM_DEVICE) instead of discarding them using invalidation. Cc: Ard Biesheuvel Cc: Christoph Hellwig Cc: Catalin Marinas Cc: Robin Murphy Cc: Russell King Cc: Link: https://lore.kernel.org/r/20220606152150.GA31568@willie-the-truck Signed-off-by: Will Deacon --- arch/arm64/mm/cache.S | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 0ea6cc25dc66..21c907987080 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -218,8 +218,6 @@ SYM_FUNC_ALIAS(__dma_flush_area, __pi___dma_flush_area) */ SYM_FUNC_START(__pi___dma_map_area) add x1, x0, x1 - cmp w2, #DMA_FROM_DEVICE - b.eq __pi_dcache_inval_poc b __pi_dcache_clean_poc SYM_FUNC_END(__pi___dma_map_area) SYM_FUNC_ALIAS(__dma_map_area, __pi___dma_map_area) From patchwork Fri Jun 10 15:12:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12877701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 36136C43334 for ; Fri, 10 Jun 2022 15:14:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MufK8UQyuPf4M9uRQuklBKrIUbuOsu+gHev1verWD/A=; b=gamPG3w+Fmgqfx u5DlLctz4G4y9cquePaM01hYvhSdINH/a0My3DlG6i9scg8VtgLCYzRcIvaPwJo4qJXanm9e9z5k1 kW9DAXX/ALl40tKb65u+TBSrNw2UHf+nDWaVXX5M7XerUnb0lqsiIIDT1dsLiv756geZaQWuj1kmw 5BWGRt/sU3TZ4M/z/w6FmK+D8PwbYsHZ3f0YndfkQzYrgE0TMKJK2N/Hv9jLvRo+Ajl9rhV0iz3HW i13kmgYK0RGnNd9sskBz0Yn8p3QOQUP1tPh9HUU2bJwUHEipx0O20XlugVdEyYbBRnXU+7LH3ZejK sBkdSoMs8rpWS2/xw2Sg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzgK9-008iAU-AS; Fri, 10 Jun 2022 15:13:21 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzgJs-008i6x-8Z for linux-arm-kernel@lists.infradead.org; Fri, 10 Jun 2022 15:13:05 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CD7B1B835F7; Fri, 10 Jun 2022 15:13:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2A6AFC3411D; Fri, 10 Jun 2022 15:13:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654873981; bh=Fu/C1oBZMxFWww70XccMQ1PQejiiSTWQ1CclDVVbWSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jFTA2zNJFioEpTjtNa9dapsCPucznh1gzeO/XcoHJRkHL3QEYKxQernwSJ2ZIuzvc gTDANJY4BgW1bOBdZC9/Tr4nZYtwPJLPtDGpftXGB91L206je4FmVh9y7Zfy+jlswU 5pqOrPsF1NdJ2H9/fsFMwyZMkf/n2BDqR2BPAtXxE2ww9MkBxxLG3sduhXMORkbB5i t5MC0UHwOR/4/uW3mR/+hhRfPEGvHYiH8Ym4Hqx7VBLhBOxJgKvxcyopMfKJFQl7HQ 6me3tGjg4sw4mxuQpVl4qbkYOrRXoEI+6t+35YULmGzrg7LeArLIT87ACP/SxxfdgC aMPmOjlZfSyJQ== From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: Will Deacon , Ard Biesheuvel , Christoph Hellwig , Catalin Marinas , Robin Murphy , Russell King Subject: [PATCH 2/2] arm64: mm: Remove assembly DMA cache maintenance wrappers Date: Fri, 10 Jun 2022 16:12:28 +0100 Message-Id: <20220610151228.4562-3-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220610151228.4562-1-will@kernel.org> References: <20220610151228.4562-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220610_081304_620803_99E0D442 X-CRM114-Status: GOOD ( 15.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Remove the __dma_{flush,map,unmap}_area assembly wrappers and call the appropriate cache maintenance functions directly from the DMA mapping callbacks. Signed-off-by: Will Deacon Reviewed-by: Catalin Marinas --- arch/arm64/include/asm/cacheflush.h | 7 ----- arch/arm64/mm/cache.S | 41 ----------------------------- arch/arm64/mm/dma-mapping.c | 19 +++++++++---- 3 files changed, 14 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index 5a228e203ef9..37185e978aeb 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -104,13 +104,6 @@ static inline void flush_icache_range(unsigned long start, unsigned long end) } #define flush_icache_range flush_icache_range -/* - * Cache maintenance functions used by the DMA API. No to be used directly. - */ -extern void __dma_map_area(const void *, size_t, int); -extern void __dma_unmap_area(const void *, size_t, int); -extern void __dma_flush_area(const void *, size_t); - /* * Copy user data from/to a page which is mapped into a different * processes address space. Really, we want to allow our "user diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 21c907987080..081058d4e436 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -194,44 +194,3 @@ SYM_FUNC_START(__pi_dcache_clean_pop) ret SYM_FUNC_END(__pi_dcache_clean_pop) SYM_FUNC_ALIAS(dcache_clean_pop, __pi_dcache_clean_pop) - -/* - * __dma_flush_area(start, size) - * - * clean & invalidate D / U line - * - * - start - virtual start address of region - * - size - size in question - */ -SYM_FUNC_START(__pi___dma_flush_area) - add x1, x0, x1 - dcache_by_line_op civac, sy, x0, x1, x2, x3 - ret -SYM_FUNC_END(__pi___dma_flush_area) -SYM_FUNC_ALIAS(__dma_flush_area, __pi___dma_flush_area) - -/* - * __dma_map_area(start, size, dir) - * - start - kernel virtual start address - * - size - size of region - * - dir - DMA direction - */ -SYM_FUNC_START(__pi___dma_map_area) - add x1, x0, x1 - b __pi_dcache_clean_poc -SYM_FUNC_END(__pi___dma_map_area) -SYM_FUNC_ALIAS(__dma_map_area, __pi___dma_map_area) - -/* - * __dma_unmap_area(start, size, dir) - * - start - kernel virtual start address - * - size - size of region - * - dir - DMA direction - */ -SYM_FUNC_START(__pi___dma_unmap_area) - add x1, x0, x1 - cmp w2, #DMA_TO_DEVICE - b.ne __pi_dcache_inval_poc - ret -SYM_FUNC_END(__pi___dma_unmap_area) -SYM_FUNC_ALIAS(__dma_unmap_area, __pi___dma_unmap_area) diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index 6719f9efea09..df0c488ae643 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -14,20 +14,29 @@ #include void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, - enum dma_data_direction dir) + enum dma_data_direction dir) { - __dma_map_area(phys_to_virt(paddr), size, dir); + unsigned long start = (unsigned long)phys_to_virt(paddr); + + dcache_clean_poc(start, start + size); } void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size, - enum dma_data_direction dir) + enum dma_data_direction dir) { - __dma_unmap_area(phys_to_virt(paddr), size, dir); + unsigned long start = (unsigned long)phys_to_virt(paddr); + + if (dir == DMA_TO_DEVICE) + return; + + dcache_inval_poc(start, start + size); } void arch_dma_prep_coherent(struct page *page, size_t size) { - __dma_flush_area(page_address(page), size); + unsigned long start = (unsigned long)page_address(page); + + dcache_clean_inval_poc(start, start + size); } #ifdef CONFIG_IOMMU_DMA