From patchwork Sun Nov 6 22:01:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2DF6C433FE for ; Sun, 6 Nov 2022 22:03:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wKIfyexx1lsML6IZw9jHi4kjLv08eTMP1Ux0APr0I7k=; b=FS/a3N6sA6Z0nb h8xAUXZI7Lhuo73bz2Mx3+LfPoSoorpyNwey4cahGQTwnoRZyZIn1J08Gv4bYqEKOYZMkUnpk4Ov0 BLscyU3kXkUV/wNCS7m2O2bb5YDXIRt01FphBLMPyJTmCB2BfpErr4IfW8po+8FwQwshA/Uj++NBS GOS+MI66MarWa2Kmq9NcGOAXy76uEr2BoafC/3N3hCVJ8hhWnLwyY7s/mE4dO5Pfr43Chz/arzGpM YQD5Uokc+Dsfl287iJB/kSeAVm6QWjoIXDGuGs6Gaajduixz6ase6hYxWgGg3OxaTdQ1fAvRSXNXj llpMQQidH9tsOd/DCrqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniQ-00A8tY-9n; Sun, 06 Nov 2022 22:02:06 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniG-00A8qi-Fk for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:01:58 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 2B864B80D3A; Sun, 6 Nov 2022 22:01:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F350C43143; Sun, 6 Nov 2022 22:01:50 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 01/13] mm/slab: Decouple ARCH_KMALLOC_MINALIGN from ARCH_DMA_MINALIGN Date: Sun, 6 Nov 2022 22:01:31 +0000 Message-Id: <20221106220143.2129263-2-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140156_831415_E285CD15 X-CRM114-Status: GOOD ( 16.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for supporting a kmalloc() minimum alignment smaller than the arch DMA alignment, decouple the two definitions. This requires that the DMA API bounces smaller kmalloc() allocations, hence the smaller ARCH_KMALLOC_MINALIGN is only enabled if a new CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC option is enabled (to be introduced in a subsequent patch). After this patch, ARCH_DMA_MINALIGN is expected to be used in static alignment annotations and defined by an architecture to be the maximum alignment for all supported configurations/SoCs in a single Image. ARCH_KMALLOC_MINALIGN becomes the default sizeof(unsigned long long). Since ARCH_DMA_MINALIGN is now always defined, adjust the #ifdef in dma_get_cache_alignment() so that there is no change for architectures not requiring a minimum DMA alignment. Signed-off-by: Catalin Marinas Cc: Andrew Morton Cc: Christoph Hellwig Cc: Robin Murphy --- include/linux/dma-mapping.h | 2 +- include/linux/slab.h | 14 +++++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 0ee20b764000..3288a1339271 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -545,7 +545,7 @@ static inline int dma_set_min_align_mask(struct device *dev, static inline int dma_get_cache_alignment(void) { -#ifdef ARCH_DMA_MINALIGN +#ifdef ARCH_HAS_DMA_MINALIGN return ARCH_DMA_MINALIGN; #endif return 1; diff --git a/include/linux/slab.h b/include/linux/slab.h index 90877fcde70b..b104d63e5456 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -221,12 +221,20 @@ void kmem_dump_obj(void *object); * alignment larger than the alignment of a 64-bit integer. * Setting ARCH_DMA_MINALIGN in arch headers allows that. */ -#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8 +#ifdef ARCH_DMA_MINALIGN +#define ARCH_HAS_DMA_MINALIGN +#if ARCH_DMA_MINALIGN > 8 && !defined(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN) +#endif #else +#define ARCH_DMA_MINALIGN __alignof__(unsigned long long) +#endif + +#ifndef ARCH_KMALLOC_MINALIGN #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long) +#else +#define KMALLOC_MIN_SIZE ARCH_KMALLOC_MINALIGN +#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE) #endif /* From patchwork Sun Nov 6 22:01:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91FB9C433FE for ; Sun, 6 Nov 2022 22:03:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aBFllpQB87mZ8kanf7tDXnSa6O+EbDiEjJOBEd6uxBw=; b=hYLWy5B0/4RAOM PVLeSR6YVrCSJa1ux4NF057+rpMU3WBId4ZRq7KyVCgAsfHFttjD/muTS40OegRBDh2xljZ1zyK/M 2AhPnBKikqTCtTF8Z7HG9AOn+GFT25pvd54T5JSrlXllCH2XLZWZQ11OGqaUsYuTNlGfTGVucFnoM M7SmYNaIp7NWStjVCK8K2CWFsMVhTFFQSk1+McaSsal77GeJK+a3GJLjz/FpU35W5nsF962ydb5xr 5LJdvaNmGVc1/lxvBuCu+oy1fCYIIy9Ke5iHxjAWVGIy0nK83BdtMlmlgCpjgMVtzfjXtqBGKerhv A52lGWJgxn41u9q/J62A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniy-00A97b-R1; Sun, 06 Nov 2022 22:02:41 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniL-00A8rL-RS for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:03 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 585E2B80D38; Sun, 6 Nov 2022 22:01:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37993C433B5; Sun, 6 Nov 2022 22:01:54 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 02/13] dma-mapping: Force bouncing if the kmalloc() size is not cacheline-aligned Date: Sun, 6 Nov 2022 22:01:32 +0000 Message-Id: <20221106220143.2129263-3-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140202_202683_C5271B5E X-CRM114-Status: GOOD ( 18.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For direct DMA, if the size is small enough to have originated from a kmalloc() cache below ARCH_DMA_MINALIGN, check its alignment against cache_line_size() and bounce if necessary. For larger sizes, it is the responsibility of the DMA API caller to ensure proper alignment. Signed-off-by: Catalin Marinas Cc: Christoph Hellwig Cc: Robin Murphy --- include/linux/dma-map-ops.h | 27 +++++++++++++++++++++++++++ kernel/dma/direct.h | 3 ++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index d678afeb8a13..785f7aa90f57 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -8,6 +8,7 @@ #include #include +#include struct cma; @@ -275,6 +276,32 @@ static inline bool dev_is_dma_coherent(struct device *dev) } #endif /* CONFIG_ARCH_HAS_DMA_COHERENCE_H */ +/* + * Check whether the given size, assuming it is for a kmalloc()'ed object, is + * safe for non-coherent DMA or needs bouncing. + */ +static inline bool dma_kmalloc_needs_bounce(struct device *dev, size_t size, + enum dma_data_direction dir) +{ + /* + * No need for bouncing if coherent DMA or the direction is + * DMA_TO_DEVICE. + */ + if (!IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) || + dir == DMA_TO_DEVICE || dev_is_dma_coherent(dev)) + return false; + + /* + * Larger kmalloc() sizes are guaranteed to be aligned to + * ARCH_DMA_MINALIGN. + */ + if (size >= 2 * ARCH_DMA_MINALIGN || + IS_ALIGNED(kmalloc_size_roundup(size), dma_get_cache_alignment())) + return false; + + return true; +} + void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs); void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index e38ffc5e6bdd..97ec892ea0b5 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -94,7 +94,8 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev, return swiotlb_map(dev, phys, size, dir, attrs); } - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (unlikely(!dma_capable(dev, dma_addr, size, true)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { if (is_pci_p2pdma_page(page)) return DMA_MAPPING_ERROR; if (is_swiotlb_active(dev)) From patchwork Sun Nov 6 22:01:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A158AC4332F for ; Sun, 6 Nov 2022 22:03:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PafOR9+td8mwbiUTBDtn2leRW1fgK8fG9wveecZ90x0=; b=Hjedl2Qyl2rO6W 6e7gZr2aDT+/joORlDnDXdWCBv09lWVG1WRp7DUCAQ+v7nZcnjJvJsLzSjVVzhVlIWimxjXbkRJ7+ MIsDMFOwLzfYXlXHriFvnP1HucPnlik+H8S4xykJV07og21zCETFd/ysFmN38pRY/1dofNTyB3Rxm JCMpzTfnkBke0FM0b5G5k+48k6/H57XEv6Pg7rOKpji6WYpv8USIGFBLzZoD7fjQ0stDZH7HwJ1Cb eQA+iWaTaTtyvMTNc6uG3OHpcFmz0UTC13qoj6CM7/Cm1FNMxnz3yu+ykP4UZcfdWRdnkwl0waL+q nZukXw1y5nxlWgWbfuuw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornj9-00A9DR-Rv; Sun, 06 Nov 2022 22:02:51 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniO-00A8sh-5G for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:05 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8877560DD5; Sun, 6 Nov 2022 22:02:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 49473C4347C; Sun, 6 Nov 2022 22:01:58 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 03/13] iommu/dma: Force bouncing of the size is not cacheline-aligned Date: Sun, 6 Nov 2022 22:01:33 +0000 Message-Id: <20221106220143.2129263-4-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140204_300239_080BEB2C X-CRM114-Status: GOOD ( 21.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Similarly to the direct DMA, bounce small allocations as they may have originated from a kmalloc() cache not safe for DMA. Unlike the direct DMA, iommu_dma_map_sg() cannot call iommu_dma_map_sg_swiotlb() for all non-coherent devices as this would break some cases where the iova is expected to be contiguous (dmabuf). Instead, scan the scatterlist for any small sizes and only go the swiotlb path if any element of the list needs bouncing (note that iommu_dma_map_page() would still only bounce those buffers which are not DMA-aligned). To avoid scanning the scatterlist on the 'sync' operations, introduce a SG_DMA_BOUNCED flag set during the iommu_dma_map_sg() call (suggested by Robin Murphy). Signed-off-by: Catalin Marinas Cc: Joerg Roedel Cc: Christoph Hellwig Cc: Robin Murphy Signed-off-by: Robin Murphy --- Not entirely sure about this approach but here it is. And it needs better testing. drivers/iommu/dma-iommu.c | 12 ++++++++---- include/linux/dma-map-ops.h | 23 +++++++++++++++++++++++ include/linux/scatterlist.h | 27 ++++++++++++++++++++++++--- 3 files changed, 55 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 9297b741f5e8..8c80dffe0337 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -948,7 +948,7 @@ static void iommu_dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg; int i; - if (dev_use_swiotlb(dev)) + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sgl)) for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_cpu(dev, sg_dma_address(sg), sg->length, dir); @@ -964,7 +964,7 @@ static void iommu_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg; int i; - if (dev_use_swiotlb(dev)) + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sgl)) for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_device(dev, sg_dma_address(sg), @@ -990,7 +990,8 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, * If both the physical buffer start address and size are * page aligned, we don't need to use a bounce page. */ - if (dev_use_swiotlb(dev) && iova_offset(iovad, phys | size)) { + if ((dev_use_swiotlb(dev) && iova_offset(iovad, phys | size)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { void *padding_start; size_t padding_size, aligned_size; @@ -1202,7 +1203,10 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, goto out; } - if (dev_use_swiotlb(dev)) + if (dma_sg_kmalloc_needs_bounce(dev, sg, nents, dir)) + sg_dma_mark_bounced(sg); + + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sg)) return iommu_dma_map_sg_swiotlb(dev, sg, nents, dir, attrs); if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 785f7aa90f57..e747a46261d4 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -302,6 +302,29 @@ static inline bool dma_kmalloc_needs_bounce(struct device *dev, size_t size, return true; } +/* + * Return true if any of the scatterlist elements needs bouncing due to + * potentially originating from a small kmalloc() cache. + */ +static inline bool dma_sg_kmalloc_needs_bounce(struct device *dev, + struct scatterlist *sg, int nents, + enum dma_data_direction dir) +{ + struct scatterlist *s; + int i; + + if (!IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) || + dir == DMA_TO_DEVICE || dev_is_dma_coherent(dev)) + return false; + + for_each_sg(sg, s, nents, i) { + if (dma_kmalloc_needs_bounce(dev, s->length, dir)) + return true; + } + + return false; +} + void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs); void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index 375a5e90d86a..f16cf040fe2c 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -16,7 +16,7 @@ struct scatterlist { #ifdef CONFIG_NEED_SG_DMA_LENGTH unsigned int dma_length; #endif -#ifdef CONFIG_PCI_P2PDMA +#if defined(CONFIG_PCI_P2PDMA) || defined(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) unsigned int dma_flags; #endif }; @@ -248,6 +248,29 @@ static inline void sg_unmark_end(struct scatterlist *sg) sg->page_link &= ~SG_END; } +#define SG_DMA_BUS_ADDRESS (1 << 0) +#define SG_DMA_BOUNCED (1 << 1) + +#ifdef CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC +static inline bool sg_is_dma_bounced(struct scatterlist *sg) +{ + return sg->dma_flags & SG_DMA_BOUNCED; +} + +static inline void sg_dma_mark_bounced(struct scatterlist *sg) +{ + sg->dma_flags |= SG_DMA_BOUNCED; +} +#else +static inline bool sg_is_dma_bounced(struct scatterlist *sg) +{ + return false; +} +static inline void sg_dma_mark_bounced(struct scatterlist *sg) +{ +} +#endif + /* * CONFGI_PCI_P2PDMA depends on CONFIG_64BIT which means there is 4 bytes * in struct scatterlist (assuming also CONFIG_NEED_SG_DMA_LENGTH is set). @@ -256,8 +279,6 @@ static inline void sg_unmark_end(struct scatterlist *sg) */ #ifdef CONFIG_PCI_P2PDMA -#define SG_DMA_BUS_ADDRESS (1 << 0) - /** * sg_dma_is_bus address - Return whether a given segment was marked * as a bus address From patchwork Sun Nov 6 22:01:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9872EC433FE for ; Sun, 6 Nov 2022 22:04:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9Z1M6NHO3mxzzk4eg4uaTXwT7QEJaHCHaCAQEoLOV4Q=; b=IbRq+8kDpsBo6O 5143AKAXCMCh+AWO8bAjm948wV3uae4MkXIhCxsFRnBD7A7u3jvh7wULmgtuzaGWNBw/wkRwXuOox wN0zWkgtiNsr1h1+8cvNhl+HITLWp/gSZAChYbzQGEWeZpQqZQzRFNcjE7m66mlLbUoCcY6NKGqLl SwZRsjbDctEeQSQHQTpiCWH03LDidyEZirfciNsMqc2QhaZxOpBiuDTtyBu0tIKpN23V5xTeBsxmS hHLTwtpRdkWRfHZev9g9lvZIKmM2sSkwuY3D7WC2g36uu3LAHV7zx25a2LvvUPOK7mNBBBqA/1okS QqI3jd7LeT3foZ1okUiA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornjM-00A9Iw-P2; Sun, 06 Nov 2022 22:03:04 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniS-00A8tq-PU for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:10 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5C4DCB80D3D; Sun, 6 Nov 2022 22:02:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5D987C4314A; Sun, 6 Nov 2022 22:02:02 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 04/13] mm/slab: Allow kmalloc() minimum alignment fallback to dma_get_cache_alignment() Date: Sun, 6 Nov 2022 22:01:34 +0000 Message-Id: <20221106220143.2129263-5-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140209_311368_742611AF X-CRM114-Status: GOOD ( 20.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For architecture with ARCH_DMA_MINALIGN larger than ARCH_KMALLOC_MINALIGN, if no default swiotlb buffer is available, fall back to a kmalloc() minimum alignment which is safe for DMA. Signed-off-by: Catalin Marinas Cc: Andrew Morton --- I'm not particularly fond of the slab_common.c code probing the default swiotlb. In one incarnation I had a __weak arch_kmalloc_minalign() overridden by the arch code but decided there wasn't anything arm64 specific in there. mm/slab.c | 6 +----- mm/slab.h | 2 ++ mm/slab_common.c | 40 +++++++++++++++++++++++++++++++++++----- 3 files changed, 38 insertions(+), 10 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 59c8e28f7b6a..6e31eb027ef6 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1241,11 +1241,7 @@ void __init kmem_cache_init(void) * Initialize the caches that provide memory for the kmem_cache_node * structures first. Without this, further allocations will bug. */ - kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( - kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], - kmalloc_info[INDEX_NODE].size, - ARCH_KMALLOC_FLAGS, 0, - kmalloc_info[INDEX_NODE].size); + new_kmalloc_cache(INDEX_NODE, KMALLOC_NORMAL, ARCH_KMALLOC_FLAGS); slab_state = PARTIAL_NODE; setup_kmalloc_cache_index_table(); diff --git a/mm/slab.h b/mm/slab.h index 0202a8c2f0d2..d0460e0f6760 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -288,6 +288,8 @@ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags); struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size, slab_flags_t flags, unsigned int useroffset, unsigned int usersize); +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, + slab_flags_t flags); extern void create_boot_cache(struct kmem_cache *, const char *name, unsigned int size, slab_flags_t flags, unsigned int useroffset, unsigned int usersize); diff --git a/mm/slab_common.c b/mm/slab_common.c index 33b1886b06eb..b62f27c2dda7 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include #include #include @@ -852,9 +854,30 @@ void __init setup_kmalloc_cache_index_table(void) } } -static void __init +static unsigned int __kmalloc_minalign(void) +{ + int cache_align = dma_get_cache_alignment(); + + /* + * If CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC is not enabled, + * ARCH_KMALLOC_MINALIGN matches ARCH_DMA_MINALIGN. + */ + if (!IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) || + cache_align < ARCH_KMALLOC_MINALIGN || io_tlb_default_mem.nslabs) + return ARCH_KMALLOC_MINALIGN; + + pr_info_once("No default DMA bounce buffer, increasing the kmalloc() minimum alignment to %d\n", + cache_align); + return cache_align; +} + +void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { + unsigned int minalign = __kmalloc_minalign(); + unsigned int aligned_size = kmalloc_info[idx].size; + int aligned_idx = idx; + if (type == KMALLOC_RECLAIM) { flags |= SLAB_RECLAIM_ACCOUNT; } else if (IS_ENABLED(CONFIG_MEMCG_KMEM) && (type == KMALLOC_CGROUP)) { @@ -867,10 +890,17 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) flags |= SLAB_CACHE_DMA; } - kmalloc_caches[type][idx] = create_kmalloc_cache( - kmalloc_info[idx].name[type], - kmalloc_info[idx].size, flags, 0, - kmalloc_info[idx].size); + if (minalign > ARCH_KMALLOC_MINALIGN) { + aligned_size = ALIGN(aligned_size, minalign); + aligned_idx = __kmalloc_index(aligned_size, false); + } + + if (!kmalloc_caches[type][aligned_idx]) + kmalloc_caches[type][aligned_idx] = create_kmalloc_cache( + kmalloc_info[aligned_idx].name[type], + aligned_size, flags, 0, aligned_size); + if (idx != aligned_idx) + kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx]; /* * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for From patchwork Sun Nov 6 22:01:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCB83C4332F for ; Sun, 6 Nov 2022 22:04:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zP8ePufOs989muRPZYrcJtE5bWV/anv9hQlk304Utu8=; b=Z8ybzAnJWuQMKx 90nv1XnG2X72UB+IkuDQV9Q6Vm9hMHJRoTJWpNk4+XhkAz9FV22NJT+Jn8G407Z0LOwbrJCBiW9w5 pcI+OnYWpmWh6zs0Gb9raPXnIH5A7/wrBQ5Ze7/P02tBeB4gHRqfPscGisZPyYOZ9py7k9pZtBWA7 MSSpCdyshOjOj/sie5w3h/DTtwnxvBF5nht2DAx6Uzxgi5bIIZjc4OlXvgBfsG9BrBG+OhDzdiEcj rgmX5FSFxeAcDMRAf4YmClxY667XWy5pjGrUn8x9sAJLOeX5UOfUMFE8vUfH7teXOFsc1dtB2FBRb Om8d4LSDyIpQL3LOE5Sw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornjV-00A9M1-Ne; Sun, 06 Nov 2022 22:03:13 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniW-00A8wF-Oo for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:14 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 70413B80D28; Sun, 6 Nov 2022 22:02:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E53CC4314C; Sun, 6 Nov 2022 22:02:06 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 05/13] mm/slab: Simplify create_kmalloc_cache() args and make it static Date: Sun, 6 Nov 2022 22:01:35 +0000 Message-Id: <20221106220143.2129263-6-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140213_153783_24436A9F X-CRM114-Status: GOOD ( 16.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org create_kmalloc_cache() is now only called from new_kmalloc_cache() in the same file, so make it static. In addition, the useroffset argument is always 0 while usersize is the same as size. Remove them. Signed-off-by: Catalin Marinas Cc: Andrew Morton --- mm/slab.h | 3 --- mm/slab_common.c | 11 +++++------ 2 files changed, 5 insertions(+), 9 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index d0460e0f6760..93ba93e7b772 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -285,9 +285,6 @@ gfp_t kmalloc_fix_flags(gfp_t flags); /* Functions provided by the slab allocators */ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags); -struct kmem_cache *create_kmalloc_cache(const char *name, unsigned int size, - slab_flags_t flags, unsigned int useroffset, - unsigned int usersize); void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags); extern void create_boot_cache(struct kmem_cache *, const char *name, diff --git a/mm/slab_common.c b/mm/slab_common.c index b62f27c2dda7..3fe3f4ad1362 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -654,17 +654,16 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->refcount = -1; /* Exempt from merging for now */ } -struct kmem_cache *__init create_kmalloc_cache(const char *name, - unsigned int size, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize) +static struct kmem_cache *__init create_kmalloc_cache(const char *name, + unsigned int size, + slab_flags_t flags) { struct kmem_cache *s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT); if (!s) panic("Out of memory when creating slab %s\n", name); - create_boot_cache(s, name, size, flags | SLAB_KMALLOC, useroffset, - usersize); + create_boot_cache(s, name, size, flags | SLAB_KMALLOC, 0, size); kasan_cache_create_kmalloc(s); list_add(&s->list, &slab_caches); s->refcount = 1; @@ -898,7 +897,7 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) if (!kmalloc_caches[type][aligned_idx]) kmalloc_caches[type][aligned_idx] = create_kmalloc_cache( kmalloc_info[aligned_idx].name[type], - aligned_size, flags, 0, aligned_size); + aligned_size, flags); if (idx != aligned_idx) kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx]; From patchwork Sun Nov 6 22:01:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22646C433FE for ; Sun, 6 Nov 2022 22:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qeCy5wYiR3SVbvXOe5vU+kb/807ZomAUsbhmDgAWjRU=; b=vchx2piFwMBXxz XTgecsIWTidYHCSTz4bronmDF0LMJ18mutbroAB0wGwywliPeUTp+z8P4L1bXcYrmFkQXVlTOBiDk wZd4lMp3DSvAz9p7jlggGTm9+J2hV/xVvZYwopz77jta2qzQMdTGoeNhT5BhMnQ0i2cb5umIlNjKX HEGbgKkLnh534nlsV5P8cMF9ocJZj6SJCATmMiDG4yF+hybUoCEkuXHlj0JQ28wJsnIT1bpnMdBfE 428qtKCgTINEMc8bvmCLVRpwi11+KXKRLewKrYLX1q8RnIgcb2JvGiQn64RrBEC4tdEo/OFYCS5kG ZGm7vJKt4E+L8O01AXog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornjf-00A9Rr-J2; Sun, 06 Nov 2022 22:03:23 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniY-00A8wu-UD for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:16 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 61DA260DCC; Sun, 6 Nov 2022 22:02:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86607C433C1; Sun, 6 Nov 2022 22:02:10 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 06/13] dma: Allow the smaller cache_line_size() returned by dma_get_cache_alignment() Date: Sun, 6 Nov 2022 22:01:36 +0000 Message-Id: <20221106220143.2129263-7-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140215_057429_A06B9CB4 X-CRM114-Status: GOOD ( 14.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On architectures like arm64, ARCH_DMA_MINALIGN is larger than the majority of cache line size configurations. Allow an architecture to opt in to dma_get_cache_alignment() returning such smaller size and select the option for arm64. Signed-off-by: Catalin Marinas Cc: Christoph Hellwig Cc: Robin Murphy Cc: Will Deacon --- Is there any architecture where ARCH_DMA_MINALIGN is larger than cache_line_size()? We could avoid another Kconfig entry. arch/arm64/Kconfig | 1 + include/linux/dma-mapping.h | 2 ++ kernel/dma/Kconfig | 6 ++++++ 3 files changed, 9 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 505c8a1ccbe0..3991cb7b8a33 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -22,6 +22,7 @@ config ARM64 select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE + select ARCH_HAS_DMA_CACHE_LINE_SIZE select ARCH_HAS_DMA_PREP_COHERENT select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_FAST_MULTIPLIER diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 3288a1339271..b29124341317 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -545,6 +545,8 @@ static inline int dma_set_min_align_mask(struct device *dev, static inline int dma_get_cache_alignment(void) { + if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CACHE_LINE_SIZE)) + return cache_line_size(); #ifdef ARCH_HAS_DMA_MINALIGN return ARCH_DMA_MINALIGN; #endif diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index 56866aaa2ae1..d6fab8e3cbae 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -76,6 +76,12 @@ config ARCH_HAS_DMA_PREP_COHERENT config ARCH_HAS_FORCE_DMA_UNENCRYPTED bool +config ARCH_HAS_DMA_CACHE_LINE_SIZE + bool + help + Select if the architecture has non-coherent DMA and + cache_line_size() is a safe alignment for DMA buffers. + config SWIOTLB bool select NEED_DMA_MAP_STATE From patchwork Sun Nov 6 22:01:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2FD5EC4332F for ; Sun, 6 Nov 2022 22:04:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=b+Y10mZmL6n8gVj8xx2bezyEXhWY80k5okPnQXDKTmA=; b=tWshdBAJnTMAfN KtmMb8ZlCzjApe6NFoQixbC4CYCRaefPReLKtZViy42vj7o8bidgRabZC9LFIn1eEgKI59YB1xp9h dzZez8CrIBOabqmT3VP4O2RTTL3zaXcyUv16FEGD1X0gvIAhQWb0g+qlgYsMIbyhQ3j7xKJYEiXLO VPQRdSskl4l3DqzDUmXVg/TTDZRTn/0QLz4/l3b5hlDUO1FRIZExsBWONTA0ywEnrBS/fkn/e0oey ubqXhyki/l0ib+VvHFXGwrxVEB/Y9PFqGNTeZxNR0cWIu7i07EsNj4NLx5lR6FMxzTJr9odAGuqtl YkTLE2LToe/yEkrS+hIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornjq-00A9Za-Kw; Sun, 06 Nov 2022 22:03:34 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornid-00A8xb-AA for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:20 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D6DB960DE9; Sun, 6 Nov 2022 22:02:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 95051C43147; Sun, 6 Nov 2022 22:02:14 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 07/13] drivers/base: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Date: Sun, 6 Nov 2022 22:01:37 +0000 Message-Id: <20221106220143.2129263-8-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140219_460467_30351997 X-CRM114-Status: GOOD ( 14.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects alignment. Signed-off-by: Catalin Marinas Cc: Greg Kroah-Hartman Cc: "Rafael J. Wysocki" --- drivers/base/devres.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/base/devres.c b/drivers/base/devres.c index 4ab2b50ee38f..5dbfa3c31af6 100644 --- a/drivers/base/devres.c +++ b/drivers/base/devres.c @@ -29,10 +29,10 @@ struct devres { * Some archs want to perform DMA into kmalloc caches * and need a guaranteed alignment larger than * the alignment of a 64-bit integer. - * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same - * buffer alignment as if it was allocated by plain kmalloc(). + * Thus we use ARCH_DMA_MINALIGN for data[] which will force the same + * alignment for struct devres when allocated by kmalloc(). */ - u8 __aligned(ARCH_KMALLOC_MINALIGN) data[]; + u8 __aligned(ARCH_DMA_MINALIGN) data[]; }; struct devres_group { From patchwork Sun Nov 6 22:01:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BAB20C4332F for ; Sun, 6 Nov 2022 22:05:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rID3ZfXI+jQZsG8IcXC56CbfgMKJZhEo/NdK5yU+vmQ=; b=yJHBdLmZRjTouw S1IHbwASCmo/+IMOA9Al6Lv2xvzJOHX0zTK3vD8jdXnxopDkjd+dMTius/ZSo1lHM+iZhxsTlzar5 sO1hPuAF+O0keuhPewxqnyFyV/0xZG2X1dISV1KoQJlIrIDayeaCZglHd0qXoVRtO7hu1Ebpx32eY Wa8Ps0hn0ypNS/n7yPaUUZSmXpjV9aBS9DXvH/0wdwKPk+Ja8BXi7YXOxO9XbD9K22P346PD4mh2B tY1MlP/JEmTOfRSB0w8MXNovu83EdWkXlL8ELfEo5jBhktbEUMCVwXySsZ81oHPc13KMMHQaDYVIO Wm5unZiGApDEu0SONAGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornlF-00AANs-OB; Sun, 06 Nov 2022 22:05:01 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornij-00A90B-1V for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:43 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B9058B80D3E; Sun, 6 Nov 2022 22:02:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7ECBC43149; Sun, 6 Nov 2022 22:02:18 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 08/13] drivers/gpu: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Date: Sun, 6 Nov 2022 22:01:38 +0000 Message-Id: <20221106220143.2129263-9-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140240_183971_69EB9D5D X-CRM114-Status: GOOD ( 14.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects alignment. Signed-off-by: Catalin Marinas Cc: Daniel Vetter --- drivers/gpu/drm/drm_managed.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/drm_managed.c b/drivers/gpu/drm/drm_managed.c index 4cf214de50c4..3a5802f60e65 100644 --- a/drivers/gpu/drm/drm_managed.c +++ b/drivers/gpu/drm/drm_managed.c @@ -49,10 +49,10 @@ struct drmres { * Some archs want to perform DMA into kmalloc caches * and need a guaranteed alignment larger than * the alignment of a 64-bit integer. - * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same - * buffer alignment as if it was allocated by plain kmalloc(). + * Thus we use ARCH_DMA_MINALIGN for data[] which will force the same + * alignment for struct drmres when allocated by kmalloc(). */ - u8 __aligned(ARCH_KMALLOC_MINALIGN) data[]; + u8 __aligned(ARCH_DMA_MINALIGN) data[]; }; static void free_dr(struct drmres *dr) From patchwork Sun Nov 6 22:01:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DF07AC433FE for ; Sun, 6 Nov 2022 22:05:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IPU3wIbDWdUzjzYRcoG8sSBFG50YgCSgyD6LTe9izWs=; b=PV4jvtucR1xqUx qMN/2mapj+M6Weh8wgNI0f6TItpCZIdu+wECTmf73Qd4eb404+3LUFEKEiV4CYF5j5prnBxMJrjVt vLsGfkHif8fyhwze1BtrfLAXjHMjBk8WsJs5pSNi/80/eyta8L0lVbFuRaABx/zZlr2DL1KDySkyM 8sqL+DapoJMjGJ5vkk5aUJd/OHfAfAbCMQNIZBeIJIXCcC/ILeNAnMsafvHEjt+8dKPBC5oFbrIpQ FPX6m1WqLenWVGxgzLcnmySDavSwURBF+RQlNbqeEmbstpAx/xrqzE5bdqZyQGJsy8a/0zzaiP8EL eYnQQ6xv54bGixlRpJKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornkQ-00A9vY-Fs; Sun, 06 Nov 2022 22:04:10 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornin-00A91D-4C for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:40 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C2D38B80D48; Sun, 6 Nov 2022 22:02:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CA6E9C433D6; Sun, 6 Nov 2022 22:02:22 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 09/13] drivers/usb: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Date: Sun, 6 Nov 2022 22:01:39 +0000 Message-Id: <20221106220143.2129263-10-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140239_422870_38D81911 X-CRM114-Status: GOOD ( 13.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects alignment. Signed-off-by: Catalin Marinas Cc: Greg Kroah-Hartman --- drivers/usb/core/buffer.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/usb/core/buffer.c b/drivers/usb/core/buffer.c index fbb087b728dc..e21d8d106977 100644 --- a/drivers/usb/core/buffer.c +++ b/drivers/usb/core/buffer.c @@ -34,13 +34,13 @@ void __init usb_init_pool_max(void) { /* * The pool_max values must never be smaller than - * ARCH_KMALLOC_MINALIGN. + * ARCH_DMA_MINALIGN. */ - if (ARCH_KMALLOC_MINALIGN <= 32) + if (ARCH_DMA_MINALIGN <= 32) ; /* Original value is okay */ - else if (ARCH_KMALLOC_MINALIGN <= 64) + else if (ARCH_DMA_MINALIGN <= 64) pool_max[0] = 64; - else if (ARCH_KMALLOC_MINALIGN <= 128) + else if (ARCH_DMA_MINALIGN <= 128) pool_max[0] = 0; /* Don't use this pool */ else BUILD_BUG(); /* We don't allow this */ From patchwork Sun Nov 6 22:01:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7418DC4332F for ; Sun, 6 Nov 2022 22:05:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8u7tuXtuAw/pDE0/GimIWcFqDMaqY822Sm6BydsxtrE=; b=uaFN0OGADBDGTV UE/OeINLf6CZ5xu16H2TVbBT6OyyQetUWoqParRhagCtpE4MIokp11tB2HzgMvCeC2g9zGUYLCS0/ QZjCUZo/OHgh81XKnFOSt3u9maRRpwQQ56EtvDAU1++nNc7V5zS4qBocIyIF7awITyDptak620ZHD 93onyqn/XpBipTPT7w+5Cps6dEGFmnWXKw05gepYzhwXRIIDC0x1Lw7d3VQJCxL5ELcgroCOqFQaG ZDMT6T62RJPgvVkumH8AZwJCujAcSwBBsR5rJxsoZFgZo3DxkwiQSMDswrxKyUdw2+t9SNbqFwmDv S4ZyHcytd4imIH0naFig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornkt-00AAAv-WD; Sun, 06 Nov 2022 22:04:40 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornir-00A92F-4t for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:42 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CB7D1B80D2E; Sun, 6 Nov 2022 22:02:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8178C43470; Sun, 6 Nov 2022 22:02:26 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 10/13] drivers/spi: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Date: Sun, 6 Nov 2022 22:01:40 +0000 Message-Id: <20221106220143.2129263-11-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140240_184342_871A0AAB X-CRM114-Status: GOOD ( 13.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects alignment. Signed-off-by: Catalin Marinas Cc: Mark Brown Acked-by: Mark Brown --- drivers/spi/spidev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/spi/spidev.c b/drivers/spi/spidev.c index b2775d82d2d7..f74486240b1f 100644 --- a/drivers/spi/spidev.c +++ b/drivers/spi/spidev.c @@ -228,7 +228,7 @@ static int spidev_message(struct spidev_data *spidev, /* Ensure that also following allocations from rx_buf/tx_buf will meet * DMA alignment requirements. */ - unsigned int len_aligned = ALIGN(u_tmp->len, ARCH_KMALLOC_MINALIGN); + unsigned int len_aligned = ALIGN(u_tmp->len, ARCH_DMA_MINALIGN); k_tmp->len = u_tmp->len; From patchwork Sun Nov 6 22:01:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13CBAC433FE for ; Sun, 6 Nov 2022 22:04:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ttrOUwcwHk76ff/TTznxDJrwleDWUfR0KUamLSvvgU8=; b=NM7AzxY22DeXl9 XDrBHGWLwIpi+y3cmC1uiys+F4sJP5275qlA8GtaOSNvQF8md3+CJZwUYWBw9/ILcfsw0iKL8EwAw qxeTxDpbCtKoi3e+K3GnQkD0ImACJVyw3F2BGimIc6kEk+Y2/lWFoWJvYlElQ/KgSCQr/102BvA6C 1kg6p0tZSneMYVUIE3RqSBB1wzpDFtn1wylpfSXY1sIfK++JvO1C2UMFU4cTivutzixAagBw8u/MB O8r1EzygX2ZCqAXcQp38pxFFCJl0UvAcrxK73CUD7EXI0ep6czsEhKmBKyw7AQPZOglOyfaW63mTw MXnWPxYnqHkE1B0jS18w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornk5-00A9iJ-Mc; Sun, 06 Nov 2022 22:03:49 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniv-00A93t-Ny for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:39 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 27EA860DEA; Sun, 6 Nov 2022 22:02:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4456C43148; Sun, 6 Nov 2022 22:02:30 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 11/13] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Date: Sun, 6 Nov 2022 22:01:41 +0000 Message-Id: <20221106220143.2129263-12-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140237_832023_1435CC52 X-CRM114-Status: GOOD ( 16.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() alignment. This will ensure that the static alignment of various structures or members of those structures( e.g. __ctx[] in struct aead_request) is safe for DMA. Note that sizeof such structures becomes aligned to ARCH_DMA_MINALIGN and kmalloc() will honour such alignment, so there is no confusion for the compiler. Signed-off-by: Catalin Marinas Cc: Herbert Xu Cc: Ard Biesheuvel --- I know Herbert NAK'ed this patch but I'm still keeping it here temporarily, until we agree on some refactoring at the crypto code. FTR, I don't think there's anything wrong with this patch since kmalloc() will return ARCH_DMA_MINALIGN-aligned objects if the sizeof such objects is a multiple of ARCH_DMA_MINALIGN (side-effect of CRYPTO_MINALIGN_ATTR). include/linux/crypto.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/crypto.h b/include/linux/crypto.h index 2324ab6f1846..654b9c355575 100644 --- a/include/linux/crypto.h +++ b/include/linux/crypto.h @@ -167,7 +167,7 @@ * maintenance for non-coherent DMA (cache invalidation in particular) does not * affect data that may be accessed by the CPU concurrently. */ -#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN +#define CRYPTO_MINALIGN ARCH_DMA_MINALIGN #define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN))) From patchwork Sun Nov 6 22:01:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E108C433FE for ; Sun, 6 Nov 2022 22:06:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/k8cPol78qS5TSUO56+xWedy+gl1e+0yROSeOcU+BQM=; b=FpfAf2WwqjvvjJ 2n+FtuodLTCij8frBEQXuUrT+PYdrdwipx86faBDoj5wZZ2GOgnXDlrBpc7nqTdQwChjtIoNqUBOU WLlD4noqsvzM7CjTYYMYvh2KX8tN2mkpo/V+BPXQKCU8hqpSZIb/rftdrL9653NkFKa9JEUdnv6eu 3tLSRBhf8zRM1Y6MjCbScFTie8pOKmvTeIGzDG8bxy6LukxYwkQ3w+5nQCtHxnh1JO0AGBUUkCi7M PeDB7Fu06aomYzOCiU2Qr7gMSQTdfzHdvH1d2mXNGCwHUfk2G7bIKZ3pbK4QBZE0mn9Fkk3zuIrTa 0Zs9A0aBJ0+tw8RVsrPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornlV-00AAXf-Ij; Sun, 06 Nov 2022 22:05:17 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1orniy-00A95H-J5 for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:44 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A78D660DF0; Sun, 6 Nov 2022 22:02:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F14A5C43149; Sun, 6 Nov 2022 22:02:34 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 12/13] drivers/md: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN Date: Sun, 6 Nov 2022 22:01:42 +0000 Message-Id: <20221106220143.2129263-13-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140240_686998_C11BA0E4 X-CRM114-Status: GOOD ( 14.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations while ARCH_KMALLOC_MINALIGN is the minimum kmalloc() objects alignment. Signed-off-by: Catalin Marinas Cc: Alasdair Kergon Cc: Mike Snitzer --- This is somewhat related to the previous crypto patch. The dm_crypt_io structure is supposed to be CRYPTO_MINALIGN-aligned. drivers/md/dm-crypt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 159c6806c19b..4f1bd5b348c3 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -3250,7 +3250,7 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv) cc->per_bio_data_size = ti->per_io_data_size = ALIGN(sizeof(struct dm_crypt_io) + cc->dmreq_start + additional_req_size, - ARCH_KMALLOC_MINALIGN); + ARCH_DMA_MINALIGN); ret = mempool_init(&cc->page_pool, BIO_MAX_VECS, crypt_page_alloc, crypt_page_free, cc); if (ret) { From patchwork Sun Nov 6 22:01:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13033647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 28FF7C433FE for ; Sun, 6 Nov 2022 22:06:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JPRwPE0+ZVuPwHwQafs12AtkOBBesLJOj5ef/wpeWn8=; b=WI/xco5tsui+4x zWI7pmnN2r9+HLc+Gi0nJWNBP5ffZ/R97CwSbIp3o37gnM1v0BY/ffTqLZ+gbkkCBE92+p7BZDbza GFzkK0Qjj9NnpEZ2o/gqX8lvXN5CWocOtwcLGrBgtlcKo1Qxqsm/+Ym6cgrZMd2Jx6DhBZnZkKl6Q 8Go74VWmsS8M58inc2FhiwoIzXFsZk+0HgRjWggqrMfg2j3UtfFBzT1b8kfjX4eA15xnik0WHBXUc 3RbNZFBmKRYcURDDGSW0szW1qXOUYYQif9CWzA/itc+AXnToVb6mJGmCFAtX3ytNSlXhuKOtok5P9 0WPuB04M77YdIDT8Y0uQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornlw-00AAq4-I9; Sun, 06 Nov 2022 22:05:44 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ornj3-00A9AB-9l for linux-arm-kernel@lists.infradead.org; Sun, 06 Nov 2022 22:02:46 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EFA1CB80D4A; Sun, 6 Nov 2022 22:02:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A241C4314C; Sun, 6 Nov 2022 22:02:38 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 13/13] dma: arm64: Add CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC and enable it for arm64 Date: Sun, 6 Nov 2022 22:01:43 +0000 Message-Id: <20221106220143.2129263-14-catalin.marinas@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221106220143.2129263-1-catalin.marinas@arm.com> References: <20221106220143.2129263-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221106_140245_563045_14D3B5A3 X-CRM114-Status: GOOD ( 13.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With all the infrastructure in place for bouncing small kmalloc() buffers, add the corresponding Kconfig entry and select it for arm64. Signed-off-by: Catalin Marinas Cc: Will Deacon Cc: Christoph Hellwig Cc: Robin Murphy --- arch/arm64/Kconfig | 1 + kernel/dma/Kconfig | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3991cb7b8a33..f889cf16e6ab 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -100,6 +100,7 @@ config ARM64 select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + select ARCH_WANT_KMALLOC_DMA_BOUNCE select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index d6fab8e3cbae..b56e76371023 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -86,6 +86,14 @@ config SWIOTLB bool select NEED_DMA_MAP_STATE +config ARCH_WANT_KMALLOC_DMA_BOUNCE + bool + +config DMA_BOUNCE_UNALIGNED_KMALLOC + def_bool y + depends on ARCH_WANT_KMALLOC_DMA_BOUNCE + depends on SWIOTLB && !SLOB + config DMA_RESTRICTED_POOL bool "DMA Restricted Pool" depends on OF && OF_RESERVED_MEM && SWIOTLB