From patchwork Wed May 31 15:48:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13262477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9DE1C77B7A for ; Wed, 31 May 2023 15:49:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79D54900002; Wed, 31 May 2023 11:49:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74CC68E0003; Wed, 31 May 2023 11:49:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61480900002; Wed, 31 May 2023 11:49:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 513858E0003 for ; Wed, 31 May 2023 11:49:43 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 08CC040277 for ; Wed, 31 May 2023 15:49:43 +0000 (UTC) X-FDA: 80850985446.08.5C35188 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf28.hostedemail.com (Postfix) with ESMTP id 54963C001C for ; Wed, 31 May 2023 15:49:41 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685548181; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=trBjKA2P3t7Lv44FdRPNpP5ETnPfkIlOly8ehL5hPTw=; b=b88AmAW0nMxrolBn6lKgNzkgylpj38S2V40TcbjMdDlicck6apt453WYgys3saU42MZ4Gt YO1CDUsiC5NeYR2G29UhigAxPKb8jBuAA3G1q2QlSibokUWMBT7gsKi5xbqEqgkNC2qlVO tbHtMQuWSZ4cJ8XE5cC9bHjeRAwnF2E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685548181; a=rsa-sha256; cv=none; b=5DVb4BA7mVTMV2JFfw8XDC78YwivqpXW1Z5/5SdTtdNThEjdgUdlYtE8jh5yUF2f5wEYer etdMHzSW2nxSdqBK2SAYBlPCQIieQxPrmOY6mrdbIJ6knu+YtBrDqoE0R/Ig/hJae3imY2 iQpr06EfWj3jFLXF2VRyjCMMSXr9j9Q= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6AFC663D9B; Wed, 31 May 2023 15:49:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26227C4339E; Wed, 31 May 2023 15:49:36 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Christoph Hellwig , Robin Murphy Cc: Arnd Bergmann , Greg Kroah-Hartman , Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Jonathan Cameron , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v6 14/17] dma-mapping: Force bouncing if the kmalloc() size is not cache-line-aligned Date: Wed, 31 May 2023 16:48:33 +0100 Message-Id: <20230531154836.1366225-15-catalin.marinas@arm.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230531154836.1366225-1-catalin.marinas@arm.com> References: <20230531154836.1366225-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 54963C001C X-Rspam-User: X-Stat-Signature: nyixsqouzi41eb97rers1hqa4in99tb4 X-Rspamd-Server: rspam03 X-HE-Tag: 1685548181-792124 X-HE-Meta: U2FsdGVkX1/ElOjxQGD7K+OztTcG+Ej/I795vBD8mIh8x+jqA5X+gYum5UpLB7eKv95rEa6vYi9xiLMXFFQeO0qkl3yuj6anyHJpu3EILL/pT+yAx42WI+8ybEZ6gyDH3KNg+ghaIQ6o+y/j/Wcl9uZca3PM2S3tfuLFuUuESPquYUA7LrONOoeh5InpK4hzikCqu25AQZg8elzAi6/oZqZZ1rDqw9Hxwriv30Vt+7qr6D2Pl5YCwRsiK96+3i8yEZkDuHs0zE+4ndK557YCd/RaoZGWCuLVMVV1ZdGrJYAQLOZpvslUNqCyTL2mQtfChWhVsx6sqGDMy6EKxsrRKd1NV0MyAOgQQaoVZQGtFjnpFOawrXEkFlus1Vc8wKpWJ/dgKKY0qif6PfllYVuodqtFzJgL6nVi747PoB2wgIcHQ+jwo2vY+NimB+5AupjFjnX6+/fjYy+96B2a6etXKfele4E6v+TxeOM1CjZuiqs458sJbtrTZJMRXN+FjXKVeX5CkkqqkFajo/78CJbtVQfAIoNSM6+H9dRplwbnCD5WJvviis7vTbh/E2NuggXoWHp8BV97Gps9WRnjN6vC9k+B+IzZIj1xL5BzwXX3EktqYQDJFmXllpRT5DC9LqrLDv9A/7KFkC4pM+1EsJM7rsz5Jlgn5QfTHouXxaJbd/pbotXvt2CW6pCv4M/xVko8sMSCsqrEUDAew5mKPKx4PCRybIpGyOkCPE5UsJZk8E9kFKRse5N0pZFmEQgxNcPjXwU9bzwhUIkWGsYVL+OotZIPfSdsl0mHZh8tuV5ejMbLloiSs0SaP7QFj/YQnnwfAkHqyap9HFJAUetdI4olE0ghEciiE+DRDRF4ukKeJDgvYYZQTYe/cvRsAbHPt7NUYbKTzK5vrg+3mJ8BVSNZcipSWML5RjVLeHspoCFdZaUZ6zFNpb7Gz73qkdjMTBlrC9NV3YsuA0gBNR4T8Eg O8sjcLRx bMsoagtuwUAfQVGaPnKOHqQQrVQSdBbIKZ18NEXM4GvIYfYhsbiCCP7fQ6ot4/8iUkduEKSwmIU7HVf4Y9EY7DLKoDdMAJ9j0QaNnVGOy+tzXbrPXvBo9pbKf+JkHOw40XshszRYW7kzU2GWi8bWmtzpfiMdsPYf11RJ8vgEuRyzIuWE3P2mdv3fpLjMu6jCN5Z/EHF+poEYSJZQ7TfnIuSxSN6HdoPrqmEaG6tSQ+1hARx5HTkDLbEolaPd7T8F0OYhF130iKDFMzopV53SMlE3eqg/pfjtEo9fCq4+tSou8I+qM9FIDFljuSz1wCJ7ZkyCko9T6xsoCjWIx1VyrXzb2ga47dCTt5aXjs6tAbwH7Fe77dhVVOO4Z9QAD97FZCfvrdnUxT1GzXI+ZwK5TaEvvLeAo4eTCQwa3zHo1iO2vddARq3cMS7J41b/tWo6QJxDpRR0zh+XJ8aRYNAJWPIsNPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For direct DMA, if the size is small enough to have originated from a kmalloc() cache below ARCH_DMA_MINALIGN, check its alignment against dma_get_cache_alignment() and bounce if necessary. For larger sizes, it is the responsibility of the DMA API caller to ensure proper alignment. At this point, the kmalloc() caches are properly aligned but this will change in a subsequent patch. Architectures can opt in by selecting ARCH_WANT_KMALLOC_DMA_BOUNCE. Signed-off-by: Catalin Marinas Reviewed-by: Christoph Hellwig Cc: Robin Murphy --- include/linux/dma-map-ops.h | 61 +++++++++++++++++++++++++++++++++++++ kernel/dma/Kconfig | 4 +++ kernel/dma/direct.h | 3 +- 3 files changed, 67 insertions(+), 1 deletion(-) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 31f114f486c4..9bf19b5bf755 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -8,6 +8,7 @@ #include #include +#include struct cma; @@ -277,6 +278,66 @@ static inline bool dev_is_dma_coherent(struct device *dev) } #endif /* CONFIG_ARCH_HAS_DMA_COHERENCE_H */ +/* + * Check whether potential kmalloc() buffers are safe for non-coherent DMA. + */ +static inline bool dma_kmalloc_safe(struct device *dev, + enum dma_data_direction dir) +{ + /* + * If DMA bouncing of kmalloc() buffers is disabled, the kmalloc() + * caches have already been aligned to a DMA-safe size. + */ + if (!IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC)) + return true; + + /* + * kmalloc() buffers are DMA-safe irrespective of size if the device + * is coherent or the direction is DMA_TO_DEVICE (non-desctructive + * cache maintenance and benign cache line evictions). + */ + if (dev_is_dma_coherent(dev) || dir == DMA_TO_DEVICE) + return true; + + return false; +} + +/* + * Check whether the given size, assuming it is for a kmalloc()'ed buffer, is + * sufficiently aligned for non-coherent DMA. + */ +static inline bool dma_kmalloc_size_aligned(size_t size) +{ + /* + * Larger kmalloc() sizes are guaranteed to be aligned to + * ARCH_DMA_MINALIGN. + */ + if (size >= 2 * ARCH_DMA_MINALIGN || + IS_ALIGNED(kmalloc_size_roundup(size), dma_get_cache_alignment())) + return true; + + return false; +} + +/* + * Check whether the given object size may have originated from a kmalloc() + * buffer with a slab alignment below the DMA-safe alignment and needs + * bouncing for non-coherent DMA. The pointer alignment is not considered and + * in-structure DMA-safe offsets are the responsibility of the caller. Such + * code should use the static ARCH_DMA_MINALIGN for compiler annotations. + * + * The heuristics can have false positives, bouncing unnecessarily, though the + * buffers would be small. False negatives are theoretically possible if, for + * example, multiple small kmalloc() buffers are coalesced into a larger + * buffer that passes the alignment check. There are no such known constructs + * in the kernel. + */ +static inline bool dma_kmalloc_needs_bounce(struct device *dev, size_t size, + enum dma_data_direction dir) +{ + return !dma_kmalloc_safe(dev, dir) && !dma_kmalloc_size_aligned(size); +} + void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs); void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index acc6f231259c..abea1823fe21 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -90,6 +90,10 @@ config SWIOTLB bool select NEED_DMA_MAP_STATE +config DMA_BOUNCE_UNALIGNED_KMALLOC + bool + depends on SWIOTLB + config DMA_RESTRICTED_POOL bool "DMA Restricted Pool" depends on OF && OF_RESERVED_MEM && SWIOTLB diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index e38ffc5e6bdd..97ec892ea0b5 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -94,7 +94,8 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev, return swiotlb_map(dev, phys, size, dir, attrs); } - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (unlikely(!dma_capable(dev, dma_addr, size, true)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { if (is_pci_p2pdma_page(page)) return DMA_MAPPING_ERROR; if (is_swiotlb_active(dev))