From patchwork Thu May 18 17:34:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13247178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69AF2C7EE2A for ; Thu, 18 May 2023 17:35:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B0D9900012; Thu, 18 May 2023 13:35:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 061A5900003; Thu, 18 May 2023 13:35:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6B7B900012; Thu, 18 May 2023 13:35:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D915B900003 for ; Thu, 18 May 2023 13:35:00 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B0588C0858 for ; Thu, 18 May 2023 17:35:00 +0000 (UTC) X-FDA: 80804076360.25.AAC00CE Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id 0B93E140005 for ; Thu, 18 May 2023 17:34:58 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none); spf=pass (imf09.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684431299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Po7IyUSvGYuNcwSRGXH+yWudEHSJDb5jHBbVqoilbOw=; b=NmoYtcV/xeO68kt1NyiJ+na12qbYb7EdD59MEZzvQEvf0V1mos3meaCFU5/75mJyUMwEii couScUJtUiUHbAABHxQ4wj/hcAHUf0ky36AnuWSiPN+VhOc2FyD/vHQT5INJuu/9gDoCV8 r4E2+K/2puevoNJu6/pItDktdgJejxQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none); spf=pass (imf09.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684431299; a=rsa-sha256; cv=none; b=0YAtZRFkSFXmREzOdvm9ab6c68m26K1KaCzGdQ4CAuBrGP8fXKMbYNVP+2Va7Oa3e/WzZq KUksa36rANswRpQSSZzAsW7DlPWWJ6HtD615Sr3nWlO7prO0HVxVu+6n5ZdTNW5aRYLSWp RI5caQUqnZqXYmKyUe+Y/6h/4zI/r1E= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 206CB642F3; Thu, 18 May 2023 17:34:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 440A6C433A8; Thu, 18 May 2023 17:34:54 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v4 12/15] dma-mapping: Force bouncing if the kmalloc() size is not cache-line-aligned Date: Thu, 18 May 2023 18:34:00 +0100 Message-Id: <20230518173403.1150549-13-catalin.marinas@arm.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230518173403.1150549-1-catalin.marinas@arm.com> References: <20230518173403.1150549-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0B93E140005 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: kjabt8j8o3iuuyygep7jjhfnyoswwnf5 X-HE-Tag: 1684431298-263971 X-HE-Meta: U2FsdGVkX1+Vp3KABr1KHik5XYFrml4JDgmD7A+6kO73EXEAFU/0NsIAmOC/6uDeOBquyu24lvJtO/ZSpR4uS3TQYGJza0Lx/c99r2MdI3o4rTNCc84FelkSHtcW+HWIixOWPVpRY0nTpxXVKNYNHUzrWFnkPEYISr7Kl3hcMxaMg5Mr5gYf0AatoIMhRu1ebACihGZhOuJXMrHOUM/l1Ar7gs4En3Lh227rQLXPzuCNjX2zu5+D/tmSJJIDUO6BBCCifl1Ic2T/dyyQW3HZYcLu1IHV+npbzqGuIded1CufE8FhiINWkiKBrGpDQ1ZmrY7eaOHSqH+8odEz6a8NSwTC5w2mjei0+CN6cKCd8YYzPEyyrBc6ObpaRDoKZKNdLdwjfxCtzPqJ7BVF8OvnuUY7i/yNMHbSD+rlxayA7etFyOsKtyR/Ui/D60A5pCNmtPkgWnaFnUjx7scm6bIfMifKPK0hvA4py3dbE7IheUVq8U5XUG8EnLX4lYxbPoJ2vWwG4DQXtSdDcf71NspNRChfDaS4BEhENDq47TiUHNLa99jDsai270RFxdDqcEcHJ116eg3E7Td1UBIwbTneaLRKJrqw4TPnXvg/U5L6bbZxb8GmpDzM1BYI6TORnZHVXKliiEKRI6a/tZRXEcA72XtWZk73wivHo6/+8Ss2/x7pddzkU45oYHd0oH9phLBFrB3yjctkN1iplTvK7MGzqRxqC77+vxEzIkzQ92IPHcapDXhs8u5DEO92uGcUA2rnwio3jluzSSikPmcU5qPqc0aaj+56+3CmKLDa4pflRQfWwCMcwyA78/TmVP1Essw+u+yM8vTq7eStQ7JQADv8n8089kfl1Wq32TCBuCFUSs2qYboNQv/LMSh200mkWPoP7eeUmVxjOVRjPPgPxaE1WJB38RCtbNEiT7+lMJ6fZDlNQdt7zMvgk0xDNguhgB6ydfiwGb3bnvsZix+h6YO LSujV97F p+Ethr33eZPIoEgNLhAsxn6kMG846DfmOMi8uZRZQ7QvL10Bua+jsSXhpnXV3cDyhe/V6ns2ULxtCucpa2jqal458y1BT2cgZ8yGIQ590QEHboCnY/PhJjzY47x+93ZP2s5qNPrmqiKnRVLOAyD5WNrekVa00pPJOjpD1W00S6Zy1eeo9J7TxF/9ZjDdQawmnfwz5UDGcFHSM45n20G/7XRNg6Ea78LjdbpxyTh/k/SYcG/dWDB6oncCjc1Uf5wNMQqLhYaz3of8l0hvQenhvhGdTYsxcKWvMfIBwYrf69bK2YPnfT6rISGuSn6+zjJCguZojE491FLvtjcdxLgyKsG43O7cO1D/t1NSYvlfGkiFRGZTKtYFvDyyNYb2CKgL1+DI2y6qIBbXBJyRdmHMEKrrYtSXgNXdczMmyGaVAUmEZsBwTSduE1TkedC9EIOSTvqtoWonO2DREUJiIbJgNE6G2+yFxSO/SlILEyC8IYePTJrWuqBvoBSnrU+PAoBRo7U4S X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For direct DMA, if the size is small enough to have originated from a kmalloc() cache below ARCH_DMA_MINALIGN, check its alignment against dma_get_cache_alignment() and bounce if necessary. For larger sizes, it is the responsibility of the DMA API caller to ensure proper alignment. At this point, the kmalloc() caches are properly aligned but this will change in a subsequent patch. Architectures can opt in by selecting ARCH_WANT_KMALLOC_DMA_BOUNCE. Signed-off-by: Catalin Marinas Cc: Christoph Hellwig Cc: Robin Murphy Reviewed-by: Christoph Hellwig --- include/linux/dma-map-ops.h | 48 +++++++++++++++++++++++++++++++++++++ kernel/dma/Kconfig | 9 +++++++ kernel/dma/direct.h | 3 ++- 3 files changed, 59 insertions(+), 1 deletion(-) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 31f114f486c4..43bf50c35e14 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -8,6 +8,7 @@ #include #include +#include struct cma; @@ -277,6 +278,53 @@ static inline bool dev_is_dma_coherent(struct device *dev) } #endif /* CONFIG_ARCH_HAS_DMA_COHERENCE_H */ +/* + * Check whether potential kmalloc() buffers are safe for non-coherent DMA. + */ +static inline bool dma_kmalloc_safe(struct device *dev, + enum dma_data_direction dir) +{ + /* + * If DMA bouncing of kmalloc() buffers is disabled, the kmalloc() + * caches have already been aligned to a DMA-safe size. + */ + if (!IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC)) + return true; + + /* + * kmalloc() buffers are DMA-safe irrespective of size if the device + * is coherent or the direction is DMA_TO_DEVICE (non-desctructive + * cache maintenance and benign cache line evictions). + */ + if (dev_is_dma_coherent(dev) || dir == DMA_TO_DEVICE) + return true; + + return false; +} + +/* + * Check whether the given size, assuming it is for a kmalloc()'ed buffer, is + * sufficiently aligned for non-coherent DMA. + */ +static inline bool dma_kmalloc_size_aligned(size_t size) +{ + /* + * Larger kmalloc() sizes are guaranteed to be aligned to + * ARCH_DMA_MINALIGN. + */ + if (size >= 2 * ARCH_DMA_MINALIGN || + IS_ALIGNED(kmalloc_size_roundup(size), dma_get_cache_alignment())) + return true; + + return false; +} + +static inline bool dma_kmalloc_needs_bounce(struct device *dev, size_t size, + enum dma_data_direction dir) +{ + return !dma_kmalloc_safe(dev, dir) && !dma_kmalloc_size_aligned(size); +} + void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs); void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index 3e2aab296986..18dd03c74734 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -97,6 +97,15 @@ config SWIOTLB bool select NEED_DMA_MAP_STATE +config ARCH_WANT_KMALLOC_DMA_BOUNCE + bool + +config DMA_BOUNCE_UNALIGNED_KMALLOC + def_bool y + depends on ARCH_WANT_KMALLOC_DMA_BOUNCE + depends on SWIOTLB + select NEED_SG_DMA_FLAGS + config DMA_RESTRICTED_POOL bool "DMA Restricted Pool" depends on OF && OF_RESERVED_MEM && SWIOTLB diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index e38ffc5e6bdd..97ec892ea0b5 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -94,7 +94,8 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev, return swiotlb_map(dev, phys, size, dir, attrs); } - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (unlikely(!dma_capable(dev, dma_addr, size, true)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { if (is_pci_p2pdma_page(page)) return DMA_MAPPING_ERROR; if (is_swiotlb_active(dev))