From patchwork Mon Jun 12 15:31:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13276871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C02AAC7EE2F for ; Mon, 12 Jun 2023 15:33:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66E6B8E0009; Mon, 12 Jun 2023 11:33:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61DFE8E0005; Mon, 12 Jun 2023 11:33:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BF548E0009; Mon, 12 Jun 2023 11:33:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3D1958E0005 for ; Mon, 12 Jun 2023 11:33:18 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9B0D340288 for ; Mon, 12 Jun 2023 15:33:17 +0000 (UTC) X-FDA: 80894489634.08.B899B77 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf27.hostedemail.com (Postfix) with ESMTP id AF54B4000E for ; Mon, 12 Jun 2023 15:33:15 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none); spf=pass (imf27.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686583995; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5qLXjBEatpN3Ri0xxgcy2oBKkS0dTlGY3nafxGrdKuY=; b=0ippCduo7WojyKbXSAp7ZZ0OaSTq98cVRCdKeMoKr5HC8a2uKoKchLpx49PBt1R5BwnSNX 4G3hgaf7oX2JDaYNibl5d0RgyLW1cSoJz6m2T0rDbTeEiNTeKfJYwZ+AKddx7BD/EJLP72 oSdxYl9t8HnPVs0fyoJgi3vwgvtJGmU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none); spf=pass (imf27.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686583995; a=rsa-sha256; cv=none; b=dncPw64UZSWsk/UZXeJcAbA2VqWS2GQ68NnTrTNUMBxuya7vuH7xKFsRAo2cdzEPw0xjN4 kggAbyhJUN8l10luw0wxLlWHHbbIG+t1Z/sMgvq1f8JMoRzTnhFyFMN5LF512hPJ+2GvOk Pgj/3f4ZVtqnLfF8SQawk+Y2O00NM/I= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DA03261FB6; Mon, 12 Jun 2023 15:33:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3352CC433B3; Mon, 12 Jun 2023 15:33:10 +0000 (UTC) From: Catalin Marinas To: Andrew Morton , Linus Torvalds Cc: Christoph Hellwig , Robin Murphy , Arnd Bergmann , Greg Kroah-Hartman , Will Deacon , Marc Zyngier , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Jonathan Cameron , Vlastimil Babka , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v7 15/17] iommu/dma: Force bouncing if the size is not cacheline-aligned Date: Mon, 12 Jun 2023 16:31:59 +0100 Message-Id: <20230612153201.554742-16-catalin.marinas@arm.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230612153201.554742-1-catalin.marinas@arm.com> References: <20230612153201.554742-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AF54B4000E X-Stat-Signature: 7aaqi7oiret4b8q5qzo5r4tddhuw83bd X-HE-Tag: 1686583995-34743 X-HE-Meta: U2FsdGVkX1/3saP4fLRIvQ5VMpB/ynb1CdJXkGQjfQf7azJpHLm+4NNu8UDcNsY1P8mBwUQQP/XtKk0+JSLz1aQq+yEU9AfzehLlZhL0t81v7Q+Rdd5yf69MQBUZ26iggZpZ1l2rVjg4ewI8Ji3W5ZjkspzOS0wVmBbX+//drrFYdWWsmTzY7csyNE6gwW02FSKTOMICscI2qwKuFgo8OCCDY2wUH4bJahSydM1g+md8Nv/YuSOrxIDNpJ/C9Im1y23W6KSe0xEkdmgzYV4s9LciSpNoTRW0ImqB06Y3MlJ2vGHpefH/hPn1vvJJDVOnbZSoa8CWdtiKKefedri3Tei+DcDevfa4ki1orsptMwuhC1K0QBhuAtsEslhTL9V0cJlbtqI99LuXY7wQKxFvIuvK8UNeBl+2WPzKBPdBMUE+9o32t52tJ5dEvJIFSxr3H8rPbz14uhn4JkKvcZbBma2QivYv3JlNdbhcEK3nus6vn5KKJ+Si1Ozp5U4oMrlYXqF/rABp45sq+YffAv0+mv5p0NoQApU+WeDpQAAhQzfpMbCXVEpzUxzyTElsZefkOY7wY0ysxGGh9eDnIxrNSyYBTgXoG7qrIrZKOi997CB3SKosCl3fcu8BVwt7y8gSDsq1tuOPCG+jLnEIl3xie9FwTwl1JWJRM46tffRSU+B3HL736Apnucj/YimUxGqfqy/vbTUTYtyYCI75UsJPjmPg4ewp+m1QOZd+EGR426bI36Jnd19K+R7xRa+i6EPdauFfp+HPt3MxnYNwHB4etaQcqUqv0/xhsV1D/EjAPyQ0Ct2XUhcyabhc1gw6QSMIWtVBX1xlvRdsWFCSSumdgV6VW3rsrnDAnYtHMxSi6QUSdsLmJRU8SUjDFgdTkn+Fni8vrnEZ1ZgOuMdo0+JJBl014sD+7EjojgTuq0GC9oE6K108pu0wa3RkOUj2Q/ZvVIcMfWybY7HOIau2bLK tfjk7m0f 2pO8Km+cDv3JiOCwF8WOAh2JWvPAbT/0ymHcv37GexRgKISM9X9NVGnuNLG1VMXdfCweOo9tiR2//ACiofA0MavD/SJz3Kl3a11aHVm9uBKjFW+mD0LhTDFaXZKAkIDUmec1kC4NWw5Bq98DuyFSadJmXey19gosK0ujUrM/cXystlyLm0UKtbTWdMd/Zw8ApVOmXN+OEy8q+5VmNJSmJ5jta+XLY4PtvkCIydOkMFAsgAn+EgqZ7G+wdEm6+NCxl0w253OIjwJILN7eCCxsMuGY9XPuxjBnFyPfS1C8r/i3prKW5NUlJVHq9Lxhg+Gr0Ob192ZaD8mqtLBA4SC6nHaP3aVCHwAoVBOntmccBEJNBNdJu9lvoyf43DXD/+eq+6XpKT6sfy/jfcGwRYCTtL++76BsNFcThAhUH+WTG79KULnAqKBPO0F8Y2Z/enhJqV5J6SZuvAXgL743fSxODbfxeY09x3KKuNopnZsrSRau+Ou3MGOxW66JpF7JIPJ/pX14tEqhMdwW2a+FXIYecRMXfotc2o6O2jOQ2taPhM6tuxd4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similarly to the direct DMA, bounce small allocations as they may have originated from a kmalloc() cache not safe for DMA. Unlike the direct DMA, iommu_dma_map_sg() cannot call iommu_dma_map_sg_swiotlb() for all non-coherent devices as this would break some cases where the iova is expected to be contiguous (dmabuf). Instead, scan the scatterlist for any small sizes and only go the swiotlb path if any element of the list needs bouncing (note that iommu_dma_map_page() would still only bounce those buffers which are not DMA-aligned). To avoid scanning the scatterlist on the 'sync' operations, introduce an SG_DMA_SWIOTLB flag set by iommu_dma_map_sg_swiotlb(). The dev_use_swiotlb() function together with the newly added dev_use_sg_swiotlb() now check for both untrusted devices and unaligned kmalloc() buffers (suggested by Robin Murphy). Signed-off-by: Catalin Marinas Reviewed-by: Robin Murphy Cc: Joerg Roedel Cc: Christoph Hellwig Tested-by: Isaac J. Manjarres --- drivers/iommu/Kconfig | 1 + drivers/iommu/dma-iommu.c | 50 ++++++++++++++++++++++++++++++------- include/linux/scatterlist.h | 41 ++++++++++++++++++++++++++++-- 3 files changed, 81 insertions(+), 11 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index db98c3f86e8c..670eff7a8e11 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -152,6 +152,7 @@ config IOMMU_DMA select IOMMU_IOVA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH + select NEED_SG_DMA_FLAGS if SWIOTLB # Shared Virtual Addressing config IOMMU_SVA diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index b8bba4aa196f..e86ae462cade 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -520,9 +520,38 @@ static bool dev_is_untrusted(struct device *dev) return dev_is_pci(dev) && to_pci_dev(dev)->untrusted; } -static bool dev_use_swiotlb(struct device *dev) +static bool dev_use_swiotlb(struct device *dev, size_t size, + enum dma_data_direction dir) { - return IS_ENABLED(CONFIG_SWIOTLB) && dev_is_untrusted(dev); + return IS_ENABLED(CONFIG_SWIOTLB) && + (dev_is_untrusted(dev) || + dma_kmalloc_needs_bounce(dev, size, dir)); +} + +static bool dev_use_sg_swiotlb(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir) +{ + struct scatterlist *s; + int i; + + if (!IS_ENABLED(CONFIG_SWIOTLB)) + return false; + + if (dev_is_untrusted(dev)) + return true; + + /* + * If kmalloc() buffers are not DMA-safe for this device and + * direction, check the individual lengths in the sg list. If any + * element is deemed unsafe, use the swiotlb for bouncing. + */ + if (!dma_kmalloc_safe(dev, dir)) { + for_each_sg(sg, s, nents, i) + if (!dma_kmalloc_size_aligned(s->length)) + return true; + } + + return false; } /** @@ -922,7 +951,7 @@ static void iommu_dma_sync_single_for_cpu(struct device *dev, { phys_addr_t phys; - if (dev_is_dma_coherent(dev) && !dev_use_swiotlb(dev)) + if (dev_is_dma_coherent(dev) && !dev_use_swiotlb(dev, size, dir)) return; phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); @@ -938,7 +967,7 @@ static void iommu_dma_sync_single_for_device(struct device *dev, { phys_addr_t phys; - if (dev_is_dma_coherent(dev) && !dev_use_swiotlb(dev)) + if (dev_is_dma_coherent(dev) && !dev_use_swiotlb(dev, size, dir)) return; phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); @@ -956,7 +985,7 @@ static void iommu_dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg; int i; - if (dev_use_swiotlb(dev)) + if (sg_dma_is_swiotlb(sgl)) for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_cpu(dev, sg_dma_address(sg), sg->length, dir); @@ -972,7 +1001,7 @@ static void iommu_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg; int i; - if (dev_use_swiotlb(dev)) + if (sg_dma_is_swiotlb(sgl)) for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_device(dev, sg_dma_address(sg), @@ -998,7 +1027,8 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, * If both the physical buffer start address and size are * page aligned, we don't need to use a bounce page. */ - if (dev_use_swiotlb(dev) && iova_offset(iovad, phys | size)) { + if (dev_use_swiotlb(dev, size, dir) && + iova_offset(iovad, phys | size)) { void *padding_start; size_t padding_size, aligned_size; @@ -1166,6 +1196,8 @@ static int iommu_dma_map_sg_swiotlb(struct device *dev, struct scatterlist *sg, struct scatterlist *s; int i; + sg_dma_mark_swiotlb(sg); + for_each_sg(sg, s, nents, i) { sg_dma_address(s) = iommu_dma_map_page(dev, sg_page(s), s->offset, s->length, dir, attrs); @@ -1210,7 +1242,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, goto out; } - if (dev_use_swiotlb(dev)) + if (dev_use_sg_swiotlb(dev, sg, nents, dir)) return iommu_dma_map_sg_swiotlb(dev, sg, nents, dir, attrs); if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) @@ -1315,7 +1347,7 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, struct scatterlist *tmp; int i; - if (dev_use_swiotlb(dev)) { + if (sg_dma_is_swiotlb(sg)) { iommu_dma_unmap_sg_swiotlb(dev, sg, nents, dir, attrs); return; } diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index 2f06178996ba..ec46d8e8e49d 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -251,11 +251,13 @@ static inline void sg_unmark_end(struct scatterlist *sg) /* * One 64-bit architectures there is a 4-byte padding in struct scatterlist * (assuming also CONFIG_NEED_SG_DMA_LENGTH is set). Use this padding for DMA - * flags bits to indicate when a specific dma address is a bus address. + * flags bits to indicate when a specific dma address is a bus address or the + * buffer may have been bounced via SWIOTLB. */ #ifdef CONFIG_NEED_SG_DMA_FLAGS -#define SG_DMA_BUS_ADDRESS (1 << 0) +#define SG_DMA_BUS_ADDRESS (1 << 0) +#define SG_DMA_SWIOTLB (1 << 1) /** * sg_dma_is_bus_address - Return whether a given segment was marked @@ -298,6 +300,34 @@ static inline void sg_dma_unmark_bus_address(struct scatterlist *sg) sg->dma_flags &= ~SG_DMA_BUS_ADDRESS; } +/** + * sg_dma_is_swiotlb - Return whether the scatterlist was marked for SWIOTLB + * bouncing + * @sg: SG entry + * + * Description: + * Returns true if the scatterlist was marked for SWIOTLB bouncing. Not all + * elements may have been bounced, so the caller would have to check + * individual SG entries with is_swiotlb_buffer(). + */ +static inline bool sg_dma_is_swiotlb(struct scatterlist *sg) +{ + return sg->dma_flags & SG_DMA_SWIOTLB; +} + +/** + * sg_dma_mark_swiotlb - Mark the scatterlist for SWIOTLB bouncing + * @sg: SG entry + * + * Description: + * Marks a a scatterlist for SWIOTLB bounce. Not all SG entries may be + * bounced. + */ +static inline void sg_dma_mark_swiotlb(struct scatterlist *sg) +{ + sg->dma_flags |= SG_DMA_SWIOTLB; +} + #else static inline bool sg_dma_is_bus_address(struct scatterlist *sg) @@ -310,6 +340,13 @@ static inline void sg_dma_mark_bus_address(struct scatterlist *sg) static inline void sg_dma_unmark_bus_address(struct scatterlist *sg) { } +static inline bool sg_dma_is_swiotlb(struct scatterlist *sg) +{ + return false; +} +static inline void sg_dma_mark_swiotlb(struct scatterlist *sg) +{ +} #endif /* CONFIG_NEED_SG_DMA_FLAGS */