From patchwork Thu May 18 17:34:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13247179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D955C7EE23 for ; Thu, 18 May 2023 17:35:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FAB5900015; Thu, 18 May 2023 13:35:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AB41900003; Thu, 18 May 2023 13:35:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 173DD900015; Thu, 18 May 2023 13:35:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 07DCB900003 for ; Thu, 18 May 2023 13:35:07 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CB125A0915 for ; Thu, 18 May 2023 17:35:06 +0000 (UTC) X-FDA: 80804076612.29.3378346 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf01.hostedemail.com (Postfix) with ESMTP id DBC854001C for ; Thu, 18 May 2023 17:35:03 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684431304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sVNhDYozvMQiV5O03sPIWzZhD42mGtaSbkIbs8wrJEY=; b=XSJK6JCRxVKwEjVIWB0FYBYp/Sns+tPxTSF4HIGy5fSfkAeV+dMzjKx4QCg6xfMHhFPSNh nEkPxbW5LU+k4rTfs20L7yuLTsNeHk4YLfxgXvpqCR9G7lsNVe63ssYUfJ2RjwzdWYSAsM pwgMBBfO6l7erera8oGA0/Fje4BQxRU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684431304; a=rsa-sha256; cv=none; b=EfjyFlKPimFYxjHB2Lcty51vshOeRke+izllpDmc7W3i7j7ZITbH5RGIln2ssCyNX3sB2p NwBuovVqff1lKgxf6S/qYeL37ChLX2mWWsdSCl+FWn67UVe0uk80bhqqupwp17XB3MeBZG 2jToDu3BticBmdOY7JX8rJ/GQbcUkck= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C82B06516C; Thu, 18 May 2023 17:35:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 50B11C4339B; Thu, 18 May 2023 17:34:58 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann , Christoph Hellwig , Greg Kroah-Hartman Cc: Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Isaac Manjarres , Saravana Kannan , Alasdair Kergon , Daniel Vetter , Joerg Roedel , Mark Brown , Mike Snitzer , "Rafael J. Wysocki" , Robin Murphy , linux-mm@kvack.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Subject: [PATCH v4 13/15] iommu/dma: Force bouncing if the size is not cacheline-aligned Date: Thu, 18 May 2023 18:34:01 +0100 Message-Id: <20230518173403.1150549-14-catalin.marinas@arm.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230518173403.1150549-1-catalin.marinas@arm.com> References: <20230518173403.1150549-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DBC854001C X-Rspam-User: X-Stat-Signature: meyfaj4z1irgy51re1weepqegpm9hji8 X-Rspamd-Server: rspam03 X-HE-Tag: 1684431303-786786 X-HE-Meta: U2FsdGVkX185tgS23H/2Pkkz8iVRZYrbv/LFHzZxmHEMyQ+k6yfGSfeQMRZmRKLl16/GI9uSvnEEWqULxLlln7UACRskUlIXGnptgK5SnhrDMYPQmDPUGntqKku54rjgvmCNaHf7D2eS18HjHg6rHTX/cdpaGjiADURhkoXmAAx7DZdx+TG6c1U5vxnrJ8SbGDOjwZ2JHVMJcNTfC+tW0uK721Dr41b6oH3Ldi7P7wqc+e72MYlMBU3aOe8tyWm5VcaHjc4Fi4WiO1PjP9nCRE7h3jndhnFcORCibuySozqAvkR8D1ahXVgg8QRKO9vPUo/4x83mRshlIqxxFK5xpLvrsTudFQUKQ9lmj01UKO0e+0w2NRlG8JyqHXUoFGLttEAigY+7mXvxKbElvIFBefZS/sRKx1eOCa4I2XQj9dhyH4O4Xl6cqhXE9vuVNzllgRz/H1XIKyJU0MnlXc56YjkiKvT2Ql+mdn8MsQMKnzC43ZwDnmjfW0hiaWYogfmlAY0xCBqpK9fnWdOZ/4yID4BK3VW5XnOMYtGZtnglvo4zuQDWy2FJ2MjibsNu8R3BK5CwutmRPkzdsbeNqZu9yL1QX/1h1K9Rg6Z9QSUjvU1JS7us8V/WIrw4qa+25wtzyKZkohAoYQAN4f/Gsz1FlFqnFnueGN35hH0UuPGowLFdtKOYk2GfKKRviSqYGgE2aK0jKaFm1xRairZzsKn5q0e/IBazeC124Os0rjQR6pE3Ur4gbgF8qoH9+yDi5iKZq03fScFneq9nv8m5YvWc9yXvSUktwcu/yYkBjwPth6AEx5qMo93hsQn1N25OBXVU3IHd5AzVXxMgeegisIp/7HRniJBopxOjyznZCFBAZanu4rk+8kfdTSs/+HHzcVmiucyqrkZI9NywM3nsorssffi1vGYleCuL859tW5lCt5pVJ04vLSRTu2x0vlZ5ShJByQx7etbH5wdVVwpjjCU fwHnVsBS a63Z/70biYnflAQoQ9WBkX2QZ9avsWRhtQiGvsJeLkdVZFrGbnbS8ZZSSX15tm2HTSOrBjeL2kZA85PYF7ycGlraukMOHLr4x5XSZXhH62qibL3tbQaWHN5ceGbWJe2be8zvEpiPaCgnWfUOul1F3r4ZPyPmdHfD34loZxFeMtdbAReAN/xsrnhAO9z63I0iLEte/S5YIzurpx/uzwqjxv6OLMbgjduQuQc3LYjHf1d7wH94QwOk9AZrYs8cBn4GWIBo3BZTCa2ihmsiuhBKSC1rmQSZGrmafueaClKm9X7blVJwk1xC4tW+IKNlbLgx0RxRBrAvUi6xB/4raUUjItAUUkrnDrQ5n/NfaYPaf7zgJjjVRtrHpSfUy/3Hne9SmKNrqe60wMlx5mrC2MO+b8fEWy7bQL691eFpA78Gx3kjQ1dGFaTVAr9UTIsiRTPC1dm1oddwQnKihE1L5EcEy+c1MQOELSOC2cQp3ySm6Eh63CK+LSFtBSDaLXrCe5QTG8VSOt0GWo77YMlIXuHlbqNuZkQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similarly to the direct DMA, bounce small allocations as they may have originated from a kmalloc() cache not safe for DMA. Unlike the direct DMA, iommu_dma_map_sg() cannot call iommu_dma_map_sg_swiotlb() for all non-coherent devices as this would break some cases where the iova is expected to be contiguous (dmabuf). Instead, scan the scatterlist for any small sizes and only go the swiotlb path if any element of the list needs bouncing (note that iommu_dma_map_page() would still only bounce those buffers which are not DMA-aligned). To avoid scanning the scatterlist on the 'sync' operations, introduce a SG_DMA_BOUNCED flag set during the iommu_dma_map_sg() call (suggested by Robin Murphy). Signed-off-by: Catalin Marinas Cc: Joerg Roedel Cc: Christoph Hellwig Cc: Robin Murphy Signed-off-by: Catalin Marinas --- drivers/iommu/dma-iommu.c | 25 ++++++++++++++++++++----- include/linux/scatterlist.h | 25 +++++++++++++++++++++++-- 2 files changed, 43 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 7a9f0b0bddbd..ab1c1681c06e 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -956,7 +956,7 @@ static void iommu_dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg; int i; - if (dev_use_swiotlb(dev)) + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sgl)) for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_cpu(dev, sg_dma_address(sg), sg->length, dir); @@ -972,7 +972,7 @@ static void iommu_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg; int i; - if (dev_use_swiotlb(dev)) + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sgl)) for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_device(dev, sg_dma_address(sg), @@ -998,7 +998,8 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, * If both the physical buffer start address and size are * page aligned, we don't need to use a bounce page. */ - if (dev_use_swiotlb(dev) && iova_offset(iovad, phys | size)) { + if ((dev_use_swiotlb(dev) && iova_offset(iovad, phys | size)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { void *padding_start; size_t padding_size, aligned_size; @@ -1210,7 +1211,21 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, goto out; } - if (dev_use_swiotlb(dev)) + /* + * If kmalloc() buffers are not DMA-safe for this device and + * direction, check the individual lengths in the sg list. If one of + * the buffers is deemed unsafe, follow the iommu_dma_map_sg_swiotlb() + * path for potential bouncing. + */ + if (!dma_kmalloc_safe(dev, dir)) { + for_each_sg(sg, s, nents, i) + if (!dma_kmalloc_size_aligned(s->length)) { + sg_dma_mark_bounced(sg); + break; + } + } + + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sg)) return iommu_dma_map_sg_swiotlb(dev, sg, nents, dir, attrs); if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) @@ -1315,7 +1330,7 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, struct scatterlist *tmp; int i; - if (dev_use_swiotlb(dev)) { + if (dev_use_swiotlb(dev) || sg_is_dma_bounced(sg)) { iommu_dma_unmap_sg_swiotlb(dev, sg, nents, dir, attrs); return; } diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index 87aaf8b5cdb4..9306880cae1c 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -248,6 +248,29 @@ static inline void sg_unmark_end(struct scatterlist *sg) sg->page_link &= ~SG_END; } +#define SG_DMA_BUS_ADDRESS (1 << 0) +#define SG_DMA_BOUNCED (1 << 1) + +#ifdef CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC +static inline bool sg_is_dma_bounced(struct scatterlist *sg) +{ + return sg->dma_flags & SG_DMA_BOUNCED; +} + +static inline void sg_dma_mark_bounced(struct scatterlist *sg) +{ + sg->dma_flags |= SG_DMA_BOUNCED; +} +#else +static inline bool sg_is_dma_bounced(struct scatterlist *sg) +{ + return false; +} +static inline void sg_dma_mark_bounced(struct scatterlist *sg) +{ +} +#endif + /* * CONFIG_PCI_P2PDMA depends on CONFIG_64BIT which means there is 4 bytes * in struct scatterlist (assuming also CONFIG_NEED_SG_DMA_LENGTH is set). @@ -256,8 +279,6 @@ static inline void sg_unmark_end(struct scatterlist *sg) */ #ifdef CONFIG_PCI_P2PDMA -#define SG_DMA_BUS_ADDRESS (1 << 0) - /** * sg_dma_is_bus address - Return whether a given segment was marked * as a bus address